Does running a local LLM make sense for a small SaaS?

“Let’s run the model on our own machine” becomes persuasive quickly in a product meeting: data does not leave, there is no per-request fee, everything is under our control. The trouble is that all three promises are conditional. This post is not a list of recommendations but a decision framework. It looks at which questions to ask along the axes of cost, latency and privacy, and which answers change the decision.

One caveat up front: this post contains no figures such as the price of a hosted model API, the rental cost of a GPU or a model’s tokens-per-second speed. Those numbers change fast, and none could be tied to a dated, verified source for this post; rather than give unsourced numbers, the reasoning is kept qualitative. Concrete figures should be recomputed with your own workload and current price pages when the decision is made. The concrete facts below come only from the llama.cpp and Ollama pages themselves.

What does running locally mean?

Two of the most common tools for running models locally are llama.cpp and Ollama. According to its project page, llama.cpp offers integer quantization (compression that lets a model fit in less memory) from 1.5 bits to 8 bits, supports hybrid inference that runs models too large for GPU memory partly on the GPU and partly on the CPU, and can launch an OpenAI-compatible API server. Hardware support is broad: CUDA for Nvidia, HIP for AMD, Metal for Apple silicon, plus Vulkan and SYCL. Both projects are MIT-licensed. Ollama, according to its GitHub page, is a runner that installs on macOS, Windows, Linux and Docker and exposes a REST API.

What this means for the decision: running locally is technically within reach and can be done with openly licensed tools. But the existence of tools does not show that the job will be cheap or easy for you. The real question is which model will answer at what quality and speed on which hardware, and the answer to that comes from your own measurements.

Cost: a variable expense becomes a fixed one

With a hosted model service, cost is proportional to use: more requests, bigger bill. When user numbers are small that is an advantage, because unused capacity costs nothing. Running locally flips the curve. Hardware is either bought or rented, and the charge runs whether or not requests arrive. Cost moves from variable to fixed.

This has two consequences. First, the break-even point depends on usage: if request volume is small and uneven, the fixed cost is wasted; if volume is large and steady, the fixed cost can lower the unit cost. Second, comparing only on price per token is misleading. A local setup carries labour items on top of the hardware, such as updates, monitoring, swapping models and responding when something breaks. In a small team, that labour is the scarce resource.

There is also a quality line item. Models that can run locally are generally smaller than the large hosted ones, and quantization can trade some quality for memory savings. The quality gap may be negligible or decisive depending on the use case: a small model may be enough for a narrow task such as classification or short summarization, and not enough for a feature that needs open-ended reasoning. That is to be learned not by guessing but by trying your own sample data.

Latency: the network gets shorter, the waiting can get longer

Running locally has one advantage for latency: the request does not travel over an internet connection to another company. But that advantage competes with the model’s own speed. The defaults on Ollama’s FAQ page give useful hints here.

According to the docs, the number of parallel requests per model (OLLAMA_NUM_PARALLEL) is 1 by default, which means that when two users arrive at once the second waits in line. The queue limit for waiting requests (OLLAMA_MAX_QUEUE) is 512, and requests beyond it get a 503 error. Raising parallelism is possible, but the docs say the required memory grows with this value multiplied by the context length. In other words, concurrency is not free; it turns into memory. The default context length is 4096 tokens, and a feature that processes long documents has to raise it, which enlarges the memory requirement.

Another detail is model load time. By default Ollama unloads a model from memory 5 minutes after its last use (OLLAMA_KEEP_ALIVE). That means the first request after a quiet spell has to reload the model; the load time depends on the model and the hardware, but it creates a “slow first request” tail in the latency profile. The setting can be changed, but then memory stays occupied.

So the latency question should be asked not as “local or remote” but as “how many concurrent users, what context length, what response-time target”. In a small SaaS with few concurrent users, a setup with one parallel request may be enough. When traffic fluctuates, the queue and the cold start shape the user experience.

Privacy: where does the data go, who sees it?

Privacy is the strongest argument for running locally and its most misunderstood side. According to Ollama’s FAQ, when models run locally Ollama does not see prompts or data. Cloud-hosted models are a separate path: there, prompts and responses are processed to provide the service, and Ollama states they are not stored or logged and never used for training. Cloud features can be turned off entirely (OLLAMA_NO_CLOUD=1). So running locally keeps prompts from reaching a third party, as long as you make sure you are using only the local path.

But this does not solve the whole privacy problem. Even when data stays on your own server, it remains a responsibility in log files, backups and access permissions. A hosted service can offer assurance through contracts and certifications; on your own server you have to build that assurance yourself. If customer data counts as personal data, legal obligations apply locally too. If the privacy concern is real, the test should be “does the data leave”; but “is it safe inside” has to be answered separately.

A decision framework

The three axes can be reduced to one question: what does running locally make cheaper, faster or better protected for this SaaS, and at what price?

If request volume is small and uneven, a hosted service is usually the easier choice on cost; a local setup brings a fixed cost and operating labour.
If volume is large and steady, the chance that a fixed cost lowers the unit cost rises, but that calculation has to be made with measurements of your own workload.
If the latency target is tight and there are many concurrent users, the parallelism and memory arithmetic has to be done in advance; since the default parallelism is 1, the default setting will not be enough on its own.
If it is a requirement that data never reach third parties (by contract or regulation), running locally is the natural way to meet it and may be worth the cost on the other two axes.
If no axis shows clear pressure, the most reasonable start is usually a hosted service; if an abstraction layer is placed in front of the model, moving to a local model later stays cheap. That llama.cpp can launch an OpenAI-compatible API server is a detail that eases such a move.

In the end, “does a local LLM make sense” has no universal answer. The cases where it does depend on specific conditions, such as a privacy requirement or high, steady volume. For a small SaaS the default is not to decide without measuring: a small trial on your own data, a calculation from current price pages, and only then a choice.