> Source: https://souverana.ch/insights/ollama-llamacpp-vllm-en/
> Language: en

# Ollama, llama.cpp or vLLM: choosing the right engine

With four concurrent users on the same card and the same model, the fourth waits 62 seconds under Ollama and 8 under llama.cpp. Two questions decide which software suits you: how many people work on it at the same time, and how much graphics memory do you have relative to the model you want?

Technical · Published 14 Aug 2026 · Updated 21 Oct 2026 · Joel Barmettler

## Ollama, llama.cpp or vLLM: which engine do I need?

**Anyone who tests local AI with a team on Ollama will find it slow: with four concurrent users, the fourth waited 62 seconds for the first character in our measurement, against 8 with llama.cpp on the same card. The choice of software depends on the number of concurrent users and on whether the model fits entirely into graphics memory.**

**In brief**

-   Ollama processes requests one after another. In our measurement with four concurrent users, the fourth waited 62 seconds for the first character of the answer. llama.cpp served the same four users, with the same model file on the same graphics card, after 8 seconds.
-   For a single person, Ollama is the most comfortable choice. It installs in minutes, and in our measurement the first answer arrived after 0.9 seconds.
-   Anyone who wants to run the largest possible model on small hardware should use llama.cpp. With it, a 35-billion model ran for us in 4.5 GB of graphics memory and a 284-billion model on two consumer cards, though more slowly.
-   vLLM pays off with many concurrent users and a model that fits comfortably into graphics memory. At eight users it delivered 240 tokens per second, llama.cpp 165.
-   SGLang, another piece of software for requests with shared context, would not start with our model. Before choosing vLLM or SGLang, check whether your model is supported.

Two measurements come first. Same model file, same graphics card (an RTX 4090), one user: Ollama delivers 43.8 tokens per second (a token is a chunk of text a few characters long), llama.cpp 43.4. Practically identical. Same file, same card, four users: Ollama falls to 38.7, llama.cpp rises to 125.1.[1]

These two measurements show the first axis of the choice: the number of concurrent users. The second axis is your graphics memory relative to the model you want. Here the two large engines differ most. vLLM is built for machines whose graphics memory comfortably fits the model, and turns the remaining graphics memory into throughput. llama.cpp is built for the reverse case and extracts the largest possible model from tight graphics memory.

The engine, in this context, is the software that loads a model, accepts requests and keeps the graphics card busy. It has no influence on how clever the answers are. We measured three tools on the same hardware, plus one specialist that never made it to the measurement stage.

This article is the fourth part of our series on local inference. After the machine, the memory and quantisation, this part turns to the software that serves the model.

**Series · Local Inference**

1.  [Self-hosting an LLM: GPT-5 performance for CHF 8’500](https://souverana.ch/en/insights/llm-selbst-hosten/)
2.  [VRAM for LLMs: how to size it correctly before you buy](https://souverana.ch/en/insights/vram-llm/)
3.  [LLM quantisation: how it works and how to use it well](https://souverana.ch/en/insights/llm-quantisierung/)
4.  Ollama, llama.cpp or vLLM: choosing the right engine (this part)
5.  [Self-hosting an LLM: the real cost and the break-even](https://souverana.ch/en/insights/llm-selbst-hosten-kosten/)

## For the single workstation: Ollama

Ollama installs itself in minutes, loads models with a single command, and uses the same compute kernels as llama.cpp internally. At one user the two reach 43.8 and 43.4 tokens per second, which shows it: same kernels, same quantisation, same card.[1]

For the person working with it alone, Ollama answers fastest: the first answer arrives after 0.92 seconds, against 3.65 seconds for llama.cpp with our server configuration.[1]

The problem begins with the second concurrent user.

**Chart: Same GGUF file on the same card: identical at first, then a factor of three**

*Output tokens per second, all users combined*

| Category | Ollama 0.32.9 (tok/s) | llama.cpp b10380 (tok/s) |
| --- | --- | --- |
| 1 user | 43.8 | 43.4 |
| 4 users | 38.7 | 125.1 |

Source: own measurement, August 2026, Qwen3.6-27B Q4\_K\_M, one RTX 4090; Ollama in default settings, llama.cpp configured with four slots

At four concurrent users, Ollama loses a little throughput, and the wait for the first character jumps from 0.92 to **62 seconds**. That is the pattern of a queue: the fourth user waits until the first three are done. llama.cpp, configured with four slots (four parallel processing positions), processes the same four requests together on the card.[1]

We also tested the setting the documentation offers for this. With `OLLAMA_NUM_PARALLEL=4`, throughput at four users stayed unchanged (38.6 tokens per second), and the single-user speed fell markedly (26.6 compared with 43.8). The four slots may have shared the configured context, which would explain the degradation. All we can say with certainty is that, with the options the documentation describes, throughput could not be raised to the level of llama.cpp.[1]

Ollama is a single-workstation tool and good for that workstation.

## For the small server: llama.cpp

llama.cpp is the software that Ollama uses internally, without its comfort and without its limits: a single program that loads GGUF files, offers an OpenAI-compatible interface, and batches requests. In our runs, the server was ready around five seconds after starting, which makes switching models and experimenting quick.

Two capabilities make it the first choice for a small team with its own server.

**It batches requests.** The 125.1 tokens per second at four users from the chart above are almost three times the 43.4 tokens per second at one user, on the same single card. The wait for the first character stayed at 8.1 seconds, while Ollama left the same four users waiting 62 seconds.[1]

**It fits the largest possible model into tight graphics memory.** llama.cpp selectively places the rarely used experts of an MoE model in system memory (`--n-cpu-moe`, covered in detail in Part 2 of this series). That let a 35-billion model run for us in 4.5 GB of graphics memory, and a 284-billion model at 9.6 tokens per second on two consumer cards. vLLM also has an offload mechanism, but an indiscriminate one: offloading 8 GB of a dense model (a model in which all parameters are used for every token) in vLLM cost a factor of 22 in speed.[2] Of the three tools, only llama.cpp offers an offload that remains usable.

The price: you configure it yourself. You set context length, parallelism, offload and flash attention through start options (flags), and the defaults rarely fit. The commands we used for our measurements are in the appendix.

## For production: vLLM

vLLM is built for the case where the model fits comfortably into graphics memory and the card is constantly busy: many users, long inputs, continuous operation. It runs continuous batching (new requests join in the middle of running computations), reads AWQ and FP8 formats, and manages the KV cache in pages.

On two cards under realistic load (10,000 input tokens, 2,000 output):

**Chart: Two cards, rising user count: vLLM pulls ahead, llama.cpp stays faster to the first character**

*Output tokens per second, all users combined*

| Category | vLLM 0.27.1 (AWQ) (tok/s) | llama.cpp b10380 (Q4\_K\_M) (tok/s) |
| --- | --- | --- |
| 1 | 65.9 | 44.8 |
| 4 | 169.5 | 83.9 |
| 8 | 240.2 | 164.9 |

Source: own measurement, August 2026, Qwen3.6-27B 4-bit, 2x RTX 4090

At four users, vLLM delivers double the throughput of llama.cpp; at eight, a 46 percent lead remains. Two disadvantages weigh against this. At a single user, the first character appeared after 3.5 seconds, against 0.6 for llama.cpp. The first start also took minutes, because vLLM compiles its kernels, while llama.cpp was ready after five seconds.[2]

vLLM benefits most from workloads that share a context, meaning RAG and agents with a fixed system prompt. Its prefix caching processes the shared part only once: in our measurement with 8,000 tokens of shared context and eight users, the wait time to the first character fell from 17.9 to 0.9 seconds, a 19-fold reduction from a single switch.[2]

> **The catch: not every model can play along**
> 
> Prefix caching requires a pure attention architecture. Hybrid models mix attention with other layer types, and our Qwen3.6 is one of them. For such models, vLLM switches the feature off, and the factor of 19 disappears with no substitute. [Anyone planning RAG](https://souverana.ch/en/insights/souveraene-ki-plattform/) should therefore check, before choosing a model, whether its architecture allows prefix caching. We measured the effect with Apertus-8B, a pure attention model.

## The specialist that never started: SGLang

SGLang specialises in shared contexts and would have been the fourth candidate for agent workloads. On our model, it never got that far: the startup failed because of a bug, open since July, affecting hybrid models (GitHub issue #30178); the documented workaround fixed the first crash, after which the first request hung in a timeout.[3]

This says little about SGLang as a whole; others report good numbers on pure attention models. The case shows how immature the tooling is for new architectures: in our experience, only a few engines actually support a new model design. Check the combination of model and engine before committing to any of them.

## Matching tool to situation

| Your situation | Tool | The measurement behind it |
| --- | --- | --- |
| One person, own computer | Ollama | first answer after 0.9 s, speed identical to llama.cpp |
| Small team, one server | llama.cpp | triples throughput at four users, starts in 5 s |
| Model larger than graphics memory | llama.cpp | 35 billion parameters in 4.5 GB, 284 billion on two cards |
| Model fits comfortably, many users | vLLM | 240 tokens per second at eight users[2] |
| RAG and agents with fixed context | vLLM with an attention model | wait time cut by a factor of 19[2] |

Each of the three tools beats the others in one case. In our measurements, the wrong choice cost 62 seconds of wait time, on a card that could already answer after 0.9 seconds.

Want engine and model sized for your load before you go into production?

[AI architecture & solution selection](https://souverana.ch/en/leistungen/ki-architektur/)

## How to get started

Four steps lead from the first check to continuous operation.

1.  **Check the model with Ollama**: To find out whether the model can handle your tasks, a single workstation is enough. The check costs an afternoon and needs no server.
2.  **Switch to llama.cpp for the team test**: The same GGUF file keeps running there. Set context length and parallelism deliberately; the start commands from the appendix are a tested starting point.
3.  **Measure vLLM against your load before continuous operation**: Measure with the built-in benchmark (vllm bench serve) and your real input length. Without that length, you cannot answer how many users the server can handle.
4.  **Record the engine version**: Our first round of measurements ran by mistake on a vLLM installation eight minor versions old. Note the version before you measure, or the figures apply only to an outdated installation.

That leaves the question of what the whole setup costs and when it pays off. The fifth and final part of the series answers that.

## Sources

[1] Souverana. «Eigene Messung, August 2026, Qwen3.6-27B Q4\_K\_M, eine RTX 4090; Ollama in Standardeinstellungen, llama.cpp mit vier Slots konfiguriert». Souverana. August 2026. Eigene Messung, Primärquelle.

[2] Souverana. «Eigene Messung, August 2026, Qwen3.6-27B 4-Bit, 2x RTX 4090». Souverana. August 2026. Eigene Messung, Primärquelle.

The author

Joel Barmettler

AI Architect · Souverana, Zurich

Joel Barmettler guides Swiss companies from AI strategy to integration: sovereign, confidential and production-ready. He built the Swiss AI Hub as its architect and today co-owns its architecture; he personally leads every Souverana mandate. Mandates from one-person firms to Fortune 500 corporations.

[Book an intro call](https://meet.brevo.com/joel-barmettler/30-minute-meeting) [More about Souverana](https://souverana.ch/en/) [LinkedIn](https://www.linkedin.com/in/joel-barmettler-b9ab361b7)

## Frequently asked questions

What is better, Ollama or vLLM?

For a single person, Ollama is the most comfortable choice: in our measurement it delivered the first answer after 0.9 seconds. With several concurrent users, the service belongs on llama.cpp or vLLM, two other engines. Ollama processed requests one after another and left the fourth user waiting 62 seconds.

Is Ollama slower than llama.cpp?

Not with a single user: we measured 43.8 against 43.4 tokens per second, because the same llama.cpp software runs underneath Ollama. The difference appears with several users, because llama.cpp batches requests and triples its throughput, while Ollama processes them one after another.

When do I need vLLM?

vLLM is an engine for many concurrent users. It pays off when requests share the same long context, as with RAG (answers based on your own documents). vLLM delivered 240 tokens per second at eight users, against 165 for llama.cpp, and its prefix caching (the shared beginning is processed only once) cut the wait time for shared context by a factor of 19.

What can llama.cpp do that vLLM cannot?

llama.cpp can selectively offload parts of an MoE model (Mixture of Experts, a model made of many sub-networks) to system memory. That let a 35-billion model run for us in 4.5 GB of graphics memory, and a 284-billion model on two consumer cards. llama.cpp also starts in seconds, where vLLM needs minutes on its first start.

Is SGLang worth it for local models?

SGLang is an engine built for loads with shared context. It would not run at all on our model, though: a bug open since July prevented operation with the hybrid model Qwen3.6. Anyone considering SGLang should first check whether their own model is supported.

## Glossary

-   **Token**: The smallest billing unit of a language model; a word is roughly one to three tokens. Providers charge per million tokens, separately for input and output.
-   **Inference**: Running a trained model in production: a request goes in, an answer comes out. Inference is what you buy from a host per token.
-   **Quantisation**: Compressing a model by lowering the numerical precision of its weights (e.g. FP8 or 4-bit). Operating it becomes markedly cheaper; past a certain point, answer quality drops measurably.
-   **MoE**: Mixture of Experts: a model that stores many sub-models ("experts") but computes only a few of them per token. All parameters are stored; only a fraction are used for any given calculation.
-   **Parameters**: The learned numerical values of a language model. Their number describes its size: "8B" means eight billion parameters. Larger models can do more but need more GPU memory.
-   **Batching**: Serving many requests in one pass, so a single read of the model weights from memory feeds many users at once. The more requests share a batch, the cheaper each token gets.
-   **Precision (BF16, FP8)**: How many bits a model uses to store its numbers. Fewer bits save memory and compute but can lower quality.
-   **KV cache**: The working memory in which a model holds the conversation so far. It grows with the text length and with every concurrent user, and often takes up more graphics memory than the model itself.
-   **RAG**: Retrieval-Augmented Generation: before answering, the model receives matching excerpts from your own documents and answers with evidence instead of from memory.
-   **System prompt**: An instruction the operator invisibly places before every request. It sets the role, tone and limits of the model for an application.
-   **Prefix caching**: The serving software stores the already processed beginnings of requests. If the next request starts the same way, only the new part is computed, which cuts waiting time sharply.