> Quelle: https://souverana.ch/en/insights/ollama-llamacpp-vllm/
> Sprache: en

# Ollama, llama.cpp or vLLM: choosing the right engine

Whether Ollama, llama.cpp or vLLM: the choice comes down to two questions. How many people work on the model at the same time, and how much graphics memory do you have relative to the model you want? Three tools, measured on the same hardware.

Technical · Published 14 Aug 2026 · Updated 24 Aug 2026 · Joel Barmettler

## Ollama, llama.cpp or vLLM: which engine do I need?

**The choice follows two questions: how many people work on the model at the same time, and does the model fit entirely into graphics memory? For a single person, Ollama is the most comfortable solution. If graphics memory is tight, llama.cpp serves an entire team and, through offloading, extracts the largest possible model from the hardware. If the model fits comfortably and many people compute on it, vLLM is built for exactly that.**

**In brief**

-   Answer two questions before choosing a tool: how many people work on the model at the same time, and does it fit entirely into your graphics memory? Between the right and the wrong answer lay 0.9 and 62 seconds of wait time for us.
-   Do not run a multi-user service on Ollama, and do not judge your hardware by it. It processed requests one after another, and someone waited 62 seconds.
-   For a single workstation, the opposite holds: there, Ollama delivered the first answer after 0.9 seconds and was the most comfortable choice.
-   If you want the largest possible model on small hardware, the path leads through llama.cpp. Only there do the experts of an MoE? model sit in system memory.
-   Before choosing vLLM or SGLang, check whether your model is supported. Our hybrid model would not start on SGLang at all.

Same model file, same RTX 4090, one user: Ollama delivers 43.8 tokens per second, llama.cpp 43.4. Practically identical. Same file, same card, four users: Ollama falls to 38.7, llama.cpp rises to 125.1.

These two measurements show the first axis of the choice: the number of concurrent users. The second axis is your graphics memory relative to the model you want, and there the two large engines face off. vLLM is built for machines whose graphics memory comfortably fits the model, and turns the surplus into throughput. llama.cpp is built for the opposite case and extracts the largest possible model from tight graphics memory.

The engine, in this context, is the software that loads a model, accepts requests and keeps the graphics card busy. It has no influence on how clever the answers are. We measured three tools on the same hardware, plus one specialist that never made it to the measurement stage.

This is the fourth part of our series on local inference. After the machine, the memory and quantisation, this part turns to the software that serves the model.

Series · Local Inference

1.  1[Self-hosting an LLM: GPT-5 performance for CHF 8’500](/en/insights/llm-selbst-hosten/)
2.  2[VRAM for LLMs: how to size it correctly before you buy](/en/insights/vram-llm/)
3.  3[LLM quantisation: how it works and how to use it well](/en/insights/llm-quantisierung/)
4.  4Ollama, llama.cpp or vLLM: choosing the right engineYou are reading this part
5.  5[Self-hosting an LLM: the real cost and the break-even](/en/insights/llm-selbst-hosten-kosten/)

## For the single workstation: Ollama

Ollama installs itself in minutes, loads models with a single command, and sets up the same engine under the hood as llama.cpp, whose compute kernels it uses. The identical 43.8 against 43.4 tokens per second at one user are the proof: same kernels, same quantisation, same card.

For the person working with it alone, it is even the fastest choice: the first answer arrived after 0.92 seconds, against 3.65 for llama.cpp with our server settings.

The problem begins with the second concurrent user:

At four concurrent users, Ollama even loses a little throughput, and the wait time to the first character jumps from 0.92 to **62 seconds**. That is the pattern of a queue: the fourth user waits until the first three are done. llama.cpp, configured with four slots, pushes the same four requests through the card together.

We also tested the setting the documentation offers for this. With `OLLAMA_NUM_PARALLEL=4`, throughput at four users stayed unchanged (38.6 tokens per second), and the single user became markedly slower (26.6 instead of 43.8). The four slots may have shared the configured context, which would explain the degradation; what we can say with certainty is only this: with the means the documentation offers, the behaviour could not be brought up to the level of llama.cpp.

Ollama is therefore not slow; it is a single-workstation tool. For exactly that workstation, it is good.

## For the small server: llama.cpp

llama.cpp is the engine inside Ollama, without the comfort and without its limits: a single program that loads GGUF files, offers an OpenAI-compatible interface, and batches requests. In our runs, the server was ready around five seconds after starting, which makes switching models and experimenting fluid.

Two capabilities make it the first choice for a small team with its own server.

**It batches.** The 125.1 tokens per second at four users from the chart above are almost three times the single-user figure, on a single card, and the wait time stayed at 8.1 seconds, where Ollama left the same four users waiting 62.

**It extracts the largest model from tight graphics memory.** llama.cpp selectively places the rarely used experts of an MoE model in system memory (`--n-cpu-moe`, covered in detail in Part 2 of this series). That let a 35-billion model run for us in 4.5 GB of graphics memory, and a 284-billion model at 9.6 tokens per second on two consumer cards. vLLM also knows an offload mechanism, but an undifferentiated one: offloading 8 GB of a dense model there cost a factor of 22 in speed. Of the three tools, only llama.cpp offers this workable trade.

The price: you configure it yourself. Context length, parallelism, offload and flash attention sit as flags under your own responsibility, and the defaults are rarely the right ones. The commands our measurements ran with are in the appendix.

## For production: vLLM

vLLM is built for the case where the model fits comfortably into graphics memory and the card never runs empty: many users, long inputs, continuous operation. It runs continuous batching (new requests join in the middle of running computations), reads AWQ and FP8 formats, and manages the KV cache in pages.

On two cards under realistic load (10,000 input tokens, 2,000 output):

At four users, vLLM delivers double the throughput of llama.cpp; at eight, a 46 percent lead remains. The counter-argument: at a single user, the wait for the first character was 3.5 seconds against 0.6 for llama.cpp, and the first start took minutes, because vLLM compiles its kernels, where llama.cpp was ready after five seconds.

vLLM plays its biggest trump card with loads that share a context, meaning RAG and agents with a fixed system prompt. Its prefix caching reads the shared part only once: in our measurement with 8,000 tokens of shared context and eight users, the wait time fell from 17.9 to 0.9 seconds, a factor of 19 from a single switch.

The catch: not every model can play along

Prefix caching requires a pure attention architecture. For hybrid models like our Qwen3.6, vLLM switches the feature off, and the factor of 19 disappears there with no substitute. [Anyone planning RAG↗](/en/insights/souveraene-ki-plattform/) should therefore check, before choosing a model, whether its architecture allows prefix caching. We measured the effect with Apertus-8B, a pure attention model.

## The specialist that never started: SGLang

SGLang specialises in shared contexts and would have been the fourth candidate for agent workloads. On our model, it never got that far: the startup failed on a bug open since July with hybrid models (GitHub issue #30178); the documented workaround fixed the first crash, after which the first request hung in a timeout.

This is not a verdict on SGLang; others report good numbers on pure attention models. It is a verdict on tool maturity with new architectures: the newer the model design, the smaller the selection of engines that actually support it. Check the combination of model and engine before committing to either one.

## The mapping

| Your situation | Tool | The measurement behind it |
| --- | --- | --- |
| One person, own computer | Ollama | first answer after 0.9 s, speed identical to llama.cpp |
| Small team, one server | llama.cpp | triples throughput at four users, starts in 5 s |
| Model larger than graphics memory | llama.cpp | 35 billion parameters in 4.5 GB, 284 billion on two cards |
| Model fits comfortably, many users | vLLM | 240 tokens per second at eight users |
| RAG and agents with fixed context | vLLM with an attention model | wait time cut by a factor of 19 |

This mapping is not a ranking; each of the three tools has the case where it beats the others. In our measurements, the wrong choice cost 62 seconds of wait time, on a card that was capable of one second.

Want engine and model sized for your load before you go into production?

[AI architecture & solution selection](/en/leistungen/ki-architektur/)

## How to get started

1.  01
    
    Check the model with Ollama
    
    For the question of whether the model can handle your tasks, the single workstation is exactly right. This check costs an afternoon and needs no server.
    
2.  02
    
    Switch to llama.cpp for the team test
    
    The same GGUF file keeps running there. Set context length and parallelism deliberately; the start commands from the appendix are a tested starting point.
    
3.  03
    
    Measure vLLM against your load before continuous operation
    
    With the built-in benchmark (vllm bench serve) and your real input length, because without it the question of the sustainable user count cannot be answered.
    
4.  04
    
    Record the engine version
    
    Our first round of measurements accidentally ran on a vLLM installation eight minor versions old. Anyone who measures notes the version first, or ends up measuring the past.
    

What remains is the question of what the whole setup costs and when it pays off. The fifth and final part of the series answers that.

The author

![Portrait of Joel Barmettler](/_astro/joel-barmettler.CGKHGWrV_sJ0IG.webp)

Joel Barmettler

AI Architect · Souverana, Zurich

Joel Barmettler guides Swiss companies from AI strategy to integration: sovereign, confidential and production-ready. He built the Swiss AI Hub as its architect and today co-owns its architecture; he personally leads every Souverana mandate. Mandates from one-person firms to Fortune 500 corporations.

[Book an intro call](https://meet.brevo.com/joel-barmettler/30-minute-meeting) [More about Souverana](/en/) [LinkedIn](https://www.linkedin.com/in/joel-barmettler-b9ab361b7)

## Frequently asked questions

What is better, Ollama or vLLM?

For a single person, Ollama is the most comfortable choice: in our measurement it delivered the first answer after 0.9 seconds. From several concurrent users onward, the service belongs on llama.cpp or vLLM, because Ollama processed requests one after another and left users waiting over 62 seconds.

Is Ollama slower than llama.cpp?

Not with a single user: we measured 43.8 against 43.4 tokens per second, because the same llama.cpp engine runs underneath Ollama. The difference appears with several users, because llama.cpp batches requests and triples its throughput, while Ollama processes them one after another.

When do I need vLLM?

As soon as many users are served at once, or requests share the same long context, as with RAG. vLLM delivered 240 tokens per second at eight users, against 165 for llama.cpp, and its prefix caching cut the wait time for shared context by a factor of 19.

What can llama.cpp do that vLLM cannot?

Selectively offload parts of an MoE model to system memory. That let a 35-billion model run for us in 4.5 GB of graphics memory, and a 284-billion model on two consumer cards. llama.cpp also starts in seconds, where vLLM needs minutes on its first start.

Is SGLang worth it for local models?

It is built as a specialist for loads with shared context, but it would not run at all on our model: a bug open since July prevented operation with the hybrid model Qwen3.6. Anyone considering SGLang should first check whether their own model is supported.

LinkedIn

## Share this article

Ready-formatted graphics and a suggested post for your LinkedIn feed: download, copy, post.

![Grouped bar chart, Qwen3.6-27B Q4\_K\_M on an RTX 4090: at one user, Ollama and llama.cpp deliver 43.8 and 43.4 tokens per second, at four users 38.7 against 125.1. Wait time to the first character: 62 against 8 seconds.](/media/ollama-llamacpp-vllm-en/infografik.png)

[Download infographic (PNG)](/media/ollama-llamacpp-vllm-en/infografik.png)

Suggested post

Are you piloting local AI with Ollama right now? For a single person, that is the right choice.

Once a team accesses it, the picture changes completely, and many pilot projects fail at exactly this point, without anyone seeing the cause.

Measured with the same model file on the same RTX 4090: at one user, Ollama and llama.cpp deliver practically identical 43.8 and 43.4 tokens per second, because Ollama runs on llama.cpp underneath. At four concurrent users, it is 38.7 against 125.1, and the fourth user waits 62 seconds instead of 8 for the first character. Ollama processes requests one after another, llama.cpp batches them.

Our conclusion: anyone who tests team operation with Ollama is not judging their hardware, but a queue. That is how the impression arises that in-house AI is too slow, even though the card is barely loaded.

The article carries all the measurements, the ready-made start commands, and the case where vLLM gets a factor of 19 out of shared context. Link in the comments.

#AI #LLM #Ollama #vLLM