> Quelle: https://souverana.ch/en/insights/prompt-caching-betrieb/
> Sprache: en

# Prompt caching in operation: what a host must set up

Whether prompt caching works at a host depends on its technology: where the cache lives, how long it keeps entries and whether the host measures hits. This article shows the building blocks so that you can ask a provider the right questions, and explains why the Swiss model Apertus 70B needs about 30 times as much cache memory than DeepSeek V4.

Technical · Published 6 Oct 2026 · Updated 6 Oct 2026 · Joel Barmettler

## What decides whether a cache works at a host

**This article describes the building blocks and what a host must set up so that the cache pays off even for agents that pause. It builds on the article Prompt caching: why agents cost more to run in Switzerland.**

**In brief**

-   DeepSeek keeps the cache on disks and makes the entries small through the architecture of the model. It does not guarantee a hit rate.
-   The serving software vLLM keeps the cache in GPU? memory. On a single machine this works well, but with pauses and many users the cache can be evicted.
-   To keep entries longer, a host offloads the cache to main memory or SSD. This pays off only when the caches no longer fit into GPU memory.
-   A plugin alone is not enough. The host must also ensure SSD storage, request routing, a guaranteed retention time and a measurement of hits.

This is the third part of our series on buying inference?. It shows what sits behind a working cache on the host’s side.

Series · Buying inference

1.  1[What LLMs really cost: the price per token is misleading](/en/insights/llm-kosten/)
2.  2[Prompt caching: why agents cost more to run in Switzerland](/en/insights/prompt-caching/)
3.  3Prompt caching in operation: what a host must set upYou are reading this part

## How DeepSeek runs the cache

DeepSeek keeps the cache on disks. The API? documentation calls this “Context Caching on Disk”, and it is active by default for all users. A hit occurs only if the start of the request matches a stored unit completely. DeepSeek does not guarantee a hit rate. According to the documentation, the system works on a best-effort basis. DeepSeek deletes unused entries after a few hours to a few days.

DeepSeek published in 2025 how often the cache is hit in practice, for its system at the time. For V4 we know of no measurement.

Key figure

56.3 %

of DeepSeek's input tokens (342 of 608 billion) were hits in the disk cache in a 24-hour window in February 2025.

Source: DeepSeek, Inference System Overview, Open Source Week, February 2025

A disk works as a cache only if the entries are small. Then they are cheap to keep for a long time and quick to load again. DeepSeek shrinks them through the architecture of the model. According to the technical report on V4 (arXiv 2606.19348), V4-Pro needs about a tenth of the KV cache? of the previous version at one million tokens? of context. The vLLM blog of 24 April 2026 calculates about 10 gigabytes per million tokens for V4. The savings come from Compressed Sparse Attention and Heavily Compressed Attention, two techniques in the model that need less cache memory per token.

## The cache in GPU memory

Serving software such as vLLM, a widely used open-source? engine for language models, ships with prefix caching?, which is prompt caching? on your own server. Prefix means start: only a start that matches token for token counts as a hit. On a single machine it works. [In our own measurement↗](/en/insights/ollama-llamacpp-vllm/) with Apertus-8B, 8,000 tokens of shared context and eight users, time to first token? fell from 17.9 to 0.9 seconds, at a 76.7 per cent hit rate. All eight users send requests at the same moment there, so the cache stays warm.

An agent works in bursts, with pauses in between. vLLM keeps the cache in GPU memory and, when space runs out, deletes the block that has gone unused longest. On a busy server, an agent’s cache can therefore be evicted after a pause of a few minutes. We have not measured this. The KV cache [can take up more GPU memory than the model itself↗](/en/insights/vram-llm/), which is why space is tight.

## Offloading: the offloading connector and LMCache

A host that wants to keep cache entries longer moves them out of GPU memory into larger and slower storage. There are two ways to do this, which hosts often describe as a “plugin” (an add-on module).

The first is the offloading connector built into vLLM. Setting it up is a single switch: from vLLM 0.14, `--kv_offloading_backend native --kv_offloading_size <GB>` is enough, and the cache moves into the server’s main memory. vLLM itself quantifies the effect: a time to first token 2 to 22 times lower (Llama-3.1-8B on an H100 GPU, depending on input length) and up to 9 times more throughput at a high hit rate (10,000 requests of 512 tokens each).

The second is LMCache, an open-source add-on to vLLM. It moves the cache out of vLLM into main memory, onto disk or into remote storage, and can share it between machines. The authors report up to 15 times more throughput (arXiv 2510.09665). When it pays off is shown by a test run by the LMCache team itself, with 739 real Claude Code sessions on two AMD MI300X cards. With 8 users and 32,000 tokens of context, running without LMCache, on the GPU-memory cache alone, came out ahead, with 52 against 25 completed requests. With 32 users and 100,000 tokens of context, the result reversed. LMCache brought a mean time to first token 3.0 times lower and 2.3 times more requests. When all caches fitted into GPU memory, LMCache cost 10 to 17 per cent of throughput.

Is the plugin enough?

For a host, a plugin is enough if the conversations together need more space than GPU memory offers, a session stays on the same machine and the host measures the hit rate. It is not enough if several machines run without deliberate request routing, no retention time is fixed or the host does not report hits. Before and after switching it on, a host should measure hit rate and wait time under load, with the real pauses of its customers. As a customer, you can check the result with the test in the article Prompt caching.

A plugin supplies only the storage. For the cache to work with agents, the host must ensure four things:

-   SSD storage for hundreds of conversations, when main memory is not enough,
-   request routing that brings a session back to the machine holding its cache, or a shared store across all machines,
-   a retention time that the host guarantees,
-   a measurement that reports per request how many tokens were hits, so that a price can be calculated for them at all.

DeepSeek has solved this for its own model. A host must solve it anew for every model. A cache price presupposes that the hit rate can be measured and maintained reliably.

## Prefill and decode on separate machines

Large operators also separate the two phases physically. One group of machines reads inputs (prefill?), another writes answers (decode?), and in between the KV cache travels over a fast network. DeepSeek ran its system this way in February 2025, with a cluster of four nodes for prefill and one of 18 for decode. A single huge input then no longer slows down the answers of the other users.

How far this can go is shown by the paper “Prefill-as-a-Service” (Qin et al., arXiv 2604.15039). It proposes processing long inputs in dedicated, compute-heavy data centres and sending the KV cache over ordinary Ethernet to the decode machines. The authors tested this with an internal model of one trillion parameters?. Throughput rose by 54 per cent, and the wait for the first token (P90, the value that 90 per cent of requests beat) fell by 64 per cent. According to the authors, this works because models that need less cache memory per token shrink the KV cache sharply. The smaller the cache, the further it can be moved.

Moonshot AI (Kimi) also puts the cache at the centre of its Mooncake platform. It uses idle main memory and SSD space on its own machines. In production, according to the authors, the system processes 75 per cent more requests (Mooncake paper, arXiv 2407.00079).

## The operator’s dials

An operator turns five dials at once. Throughput is the total output of the server in output tokens per second across all users. Output speed is the writing pace for a single user. Memory is the memory in use. Queue delay is the wait in the queue before the request is processed at all. Time to first token is that wait plus the time for the prefill, that is, the time until the first word. The two are linked: a long prefill lengthens the wait of the next requests. Every decision improves one dial and worsens another. Larger batches raise throughput but worsen everything the individual user feels. The matrix shows three typical decisions.

| Decision | Throughput | Output speed | Memory | Queue delay | Time to first token |
| --- | --- | --- | --- | --- | --- |
| More users per batch | better more tokens per hour | worse every user slower | worse more KV cache | worse waiting for a slot | worse first word later |
| Higher cache hit rate | better less prefill work | unchanged decode unchanged | worse cache has to be kept | better less load before the start | better prefill partly skipped |
| Separate prefill and decode | better scale both sides alone | better big inputs no longer stall | worse KV cache crosses the network | unchanged separate queues | better prefill machines only for inputs |

Qualitative assessment based on the vLLM documentation and the Mooncake and Prefill-as-a-Service papers. Not a measurement.

With caching, the picture looks different. Three dials get better, output speed stays the same, and the operator pays with memory. That is what this article is about: the memory, the bottleneck.

## Why the model maker has the advantage

A model’s maker knows it down to the detail and writes custom kernels for it, that is, compute programs that fuse several steps into one and so move less data back and forth between memory and compute units. A third party must first find these bottlenecks and retune them for every card generation.

In the price without cache hits, little of this is visible. We compare in francs, with DeepSeek’s standard tariff and at the ECB rate of 1 October 2026 (1 US dollar = 0.8353 francs), because this tariff applies for almost 80 per cent of the week. Phoeniqs runs DeepSeek V4 Flash and charges 0.1462 Swiss francs per million input tokens and 0.3925 francs per million output tokens. DeepSeek itself charges 0.125 and 0.501 francs. Input is thus just over 15 per cent more expensive at Phoeniqs, output just over 20 per cent cheaper. The gap opens only when inputs recur.

## A rented GPU instead of a price per token

Anyone who rents a GPU and does not buy tokens becomes the operator. The provider earns the same whether the card does useful work or reads the same 20,000 tokens for the hundredth time. Everything in this article that applies to the host then applies to the renter: they are responsible for utilisation, memory and the cache. Every cache hit spares the card prefill work and creates time for other requests. From each rented hour, the renter gets more answers.

We measured the effect. Apertus-8B ran on one card with 8,000 tokens of shared context and eight users. With prefix caching the server managed 285.2 tokens per second, without it 122.1. That is 2.3 times more throughput on the same hardware, and time to first token fell from 17.9 to 0.9 seconds.

How large the effect is depends on the share of repetition in the load. With short inputs and long outputs, caching brings little. With agents and with queries over long shared documents, most of the input is repetition.

## Apertus: why its cache is large

How much memory a conversation takes in the KV cache depends on the architecture of the model. Apertus 70B is a dense model with 70 billion parameters, in which every token computes with all parameters. It has 80 layers and 8 KV heads, with the same basic structure as Llama 3. For each token it stores keys and values of all layers, which comes to about 320 kilobytes (BF16? number format, calculated with the Apertus 1.0 configuration, which matches the 1.5 model card). A conversation of 20,000 tokens thus takes about 6.6 GB. At the maximum context length of 262,144 tokens it is about 86 GB, more than a card with 80 GB holds.

According to vLLM, DeepSeek V4 comes to about 10 kilobytes per token, because its architecture deliberately shrinks the cache. Apertus therefore needs about 30 times as much. With an FP8 KV cache (8 bits instead of 16 bits per number) the figure for Apertus halves. The gap remains large.

This memory costs money. About 150 conversations with Apertus 70B fit into a terabyte of main memory (20,000 tokens, not counting the model weights). With DeepSeek V4 it is about 5,000. The fewer conversations fit into fast memory, the more often a host must fall back to SSD or let the cache expire.

Then there is the price. Phoeniqs charges 0.6195 francs per million input tokens for Apertus 1.5 70B and 2.2201 for output, which is 4.2 times and 5.7 times the price of DeepSeek V4 Flash at Phoeniqs (0.1462 and 0.3925). DeepSeek V4 Flash is a mixture-of-experts model? (it consists of many sub-networks, of which only a few compute per token) with 284 billion parameters, of which 13 billion compute per token. The gap is widest for repeated input: a cache hit costs 0.0025 francs per million tokens at DeepSeek, while Phoeniqs charges a flat 0.6195 francs for Apertus 70B, which is 247 times as much (124 times at peak hours). At 96.5 per cent hits, as in our coding session, a million input tokens cost 0.0068 francs on average at DeepSeek, and Apertus at Phoeniqs is then about 91 times more expensive. At the 86 per cent from our own DeepSeek bill it is about 31 times. The coding session of 50 rounds would cost 7.98 francs with Apertus, 71 times DeepSeek, and does not fit at all, because the context window? ends at 262,144 tokens. We do not compare the quality of the models here.

Architectures that shrink the cache exist. DeepSeek V4 shows one, and according to vLLM prefix caching can still be used with it. Hybrid models that mix different attention techniques behave differently, as our test series shows. They save KV memory, but vLLM switches prefix caching off for them. The Apertus team should test every new architecture against the inference software before training begins.

## What we demand

Three audiences, three concrete demands.

**To the hosts.** Put the price for cache hits and the retention time into the price list. Report in every response how many tokens were hits, otherwise no price can be calculated for them. Phoeniqs already runs DeepSeek V4 Flash, and its price without hits is in the range of the maker’s. The cache price is missing.

**To the Apertus team.** Set cost per token, active parameters (those that compute per token) and the size of the KV cache per token as targets of the next generation, on a par with quality. Test the architecture in vLLM with prefix caching and offloading before training starts.

**To policymakers and public administration.** Whoever procures or funds AI as basic infrastructure should write cache price and retention time into the tender as a criterion. Whoever funds the research on Apertus should also co-fund the software for operation with many users: cache storage, routing of requests, utilisation. This software should be open, so that all Swiss hosts can use it.

Do you want to know how a cache would have to be set up at your host or on your own server for your agent to benefit?

[Architecture & Solution Selection](/en/leistungen/ki-architektur/)

The author

![Portrait of Joel Barmettler](/_astro/joel-barmettler.CGKHGWrV_sJ0IG.webp)

Joel Barmettler

AI Architect · Souverana, Zurich

Joel Barmettler guides Swiss companies from AI strategy to integration: sovereign, confidential and production-ready. He built the Swiss AI Hub as its architect and today co-owns its architecture; he personally leads every Souverana mandate. Mandates from one-person firms to Fortune 500 corporations.

[Book an intro call](https://meet.brevo.com/joel-barmettler/30-minute-meeting) [More about Souverana](/en/) [LinkedIn](https://www.linkedin.com/in/joel-barmettler-b9ab361b7)

## Frequently asked questions

Is prefix caching worthwhile if I rent a GPU instead of paying per token?

Yes, if the load contains a lot of repetition. Anyone who rents a GPU is the operator: the provider earns the same whether the card works efficiently or not, so efficiency is the renter's concern. Cache hits spare the card from reading the input again (prefill) and leave room for other requests. In our measurement with Apertus-8B on one card (8,000 tokens of shared context, eight users, vLLM as the serving software), throughput rose from 122.1 to 285.2 tokens per second.

Is it enough for a host to switch on prefix caching in vLLM?

Not for agents with pauses. The serving software vLLM keeps the cache in GPU memory and, when space runs out, evicts the block that has gone unused longest. On a busy server, a customer's cache can be evicted after a pause of minutes (not measured). The customer cannot influence this. To hold entries longer, the host needs more memory, for example through offloading.

Why does Apertus 70B take more cache memory than DeepSeek V4?

Apertus 70B stores about 320 kilobytes per token in the KV cache, calculated with the Apertus 1.0 configuration (80 layers, 8 KV heads, head dimension 128, BF16 number format). According to the vLLM blog of 24 April 2026, DeepSeek V4 comes to about 10 kilobytes per token, because its architecture shrinks the cache. A conversation therefore occupies about 30 times as much memory with Apertus.

LinkedIn

## Share this article

Ready-formatted graphics and a suggested post for your LinkedIn feed: download, copy, post.

![Bar chart: completed requests with LMCache relative to running with the GPU-memory cache alone. 8 users, 32,000 tokens of context: 0.48x (25 against 52). 32 users, 100,000 tokens: 2.3x. Test by the LMCache team, 739 Claude Code sessions.](/media/prompt-caching-betrieb-en/infografik.png)

[Download infographic (PNG)](/media/prompt-caching-betrieb-en/infografik.png)

Suggested post

Do you run your own LLM server, or are you choosing a host that promises prompt caching? Then it is worth asking where the cache lives: in the GPU memory of a card, it is evicted as soon as other users need the space.

The LMCache team's test with 739 real Claude Code sessions on two AMD MI300X cards shows when an additional cache in main memory pays off. With 8 users and 32,000 tokens of context, running on the GPU-memory cache alone came out ahead (52 against 25 completed requests). With 32 users and 100,000 tokens of context, LMCache brought a mean time to first token 3.0 times lower and 2.3 times more requests. When all caches fitted into GPU memory, LMCache cost 10 to 17 per cent of throughput. We conclude that offloading is a question of utilisation and should be measured before it is switched on.

In the article: how DeepSeek runs its cache on disks, what a host must ensure besides the plugin, why the KV cache of Apertus 70B is about 30 times larger than that of DeepSeek V4, and what we demand of hosts and of the Apertus team. Link in the comments.

#AI #LLM #Inference #Switzerland #Apertus