> Source: https://souverana.ch/en/insights/deepseek-inferenz-betrieb/
> Language: en

# How DeepSeek can be so cheap: a look at its operation

DeepSeek runs V4.1 Flash for USD 0.30 per million input tokens (OpenAI's top-tier model costs 2.00) and makes money on it. The article shows how this operation works and how DeepSeek and the US labs differ in philosophy and disclosure. From this follows what you should ask a host before you buy from it.

Analysis · Published 20 Oct 2026 · Updated 20 Oct 2026 · Joel Barmettler

## What sits behind the price per token: how DeepSeek runs inference

**A model such as DeepSeek V4.1 Flash costs USD 0.30 per million input tokens, OpenAI’s GPT-6.1 Sol USD 2.00. DeepSeek still makes money. It takes in about USD 38 of revenue per GPU and hour, against costs of roughly USD 2 to 2.50. DeepSeek disclosed how that works on 20 October 2026.**

**In brief**

-   DeepSeek takes in about USD 38 per GPU and hour. That is a margin of over USD 35 per GPU.
-   DeepSeek and the large US labs set different priorities. DeepSeek optimises for the cost per token. The US labs bet on ever larger models. OpenAI has so far financed this strategy with investor money and, according to media reports, keeps making losses.
-   Two levers explain DeepSeek’s low price: smarter caching and better GPU utilisation.
-   DeepSeek discloses weights, software and operating figures, but not the training data. Open weights alone therefore do not mean that a model is transparent. In this respect Apertus remains ahead.

This article is the evidence behind our calculation for a [public AI in Switzerland](https://souverana.ch/en/insights/ki-infrastruktur-schweiz-kosten/), which rests on DeepSeek’s operating figures. Anyone who doubts that AI can be run this cheaply will find the evidence here.

DeepSeek is a Chinese AI lab that develops language models and releases them free of charge with open weights. It optimises throughout for performance and cost, from the model architecture through TileLang, its own programming language for GPU programs, to the data centre. DeepSeek’s models can either be downloaded and run independently, or obtained directly from DeepSeek via an API. The prices for its inference service are unbeatably low: USD 0.30 per million input tokens for DeepSeek V4.1 Flash against USD 2.00 for OpenAI’s GPT-6.1 Sol (list prices, October 2026).[1–3] The price gap has to do with DeepSeek’s optimised model architecture, but also with its sophisticated way of optimising its model hosting.

This article is the fourth part of our series on buying inference.

**Series · Buying inference**

1.  [What LLMs really cost: the price per token is misleading](https://souverana.ch/en/insights/llm-kosten/)
2.  [Prompt caching: why agents cost more to run in Switzerland](https://souverana.ch/en/insights/prompt-caching/)
3.  [Prompt caching in operation: what a host must set up](https://souverana.ch/en/insights/prompt-caching-betrieb/)
4.  How DeepSeek can be so cheap: a look at its operation (this part)

## Two philosophies: cost per token or size of the model

DeepSeek and the large US labs set different priorities.

DeepSeek optimises for the cost per token. The model computes only a fraction of its parameters per token, a cache delivers results more cheaply for recurring requests, and the servers are tailored to their task. The company optimises at every point for low operating costs.

The large US labs bet on maximum intelligence. They build models that lead in tests, whatever the cost. Their models keep growing, and keep getting more expensive. In 2025, OpenAI named commitments of about USD 1.4 trillion for 30 gigawatts of computing capacity. According to a Reuters report, OpenAI now plans with about USD 600 billion through 2030.[4] Amazon, Microsoft, Alphabet and Meta together plan investments of roughly USD 700 to 725 billion for 2026.[5] OpenAI itself is not profitable. According to media reports based on internal figures, its 2025 revenue was about USD 13 billion, and a loss of about USD 14 billion is expected for 2026.[6] The revenue and loss figures are unaudited and differ by source.

Both strategies have their reasons. The larger model achieves more: in the Artificial Analysis index, GPT-6.1 Sol scores 42 to 52 points depending on reasoning effort, DeepSeek V4.1 Flash 39.[1, 7] A cheaper model is enough for many tasks, but the lead of the strongest model is real, especially for complex tasks such as coding.

## What the labs disclose

Three things can be disclosed about an AI model: the weights, with which you can run it yourself, the training data, with which you can examine it, and the software for running it.

|  | DeepSeek | OpenAI, Anthropic, Google (frontier models) |
| --- | --- | --- |
| Weights | open, MIT licence | closed, only via an API |
| Training data | not open[8] | not open |
| Software for operation | engine DeepInfer open, cache management and load balancing only described | not published |
| Operating figures | hit rate, throughput, revenue per GPU and hour (2025 and 2026)[9] | we know of no comparable publication |

OpenAI (gpt-oss) and Google (Gemma) have released smaller models with open weights. None of these three companies has released a frontier model with open weights.

What sets DeepSeek apart is the disclosure of its operation. The operating figures, the software and the description of cache management make it possible to recalculate a host’s price.[9]

## The path of a request

How DeepSeek drives down the cost per token is best seen along a single request.

At DeepSeek, a request passes through four stations. First, Recipe, DeepSeek’s tokenisation component, translates the text into tokens. Then the Entrypoint, the first server in the sequence, checks whether the prefix of the request is already in the cache. It then passes the request to a load balancer, which distributes it to free servers. Two servers then work on the request in turn: a prefill server reads the whole input at once, and a decode server writes the answer token by token. Images are processed beforehand by dedicated servers that convert them into numbers.[9]

Path of one request

1.  01 **Recipe** Turns the request into tokens and the answer back into JSON
2.  02 **Entrypoint** Looks up the cache, hands the request to the load balancer
3.  03 **Prefill** Reads the input. 2 to 8 GPUs per instance
4.  04 **Decode** Writes the answer. 64 to 128 GPUs per instance

**Vision encoder** Images, one GPU, the cheapest that fits

**Shared cache store (3FS on SSD)** Decode writes, every prefill server reads. Kept for days. Short window (last 128 tokens): RAM of the server that produced it, about 3 minutes

Our own illustration based on the description in DeepSeek's post of 20 October 2026. GPU counts: self-reported by DeepSeek.

DeepSeek runs the model in several instances. An instance is a complete copy of the model, spread over a fixed number of GPUs. Prefill and decode instances differ in size. A prefill instance uses 2 to 8 GPUs, so at most one ordinary server with 8 GPUs. DeepSeek uses the smallest number of GPUs on which the model still fits in GPU memory. A decode instance uses 64 to 128 GPUs, the number that a supernode holds. A supernode is a cluster of many GPUs working together over a very fast network.[9]

The reason for the different sizes lies in the model’s architecture. V4.1 Flash is a mixture-of-experts model. It consists of very many experts, which are specialised computing blocks, and a router selects only a few of them for each token. Yet all experts must sit in GPU memory.

In decode, DeepSeek spreads the experts particularly widely, over all 64 to 128 GPUs of an instance (expert parallelism). Each GPU then holds only a small share of the experts and so has plenty of memory free for the intermediate results of many concurrent requests. Each GPU serves its own requests and sends every token to the GPUs on which the selected experts sit. This distribution pays off most in decode, because memory is the bottleneck there.[9]

## Production figures

> **96 %** of the input comes from the cache on average in production for DeepSeek V4.1 Flash, with inputs averaging 77,000 tokens.
> 
> Source: DeepSeek, V4.1 Flash in large-scale deployment, Zhihu, 20 October 2026 (self-reported)

| Stage | Metric | Value in production |
| --- | --- | --- |
| Prefill | Throughput per GPU, without cache hits | 60,000 tokens/s[9] |
| Prefill | Throughput per GPU, with cache hits | 1.5 M tokens/s |
| Prefill | Mean input length | 77,000 tokens |
| Prefill | Time to first token, mean | 400 ms |
| Prefill | Time to first token, P99 | 1,000 ms |
| Decode | Throughput per GPU | 14,000 tokens/s |
| Decode | Decode speed per user | 230 tokens/s (coding: 300) |
| Cache | Retention after last use | 72 hours |

According to DeepSeek, the servers are planned with headroom for load peaks. The ratio of prefill servers to decode servers is close to 1:1 in production.[9]

One prefill figure can be derived from the other. If 96 per cent of the input comes from the cache, the GPU has to compute only 4 per cent. Dividing 60,000 by 0.04 gives 1.5 million, so the cache makes prefill 25 times faster. In principle, this saving leaves room for a price far below the normal price.

With a mean input of 77,000 tokens, the time to first token is 400 milliseconds on average and 1,000 milliseconds for the slowest 1 per cent of requests (P99). In decode, a GPU delivers 14,000 tokens per second, and each user receives 230.[9] By our calculation, this means that about 60 users share one GPU.

## Why prefill and decode run on separate servers

In prefill, each GPU processes a batch of input tokens per step. At batches of 4,096 tokens it delivers about 66,000 tokens per second, at 32,768 tokens about 87,000. At 65,536 tokens per batch, the latency per step doubles from about 380 to 760 milliseconds, and throughput no longer grows. The GPU is then fully loaded, and more input at once only means a longer wait.[9]

In decode, the curve runs the other way: the more requests a GPU serves at once, the slower each individual user becomes. With 256 concurrent requests per GPU, a GPU delivers about 26,000 tokens per second, but each user gets only about 100. With 8 requests per GPU, each user gets about 325, and the GPU delivers only about 2,600. Offering each user 3.2 times the speed leaves only a tenth of the GPU’s throughput.[9]

**Chart: Prefill: batch size against throughput and latency per step**

*Throughput per GPU (tokens per second) / Latency per step (milliseconds)*

| Category | Throughput per GPU (tokens/s) | Latency per step (ms) |
| --- | --- | --- |
| 4,096 | 66400 | 62 |
| 8,192 | 75900 | 108 |
| 16,384 | 84300 | 194 |
| 32,768 | 86700 | 378 |
| 65,536 | 86800 | 756 |

Source: DeepSeek V4.1 Flash, offline measurement, EP4/DP4, context 131,072, values read from the chart

**Chart: Decode: concurrent requests per GPU against throughput and speed per user**

*Throughput per GPU (tokens per second) / Speed per user (tokens per second)*

| Category | Throughput per GPU (tokens/s) | Speed per user (tokens/s) |
| --- | --- | --- |
| 8 | 2600 | 325 |
| 16 | 5000 | 314 |
| 32 | 9500 | 297 |
| 64 | 15700 | 246 |
| 128 | 22800 | 179 |
| 256 | 26300 | 103 |

Source: DeepSeek V4.1 Flash, offline measurement, EP64/DP64, 3 draft tokens, 75 % acceptance per token, context 131,072, values read from the chart

The curves roughly match production, where a GPU delivers 14,000 tokens per second at 230 per user. At this speed the offline curve gives about 17,000, read between the points for 128 and 64 requests. Production is about a fifth lower.[9]

For purchasing, this means: a host that guarantees a high speed per user runs its GPUs less full and has to pass that on in the price, unless it has enough users to pack many requests onto one GPU. A small host with few customers will therefore struggle to offer high speed and low prices together. That is our reading of the curves; DeepSeek says nothing about it. Anyone who hosts their own model moves along the same curve, but with fewer users, as our calculation of the costs of self-hosting shows.

## The calculation: why USD 38 per GPU and hour is enough

DeepSeek reports revenue of USD 38 per GPU and hour, averaged over its prefill and decode servers and calculated at off-peak prices.[9] That is about CHF 32 (rate of 1 October 2026).[10] DeepSeek’s post does not say what a GPU hour costs the company. We spread the purchase cost of the hardware over four years and arrive at roughly USD 2 to 2.50 per GPU hour, excluding electricity, data centre and staff. Revenue is thus 15 to 19 times the hardware cost. That is not a margin, because the other costs are missing, and revenue at off-peak prices is not the revenue over the whole day. The gap does show, however, that there is enough room to sustain a price such as USD 0.30.

The same operating values underlie our calculation for Switzerland. There we include acquisition, electricity, data centre and staff in full. The result is in the article on the public AI.

## What to take from this for purchasing

DeepSeek’s figures give a benchmark against which a host can be measured. Six questions for any host follow from them. The notes give DeepSeek’s own figures as comparison values.[9]

01 · Cache location

Does the cache sit in the individual servers or in a shared store?

DeepSeek: shared SSD cluster. With a local cache, a hit counts only if the request lands on the same server.

02 · Retention

How long is an entry kept after the last use, and is that period in the contract?

DeepSeek: 3 days for agents, 7 days for chats. From Friday 18:00 to Monday 08:00 is about 62 hours, which is less than 3 days.

03 · Measurement

Does every response report how many input tokens were hits?

DeepSeek reports 96 per cent on average. Without a measurement per request, no price for cache hits can be calculated.

04 · Division of labour

Do prefill and decode run on separate servers, and how long do the slowest 1 per cent of requests have to wait?

DeepSeek: the time to first token is 400 milliseconds on average and 1,000 milliseconds at P99 (the slowest 1 per cent), with inputs of 77,000 tokens on average.

05 · Speed

How many concurrent users per GPU does the price assume, and what speed per user is guaranteed?

Calculated from DeepSeek's figures: about 60 users per GPU at 230 tokens per second per user. More speed per user means fewer tokens per GPU per second.

06 · Spare capacity

How much spare capacity does the host keep, and what happens when a GPU fails?

DeepSeek runs more instances than demand requires and does not shut down any instance while utilisation is above 0.8.

## The technology in detail

The following sections describe how DeepSeek implements the three levers (reusing what has been computed, keeping GPUs busy, securing the cache) technically: the cache, tokenisation, load balancing, failure handling and faster decoding. They are aimed at readers who want to check whether such an operation can be built.

## The cache: days on SSD, minutes in main memory

While reading the input, the model computes intermediate results for every token: the cache. If they are stored, the model does not have to compute them again for the next request with the same beginning. This pays off above all for agents and chats, which resend almost the same text at every step. DeepSeek divides the cache of V4.1 Flash into two parts with different lifetimes.[9]

|  | Block cache | Sliding-window cache (SWA) |
| --- | --- | --- |
| Content | Intermediate results for the whole text so far, in blocks of 128 tokens | Intermediate results for the last 128 tokens (134 slots, 6 spare for speculative decoding), in the first 20 layers |
| Location | Shared SSD cluster (3FS) | Main memory of the server that produced it, one copy only |
| Retention | 3 days (agents), 7 days (chat) | about 180 seconds |
| Access | Every server | The producing server, others over a 400-gigabit network |
| Hit rate | Decides how much is reused | Over 98 per cent, decides whether the last block must be recomputed |

### The block cache on shared storage

DeepSeek places the block cache exclusively on a shared cluster of SSDs. This has three advantages. Every prefill server can access every stored prefix. Storage and compute can be expanded independently of each other. And if a server fails, no cache is lost. According to DeepSeek, compute nodes fail more than ten times as often as storage nodes. The cluster holds about 110 petabytes (110,000 terabytes) and is designed for random access to blocks of about 115 kilobytes. It reads at peaks of about 11 terabytes per second and writes at about 910 gigabytes per second.[9]

The decode server writes the cache, so even text the model itself generated can be reused: in an agent’s next turn, the last answer is part of the input.[9]

Retention depends on the type of traffic. DeepSeek keeps agent traffic for 3 days and chat traffic for 7 days. With 3 days instead of 7, chat traffic would, according to DeepSeek, lose 0.5 to 1 per cent of hit rate. The table in the post gives 72 hours as the general figure.[9]

When a stored prefix is found, loading runs in three stages, and only the first delays the request. First, the Entrypoint looks up the longest stored prefix before passing the request on. Second, a separate process copies the data from the SSD cluster into the server’s main memory. Third, the engine moves it layer by layer into GPU memory while the GPU is already computing. For this, DeepSeek uses GPUDirect RDMA, direct access by the network card to GPU memory. The GPU’s usual copy engine handles small blocks poorly, whereas RDMA saturates the link without occupying the GPU’s compute units.[9]

The cluster protects the data with erasure coding at a ratio of 10 to 2: 2 parity parts are added to every 10 data parts, and the failure of two nodes loses no data. When reading, the system queries 11 nodes at once and reassembles the data from the first 10 replies. On each node, a background process removes the longest-unused entries so that storage stays below a limit.[9]

### The window cache in main memory

The window cache holds the state of the last 128 tokens. It is needed only within a session, for a few minutes, so it stays in the main memory of the producing server. A directory on the Entrypoint servers records which server holds which entry. An entry is stored only at multiples of 128 tokens. Retention is about 180 seconds, and the hit rate is over 98 per cent. If an entry is missing, it is recomputed with bounded replay.[9]

Block cache and window cache together determine the hit. The block cache sets how much is reused, and the window cache sets whether the last block must be recomputed. For an input of length L, at most L minus 134 tokens count as a hit. If the window entry is missing, the hit falls short by one block, that is by 128 tokens.[9]

## Bounded replay: part of the cache is now only approximated

Bounded replay is the most important change to the cache from V4 to V4.1. In V4, the window cache also sat on SSD storage. If an entry was missing, the system recomputed it exactly, and the effort grew with the number of layers times the window size in tokens. The window cache took up almost half of the persistent storage although it was reused for only minutes, while the block cache was reused for days. Storing it on SSD did not pay off, above all for short chats. The alternative considered in the V4 report, not storing the window cache at all and always recomputing exactly, was too expensive in operation.[9]

If a window entry is missing, V4.1 recomputes only the last 128 tokens and restricts the model’s attention to that section. The window cache is therefore only approximated, but always costs the same, independent of text length and number of layers. Exact recomputation would have required 20 layers over the last 2,560 tokens. According to DeepSeek, recomputing only 128 tokens halves the total compute for prefill.[9]

With an approximation, the answer can differ from the answer the model would give without the cache. We do not know how large the difference is, because we know of no measurement. As a buyer, you cannot tell from outside whether your host uses such an approximation. Ask your host whether it restores the cache exactly or only approximately.

## What else goes into running it

### Tokenisation as a security question

Recipe, the tokenisation component, accepts requests in the three common formats (Chat Completions, Responses and Messages) and converts them into a shared conversation format. From this it builds the prompt and the tokens for the given model version. It splits the output into events for answer text, reasoning steps and tool calls and returns them in the respective streaming format. DeepSeek wrote it in Rust and released it as `deepseek-recipe`, so that third-party providers get the same behaviour as the official API.[9]

Behind this is a security problem. Control tokens mark roles, reasoning boundaries and tool calls. If a user types something like `<｜User｜>` into their text field, it must not act as a control token. When building the prompt, Recipe records where control tokens sit and tokenises user text that looks like a control token as ordinary text. In the output, it recognises the end of the reasoning phase and the tool markers by their token ID, so that a model cannot spell them out with normal tokens and trigger them. With agents that have tool access, an injected control token can trigger a tool call.[9]

Because the positions of the control tokens are known, Recipe can split long inputs and tokenise the pieces in between in parallel. This reduces the time for long requests.

### Load balancing: four load balancers for even work

A load balancer distributes work across servers and GPUs. To give all GPUs a similar amount of work so that none waits for the slowest, DeepSeek runs four of them.[9]

| Load balancer | What it balances | How |
| --- | --- | --- |
| Prefill | Uncached tokens and length of the already computed prefix | Cost model for attention, buckets by length (up to 16,000, up to 64,000, up to 256,000, longer), a time budget per step which determines the largest batch that fits |
| Decode | Number of requests and sum of cache lengths | Short requests to GPUs with few requests but a long total cache, long requests to GPUs with a short one |
| Vision encoder | Image parts in progress | Random server, then the GPU with the fewest parts |
| Experts | Load of the popular experts | Replicas of popular experts, regular redistribution |

The prefill load balancer does not need to steer requests to the server with the stored prefix, because the cache is shared. With a local cache, by contrast, sessions have to return to the same server again and again (see part three).

When balancing the experts, DeepSeek measures how often each expert is used. Popular experts receive replicas on other GPUs, and these replicas are redistributed regularly. In prefill, that is 4 redundant experts per GPU, measured over 100 steps and redistributed every 1,000 steps. In decode, it is 1 redundant expert per GPU: 384 experts plus 64 replicas give 7 experts per GPU on 64 GPUs, measured over 1,000 steps and redistributed every 10,000 steps. After this, the slowest GPU takes 1.032 times (2 GPUs), 1.062 times (4 GPUs) and 1.090 times (64 GPUs in decode) as long as the mean. Because every step waits for the slowest GPU, about 3, 6 and 8 per cent of GPU time is idle (our calculation).[9]

### Failures: spare capacity and rerouting

The ratio of prefill servers to decode servers adapts to traffic. DeepSeek scales prefill by the number of uncached tokens and decode by the number of active requests, with upper and lower limits. When a decode server is shut down, it has one minute to finish. Requests still running after that are moved, together with their cache, to other servers.[9]

For failures, DeepSeek relies on spare capacity and rerouting, because a restart takes minutes. Standby servers sit idle, and more instances always run than demand requires. Shutting down is blocked while utilisation is above 0.8. With large expert parallelism, one defective GPU brings down the whole instance, because every GPU holds experts that all the others need. On restart, the platform therefore rebuilds the instance with the remaining healthy GPUs. The P99 value for the time to first token shows the effect of this spare capacity: at DeepSeek it is 1,000 milliseconds, two and a half times the mean of 400 milliseconds.[9] At a host without spare capacity, this value is the first to rise at load peaks. That is our view.

### Faster decoding and enforced formats

With speculative decoding, a small draft model proposes several tokens and the large model verifies them in one step. DeepSeek uses a method called DSpark for this. The draft model proposes 5 tokens, an add-on component estimates for each token how likely the large model is to accept it, and the system verifies only as many tokens as give the best throughput according to the measured curves of the hardware. In production, this outperformed the alternative of always verifying all 5 tokens by 6 per cent.[9]

Each decode step consists of planning, computing, choosing the next token and post-processing. Planning and post-processing run on the CPU, and post-processing needs the result of the token choice. Without overlap, the GPU would sit idle while the CPU plans and post-processes. DeepSeek therefore overlaps the steps: while the GPU computes step N, the CPU plans step N+1 and post-processes step N minus 1.

For formats such as JSON Schema, regular expressions or grammars, separate servers translate the specification in advance into an automaton. At each step, it blocks the tokens that are not allowed. Anyone who has a model fill in forms or database queries in a fixed format relies on exactly this feature.

## What DeepSeek publishes and what it does not

With its post, DeepSeek has released the engine **DeepInfer**. It supports distributed inference, speculative decoding, bounded replay and structured output, and runs on NVIDIA GPUs and Huawei Ascend. Also released are `deepseek-recipe`, TileLang (the GPU programming language released earlier) and several kernel libraries. DeepSeek also intends to add the inference code of future models to DeepInfer.[9]

Cache management and load balancing are not fully open. Both are tied to DeepSeek’s internal infrastructure, and the post only describes how they work.[9] We described cache management and load balancing in part three of this series as the host’s task. A Swiss host that wants to run V4.1 Flash this way gets the engine but has to build or buy the cache layer and the load balancing itself. That is our view.

DeepSeek names three design goals. The system should run well on GPUs from several manufacturers and on common hardware (8-GPU servers for prefill, supernodes with 64 to 128 GPUs for decode), and it should keep the cache for at least 72 hours. Storage costs are factored into the design from the start.[9]

This article assesses the technology. For personal data, the standards of our articles on the revFADP and sovereign AI apply unchanged.

## How to get started

Take the six questions into your next conversation with a host, and ask for the answers in writing. Also measure the cache hit rate of your own agent with the test from the second part of the series. The result shows what share of your input recurs at all.

DeepInfer and expert parallelism on 64 GPUs are built for hosts with many customers. For your own server with one to eight GPUs, the same relationships apply on a smaller scale. The levers there are batch size and the prefix caching of the serving software.

Do you want to know which requirements for cache, speed and spare capacity fit your use case before you choose a host?

[Architecture & Solution Selection](https://souverana.ch/en/leistungen/ki-architektur/)

## Sources

[1] DeepSeek. «DeepSeek V4.1 Flash, large-scale inference deployment in practice». Zhihu. 20 October 2026. Eigenangabe von DeepSeek.

The author

Joel Barmettler

AI Architect · Souverana, Zurich

Joel Barmettler guides Swiss companies from AI strategy to integration: sovereign, confidential and production-ready. He built the Swiss AI Hub as its architect and today co-owns its architecture; he personally leads every Souverana mandate. Mandates from one-person firms to Fortune 500 corporations.

[Book an intro call](https://meet.brevo.com/joel-barmettler/30-minute-meeting) [More about Souverana](https://souverana.ch/en/) [LinkedIn](https://www.linkedin.com/in/joel-barmettler-b9ab361b7)

## Frequently asked questions

How does DeepSeek's approach differ from that of the US labs?

DeepSeek optimises for the cost per token: a model that computes only part of its parameters per token, a cache with a 96 per cent hit rate and separate servers for prefill and decode. The US labs bet on ever larger models and build data centres for them. According to a Reuters report, OpenAI plans with about USD 600 billion for computing capacity through 2030 and, according to media reports, makes losses.

Is DeepSeek more transparent than OpenAI?

On operation, yes; on training data, no. DeepSeek publishes weights, engine and operating figures, but not the training data. In the Foundation Model Transparency Index of Stanford University from December 2025, DeepSeek scores 32 out of 100; the mean across 13 companies is 41.

Why does DeepSeek run prefill and decode on separate servers?

Prefill processes the input, decode generates the answer token by token, and the two tasks load the hardware differently. Prefill is compute-bound and saturates a GPU at batches of about 32,000 tokens. Decode is memory-bound and becomes efficient when many requests run at the same time. DeepSeek runs V4.1 Flash accordingly with 2 to 8 GPUs for prefill and 64 to 128 GPUs for decode (self-reported, 20 October 2026).

How long does DeepSeek keep the cache of V4.1 Flash?

According to DeepSeek (post of 20 October 2026), the cache is kept for 3 days for agent traffic and 7 days for chat traffic, counted from the last use. It sits on a shared SSD cluster of about 110 petabytes (110,000 terabytes). A second, short cache (the window cache) for the last 128 tokens stays in main memory for only about 180 seconds.

Is the software behind DeepSeek's operation open?

In part. DeepSeek has released the inference engine DeepInfer (runs on NVIDIA GPUs and Huawei Ascend), the tokenisation component deepseek-recipe and the GPU programming language TileLang. Cache management and load balancing are tied to internal infrastructure and not fully open. The post describes how cache management and load balancing work.

What does a cache hit rate of 96 per cent mean?

Of 100 input tokens, the GPU has to compute only 4 from scratch, and the cache supplies the other 96. For DeepSeek V4.1 Flash in production, with inputs of 77,000 tokens on average, a GPU processes 1.5 million instead of 60,000 tokens per second in prefill. The figure is self-reported and not independently verified.

Is this architecture worthwhile for a single company?

As a rule, no. Decode on 64 to 128 GPUs and an SSD cluster in the petabyte range are built for hosts with many customers. For your own server with one to eight GPUs, the same relationships apply on a smaller scale: batch size and the prefix caching of the serving software are the levers there.

## Glossary

-   **GPU**: Graphics processing unit. Language models run almost exclusively on GPUs because they execute very many computations in parallel.
-   **Token**: The smallest billing unit of a language model; a word is roughly one to three tokens. Providers charge per million tokens, separately for input and output.
-   **Open weight**: Models whose weights are publicly available and may be self-hosted (e.g. Apertus, DeepSeek, Llama). A precondition for sovereign operation without vendor lock-in.
-   **API**: A programming interface through which software addresses a language model directly, without a chat interface. Usually billed per token.
-   **Inference**: Running a trained model in production: a request goes in, an answer comes out. Inference is what you buy from a host per token.
-   **Parameters**: The learned numerical values of a language model. Their number describes its size: "8B" means eight billion parameters. Larger models can do more but need more GPU memory.
-   **MIT licence**: A very permissive open-source licence: software or model weights may be used, modified and redistributed freely, including commercially.
-   **Load balancer**: A component that distributes requests over the available GPUs and machines so that none is overloaded and none has to wait for the slowest.
-   **Cache hit**: A hit means the start of a request is already in the cache and does not have to be computed again. The hit rate is the share of the input that comes from the cache.
-   **Prefill**: The first phase of an answer: the model reads the whole input in one pass and builds its working memory (the KV cache). It is compute-heavy, and it is the part prompt caching saves.
-   **Decode**: The second phase of an answer: the model writes its output one token at a time, each depending on the previous one. Speed here is limited by how fast data moves through memory, not by compute.
-   **MoE**: Mixture of Experts: a model that stores many sub-models ("experts") but computes only a few of them per token. All parameters are stored; only a fraction are used for any given calculation.
-   **Expert parallelism**: A mixture-of-experts model consists of many specialised blocks, the experts. With expert parallelism they are spread over several GPUs: the more GPUs, the fewer experts per GPU and the more memory for concurrent requests.
-   **Time to first token**: The wait between sending a request and the first word of the answer. For long inputs it is dominated by the prefill, so prompt caching shortens it directly.
-   **P99**: The value that 99 of 100 requests stay below. It shows the slow outliers that an average hides.
-   **GPU hour**: One hour of compute on one graphics processor. The usual measure of compute effort when training large models.
-   **Benchmark**: A standardised test with fixed tasks used to compare models. A good benchmark result says little by itself about your own task.
-   **Window cache (SWA)**: The part of the KV cache that covers only the last 128 tokens (sliding window attention). It stays in main memory for minutes only, because it is needed only within a session.
-   **Speculative decoding**: A small draft model proposes several tokens at once, the large model checks them in a single step and keeps the correct ones. Each compute step then often yields several tokens, without changing the answer.
-   **SSD**: Fast flash storage without moving parts. Faster and more expensive than hard disks, but far cheaper than main memory.
-   **RDMA**: Direct memory access across the network: data moves from one machine into the memory of another, or of a GPU, without burdening the processor.
-   **Erasure coding**: A method that spreads data across several storage devices with extra information, so that it can be fully restored after some devices fail. With 10+2, two failures are survived.
-   **Bounded replay**: If a window cache entry is lost, the system recomputes only the last 128 tokens approximately instead of rebuilding it exactly. The effort stays the same regardless of text length.
-   **Prompt**: The input to a language model: question, instruction and any text sent along. What is in the prompt determines the answer and the cost.
-   **Chain of thought**: The intermediate steps a model writes before its answer. They often improve the answer and are billed as reasoning tokens.
-   **Personal data**: Any information relating to an identified or identifiable person, so customer records, HR files or even an email address. As soon as such data is involved, the revFADP applies.
-   **revFADP (revDSG)**: Switzerland's revised Federal Act on Data Protection, in force since 1 September 2023. It governs how companies may process personal data, and applies to AI use exactly as it does to any other processing.
-   **Prefix caching**: The serving software stores the already processed beginnings of requests. If the next request starts the same way, only the new part is computed, which cuts waiting time sharply.