Self-hosting an LLM: GPT-5 performance for CHF 8’500

What can a self-hosted LLM actually do?

A self-hosted LLM, meaning a language model that runs on your own hardware in your own business, is enough for twenty to thirty employees. On a machine with two graphics cards costing around CHF 8’500, a freely downloadable model runs at the level of GPT-5.1, a model from OpenAI. It writes letters, searches documents and translates. Demanding coding work and long task chains overwhelm the machine.

In brief

• Start with the tasks and only then turn to the hardware. On your own hardware, tasks with a clear frame and a checkable result run well, for example extracting information from documents, sorting incoming items and translating. Demanding coding work stays with a provider for now, as do tasks where the AI operates many tools one after another on its own.
• What matters is the number of concurrent requests, meaning the questions the model works on at the same moment. If twenty to thirty employees all work at once, that is three to five requests. The machine copes well with eight concurrent requests.
• Graphics memory decides which model fits at all. Speed depends on how many parameters?ParametersThe learned numerical values of a language model. Their number describes its size: "8B" means eight billion parameters. Larger models can do more but need more GPU memory., the model’s computational values, actually compute per word. A model with 35 billion parameters that uses only three billion per word answered twice as fast for us as a model with 27 billion parameters that uses all of them.
• If a task can wait, larger models become an option. They no longer fit entirely into graphics memory and therefore run slowly, but they suit work overnight, such as checking a hundred contracts. That is the cheapest route to higher quality.
• Do not expect savings. The reason for your own hardware is control over the data.

In our consulting engagements, law firms, medical practices and trust offices now regularly ask whether they should run a language model in their own business. The data of these businesses is especially sensitive, often subject to professional secrecy?Professional and official secrecyThe criminally protected duty of confidentiality of authorities (official secrecy, Art. 320 of the Swiss Criminal Code) and a list of professions named in the law (Art. 321), such as lawyers, doctors and pharmacists. It applies independently of data protection law and reaches further. Banking secrecy sits separately in the Banking Act.. A contract with a provider resolves the problem legally↗Law & Regulation · Part 3Professional secrecy and AI: what may go to the cloud'That does not work for us' is often the answer to AI in law firms and medical practices. The criminal-law article on professional secrecy (Art. 321 of the Swiss Criminal Code) is drawn more narrowly, and most of daily work falls outside it entirely.Read article, yet for some records management simply wants the files never to leave the business.

When asked “which model should we take?”, we answer with two questions of our own: Which tasks should the AI take on in your own business? And how many employees work with it at the same time? The model class follows from the answers, and the machine follows from the model class.

We walked this path ourselves and measured for three weeks. What follows are our own figures from August 2026.

This article opens our five-part series on local inference?InferenceRunning a trained model in production: a request goes in, an answer comes out. Inference is what you buy from a host per token., meaning the operation of language models on your own hardware. It tells what we built and measured; the technical parts that follow show how to size it yourself.

Tasks first, hardware second

On your own hardware, the tasks that run reliably are those with a clear frame and a checkable result. For a law firm or a practice, that is most of what AI is meant to do day to day. Sort your tasks into two lists.

The model also works well in German. We tested it with knowledge questions in multiple-choice format that had been translated into German. On one test, translated by professional translators, the model we run answered 82 percent correctly. A second test of the same kind, machine-translated, produced 88 percent. Both values are high enough that German did not stand out to us as a bottleneck.

So start by writing down ten to twenty real cases from your day to day and assign each to one of the two lists. If most fall into list 1, reading on is worthwhile. If they fall into list 2, save yourself the hardware.

Twenty employees are not twenty requests

The second question is almost always framed wrongly. “Is this enough for 200 employees?” cannot be answered. A machine does not serve a headcount, it processes concurrent requests.

Someone who reads an answer, checks it and types the next question makes about one request every two minutes. The machine needs around 20 seconds for each, so it is busy for one sixth of the time per active person. If 40 of 200 people are active around lunchtime, around seven requests run concurrently on average. If twenty to thirty employees all work at once, it is three to five.

Work out your own number before reading on. Your choice of graphics cards depends on it later, and we return to it when we get to the measurements.

The machine that follows from it

A language model has to fit entirely into the memory of the graphics card, or it becomes very slow. Graphics memory is therefore what matters when self-hosting, and the price per gigabyte differs greatly. An RTX 4090 from the gaming segment offers 24 GB for around CHF 1’700. Professional cards offer 96 GB in a single slot and cost several times as much per gigabyte.

We therefore take two of these gaming cards. Together they offer 48 GB, and the model can be split across both. The rest of the parts list follows from that: a motherboard with two long slots, a power supply that can power both cards, and plenty of system memory, whose purpose becomes clear further down.

Component	Quantity	CHF
ASUS TUF Gaming RTX 4090, 24 GB	2	approx. 1’700 each
AMD Ryzen 9 7950X	1	489
ASUS ROG Crosshair X670E Hero	1	475
Kingston FURY Beast DDR5-6000, 2 x 32 GB	2	909 each
Crucial T700, 4 TB	2	520 each
Corsair HX1500i, 1500 W	1	719
Case, cooling, small parts		approx. 460
Total		approx. 8’450

The whole thing is a desktop machine with two graphics cards, available from any Swiss retailer. It stands under a desk and plugs into a normal wall socket. The result is 48 GB of graphics memory (two cards of 24 GB each) and 128 GB of system memory.

That settles the hardware. What is missing is the model.

The search for a model starts with a leaderboard

Let us stay with the machine we have just put together, with its 48 GB of graphics memory. Which model runs on it? Hugging Face?Hugging FaceThe largest platform on which open AI models and datasets are published and downloaded. hosts tens of thousands of open-weight models?Open weightModels whose weights are publicly available and may be self-hosted (e.g. Apertus, DeepSeek, Llama). A precondition for sovereign operation without vendor lock-in., many of them variants of variants. We started with Artificial Analysis. The site runs models through the same set of tests and rolls the result up into an Intelligence Index, a number on a scale up to 100. For our case, it has a dedicated view of small open models, exactly the field that fits a machine like ours.

Before a list like that is useful, you need to be able to read the names. They look confusing and contain exactly the two numbers that matter.

Model name, decoded

Qwen 3.6 -35B -A3B

Qwen

The provider. Alibaba, in this case. It says nothing about capability, but a lot about maintenance, licence and how many sizes exist within the same family.

3.6

The generation. A higher number beats a lower one from the same family almost always, even at the same size.

-35B

Total parameters. 35 billion, that is 35,000 million. The B stands for billion. This number determines how much graphics memory you need, because that much has to be loaded.

-A3B

Active parameters. Three billion. This number determines the speed, because only that much is computed per word. If the figure is missing, the model computes with everything it has.

The two red numbers do different things. The first costs you memory, the second buys you speed.

If the second number block is missing, it is a dense model: it computes with every single parameter for every single word. If an A appears there, it is a mixture-of-experts model?MoEMixture of Experts: a model that stores many sub-models ("experts") but computes only a few of them per token. All parameters are stored; only a fraction are used for any given calculation., MoE for short. It consists of many smaller sub-models. An upstream component decides for each word which few of them work, and the rest stay idle. The sub-models are called experts because each is particularly well suited to different tasks. This design is the reason a larger model can answer faster than a smaller one.

Three classes, and why only one is in play

Sorted by this pattern, the open models fall into three classes as of August 2026.

Small, up to around four billion parameters. These models run on a laptop, the smallest on a phone. They summarise texts, sort incoming items and locate passages. For anything that has to get several steps right in sequence, they are too weak.

Medium, roughly nine to forty billion parameters. Qwen, Gemma, Mistral and Apertus sit in this class. Quantised, a model of this size fits into 24 to 48 GB of graphics memory.

The open flagship, 300 billion to 2.8 trillion parameters. DeepSeek, Kimi, GLM. These models compete against the closed flagship models and come close. For that, they need dozens of data-centre cards and acquisition costs running into the millions.

From the medium class, we took Qwen, for three reasons. At the time of measurement, it was the highest-ranked open model that fits entirely into 48 GB. The same family exists in many sizes and as both a dense and an MoE variant, which is what makes comparisons possible in the first place. And ready-made 4-bit versions exist from several independent providers.

That is why the table lists only this one model from the medium class.

In the index, the gap between these worlds looks like this:

Model	Index
Claude Opus 5 (proprietary, flagship)	63.1
Kimi K3, 2.8 trillion parameters (open flagship)	59.7
Qwen3.6-27B (fits entirely in 48 GB)	37.7
GPT-5.1, for comparison	37.5
GPT-4o (March 2025)	12.3

Source: Artificial Analysis, retrieved 12 August 2026.

The last three rows are the good news. The model that fits on a machine under the desk is level with GPT-5.1 and far above GPT-4o, the model most people first came to know ChatGPT by.

A leaderboard has its limits, though. Anyone who knows an exam can train for it, and an index says nothing about your documents, your language and your tools. So we downloaded the candidates and remeasured them ourselves.

What size buys you

We measured with tests whose result can be counted: arithmetic problems in text form, small coding tasks with an executable solution, and German knowledge questions in multiple-choice format. The figure given is the share of correct answers in each case. The first four rows come from the same model family and ran without quantisation?QuantisationCompressing a model by lowering the numerical precision of its weights (e.g. FP8 or 4-bit). Operating it becomes markedly cheaper; past a certain point, answer quality drops measurably.. The fifth is the model we actually run on this machine:

Size	Arithmetic	Coding	German knowledge
0.8B	38 %	18 %	41 %
2B	57 %	22 %	55 %
4B	73 %	39 %	66 %
9B	87 %	55 %	74 %
27B (4-bit, newer generation)	66 %	74 %	82 %

Below nine billion parameters, a model is not fit for professional work. Four billion does not solve even two out of five coding tasks and does not know a third of the German knowledge questions. For summarising and sorting, that is enough. As soon as someone relies on the result, it is not enough.

It becomes usable in the class around thirty billion. The 27-billion model solves three out of four coding tasks, where the 9-billion model manages just half. This is exactly the class that fits on 48 GB of graphics memory. On arithmetic, it is the odd one out at 66 against 87 percent. Three reasons account for that, and none of them is reassuring: the model belongs to a different generation than the models in the rows above it, it ran quantised to 4 bit rather than unquantised, and it thinks before answering, for which we gave it no extra token?TokenThe smallest billing unit of a language model; a word is roughly one to three tokens. Providers charge per million tokens, separately for input and output. budget. This row deserves a remeasurement.

For the hardest tasks, you need more than a hundred billion parameters. And that brings us to the real issue, because by then the hardware has long since been bought.

What determines speed: active parameters and bandwidth

From here, the question changes. The 48 GB decide which model fits at all. How fast it then answers is decided by two other quantities: how many parameters actually compute per word, and how fast the card reads its memory.

Two models on the same machine, under the same load, with eight concurrent users, illustrate the first quantity. Only the model changes.

Configuration	Tokens per second and person	Wait time to first character
Qwen3.6-35B (MoE), entirely on the cards	104.6	1.6 s
Qwen3.6-27B, entirely on the cards	51.6	5.8 s

The larger model is twice as fast and keeps its users waiting a quarter as long. Qwen3.6-35B computes with three of its 35 billion parameters per word, the 27-billion model with all of them. For memory requirements, the total count matters; for speed, the active one does. This is exactly why we recommend MoE models in this class: they occupy the memory of a large model and compute at the speed of a small one.

The second quantity is memory bandwidth, and it explains a surprise in hardware purchasing. For every single word, the active weights have to pass through the card’s memory once. An RTX 4090 reads around 1,000 GB per second for that. Nvidia’s L4, a widely used data-centre card with the same amount of memory, reaches 300 GB per second (manufacturer figures). For inference, that means: the more expensive server card answers more slowly than the gaming card. It is designed for 72 watts, half-height and continuous operation, not for bandwidth.

So as long as the model fits entirely onto the cards, there is no trade-off between quality and speed. That trade-off only begins once it no longer fits, and that is exactly what we measure further down with the 284-billion model.

Now to your number from the second question. Besides the model, every running conversation also takes up space on the card, because the model has to carry its history along. This buffer is called the KV cache?KV cacheThe working memory in which a model holds the conversation so far. It grows with the text length and with every concurrent user, and often takes up more graphics memory than the model itself., and it grows with the text length and with every concurrent user.

It explains why two cards deliver more than one does. On a single RTX 4090, a 27-billion model takes up so much space that only around 15,000 tokens remain for all conversations combined, not even a single longer request. On two cards, there is room for 337,000. The second card does not make the model smarter, it makes it operable.

Measured under realistic load, meaning 10,000 input tokens and 2,000 output tokens per request:

The middle column counts all users together, the right-hand one measures how fast the text appears for a single person. The two cannot be converted into each other, because a request spends part of its time reading rather than writing.

Concurrent requests	Tokens per second, total	Per person	Longest wait time
1	65.8	74.3	3.5 s
4	169.0	59.7	13.7 s
8	240.6	51.4	27.5 s
16	301.6	39.3	55.1 s

Total throughput keeps rising to the end, and the per-person writing speed stays readable. What tips over is the longest wait time: from 3.5 to 55 seconds. At 10,000 input tokens, the card spends most of its time reading the requests, and these reading phases get in each other’s way.

The machine copes well with eight concurrent requests. At sixteen, the longest wait time rises to 55 seconds, and nobody can promise reliably any more how long an answer will take. If your number from the second question sits below that, the machine fits.

The extreme case: 284 billion parameters

How far this can be pushed is shown by the most extreme case. DeepSeek V4 Flash has 284 billion parameters and 51.8 index points, closing a good two thirds of the gap to the open flagship. Does a model like that run on a machine costing CHF 8’500?

It runs. In 10.9 GB of graphics memory.

What makes that possible is the design pushed to its most extreme form. The full name reads DeepSeek-V4-Flash-284B-A13B: 284 billion parameters are stored, 13 billion are computed per word. At a ratio like that, the rarely used experts are allowed to sit in cheap system memory, and only the ones actually needed travel to the graphics card. That is exactly what the 128 GB of system memory in the parts list is for.

The price paid for that is time. We asked a single German question and timed it: after 10.7 seconds the model began working, the first character became visible after 37.3 seconds, and the three-sentence answer was complete after 53.9 seconds. In between lay 196 reasoning tokens?Reasoning tokensReasoning models generate internal intermediate steps ("thinking") before the actual answer. These reasoning tokens are invisible but are billed as output. that nobody gets to see.

The bar counts down, and that is exactly the experience: a good half minute in front of an empty screen. Here the trade-off appears that did not exist for the models that sit on the cards. The same machine that delivered 104.6 tokens per second and person with the 35-billion model now writes 7.9, and the wait rises from 1.6 to 37.3 seconds. The reason is not the parameter count as such: the experts sit in system memory, which is connected at a fraction of the speed of the graphics card’s memory.

What this means for your company

For a business with twenty to thirty employees that writes correspondence, searches documents and translates, a machine of your own is feasible. It costs around CHF 8’500, stands under a desk and delivers a model at GPT-5.1 level. Anyone who also wants to do demanding coding or run agents over long chains keeps buying that part in.

If a task needs more quality and can wait, models of up to 284 billion parameters become an option, running overnight. A year ago, a model of this class only ran in a data centre.

For model choice, our recommendation is: MoE over dense, because the ratio of quality to speed comes out best there. Today that means Qwen. In a few months it will mean something else, and this paragraph will be out of date. So check the leaderboard yourself and remeasure the candidate on your own tasks before committing.

Two closing points, neither of which appears in any measurement series. First, the arithmetic: over three years, the machine has to run at full load around 39 percent of the time to come out cheaper than Swiss provider prices; over two years, 57 percent. We work through how these figures come about in the final part of this series. Second, a question on which whole projects regularly hinge in our engagements: Who operates the machine when it stops answering on a Friday evening?

Anyone who has answered both can decide. Going by our figures, the reason for a yes lies in control over the data, almost never in the price.

The measurements add up to a procedure in four steps. Work through them in this order before you order hardware:

1. 01

   Capture the tasks

   Write down ten to twenty real cases from your day to day, with the typical document length. Without this list, neither the model nor the hardware can be chosen sensibly.

2. 02

   Estimate the concurrent requests

   Work it out from the number of people active at the same time and their thinking pauses. The graphics cards depend on this number.

3. 03

   Measure with rented hardware

   A GPU hour with a Swiss or European provider costs a few francs. Only once your own tasks run cleanly there does the purchase pay off.

4. 04

   Staff the operation

   Updates, model swaps, monitoring and restarts are recurring work. Clarify who takes it on before the cards are ordered.

The remaining parts of the series go deeper into what only appeared in outline here: the memory arithmetic, quantisation, engine choice and finally the cost question.

Considering running a language model in your own business? The first question is usually whether the quality is even good enough.

The headlines suggest it is: open models, whose weights can be downloaded freely, now trail the closed flagship models by only 3.4 points on the Intelligence Index of Artificial Analysis, a performance comparison for AI models (12 August 2026). That figure, however, applies to models with trillions of parameters, ones nobody puts under a desk.

So we measured what actually runs on a machine costing CHF 8’450 with two RTX 4090 graphics cards. The best model that fits entirely into the 48 GB of graphics memory is Qwen3.6-27B from Alibaba at 37.7 points, level with GPT-5.1 (OpenAI). 22 points separate it from the open flagship. One special case remains: DeepSeek V4 Flash, a model made of many sub-models (mixture-of-experts) whose rarely used parts sit in system memory. It reaches 51.8 points at 7.9 tokens per second, so it suits overnight processing and is too slow for dialogue.

Our conclusion: for letters, document search and translation in your own business, that is enough today. You will keep buying in demanding coding work and long task chains with many tool calls for now.

The article has the parts list, all the measurements, and the explanation for why a larger model can be faster than a smaller one. Link in the comments.

#AI #DataSovereignty #LLM #Switzerland