Best hardware for a local AI server
Last updated
The best hardware for a local AI server holds your largest model in fast memory. For models up to about 32B, a 24 GB NVIDIA GPU like Olares One's RTX 5090 Mobile fits. For 70B and up, buy 128 GB of unified memory.
Olares One → Local AI server → Ollama · vLLM · Open WebUI → Private AI on your own network
How much memory does each model size need?
By an approximate rule of thumb, a model quantized to 4-bit needs 0.5-0.6 GB of memory per billion parameters, plus a few GB for context. A 24 GB GPU therefore covers models up to about 32B, and anything from 70B upward needs a machine with far more memory.
| Model size (4-bit) | Approximate memory |
|---|---|
| 8B | ~5 GB |
| 14B | ~9 GB |
| 32B | ~19-20 GB |
| 70B | ~40 GB |
| 120B | ~65 GB |
What is the best hardware for a local AI server at each budget?
There are four kinds: ready-made GPU servers, self-built GPU towers, mini PCs with 128 GB of unified memory, and Macs. The prices below were read on makers' stores, or in the named build guide, on 12-13 Nov 2026. Several were sale prices and may have changed since.
| Machine | Memory for models | Price |
|---|---|---|
| Olares One | 24 GB VRAM (RTX 5090 Mobile) | 4,999 USD list |
| Used RTX 3090 tower | 24 GB VRAM, 936 GB/s | 3,800 USD (SpecPicks estimate, no case) |
| Minisforum MS-S1 Max | 128 GB shared (Ryzen AI Max+ 395) | 3,959 USD (sale; list 4,949 USD) |
| GMKtec EVO-X2 | 128 GB shared (Ryzen AI Max+ 395) | 3,499.99 USD (sale) |
| NVIDIA DGX Spark | 128 GB unified, 273 GB/s | 6,950 USD (4 TB) |
| Mac Studio (M5 Ultra) | 96 GB unified, 1.2 TB/s | 5,499 USD base |
Is unified memory as fast as VRAM?
Usually not. Unified memory buys capacity, and bandwidth decides speed once a model fits. NVIDIA lists 273 GB/s for DGX Spark, while the SpecPicks guide lists 936 GB/s for a used RTX 3090's 24 GB. Apple's M5 Ultra is the exception, with 1.2 TB/s on Apple's spec page.
On the Ryzen AI Max+ 395 boxes, the GPU can use only part of the 128 GB. GMKtec does not state how much, so check it for the model you plan to run.
Where does Olares One fit?
Olares One is our pick for a home or small-office server that runs models up to about 32B, image generation and several users at once. If your main model is a dense 70B, it is the wrong machine, and a 128 GB unified-memory box such as DGX Spark is the better buy.
In Olares' own test it served Qwen3-30B-A3B at 157 tokens/s for one request. Larger models spill out of VRAM into system memory, and Olares' FAQ gives about 36 tokens/s for gpt-oss-120b run that way. Weigh the trade in DGX Spark alternatives and the GPU guide.
Questions & answers
Is 24 GB of VRAM enough for a local AI server?
For models up to about 32B at 4-bit, yes, by the approximate rule of 0.5-0.6 GB per billion parameters plus context. A dense 70B model needs about 40 GB, so it does not fit.
Which machine has the most memory for the money?
Among prices read on 12-13 Nov 2026, the 128 GB Ryzen AI Max+ 395 boxes, such as the GMKtec EVO-X2 at 3,499.99 USD on sale. They have no CUDA, and the GPU can use only part of the 128 GB.
Sources for this entry
- Olares One specifications (Olares docs) (Company-published)
- Olares One FAQ (Olares docs) (Company-published)
- Olares blog: local AI hardware benchmarking (Oct 2025) (Company-published)
- Used RTX 3090 tower build: maker's page (Maker-published)
- Minisforum MS-S1 Max: maker's page (Maker-published)
- GMKtec EVO-X2 (Ryzen AI Max+ 395, 128GB): maker's page (Maker-published)
- NVIDIA DGX Spark: maker's page (Maker-published)
- Apple Mac Studio (M5 Ultra): maker's page (Maker-published)
Continue this reading path
How the related websites connect
Cite this entry
Best Local AI Server. “Best hardware for a local AI server.” Content reviewed 13 November 2026.
https://bestlocalaiserver.com/best-hardware-for-local-ai-server/