What a 32 GB RTX 5090 actually runs

Someone has probably already told you this machine will run a 70B. Whether it does is settled by arithmetic rather than by opinion, and the arithmetic starts with the card. An RTX 5090 has 32 GB of GDDR7, and that 32 GB is the entire space a model has to live in.

NVIDIA's product page lists the 5090 at 32 GB of GDDR7 on a 512-bit bus, 21,760 CUDA cores and 575 W of board power. The vendor spec sheets put memory bandwidth at 1792 GB/s.

Bandwidth sets how fast the weights can be read. Capacity decides whether they load at all, and capacity is what this page is about. Three things compete for the same 32 GB: the model weights, the KV cache that holds your context and the Windows desktop.

The desktop is the part people forget. On our own builds we see 0.5 to 1.5 GB go to Windows and the monitors when the same card is drawing your screen, so we plan against about 31 GB rather than 32. That is our measurement on the machines we deliver, not a published NVIDIA figure. A second cheap display card, or running the 5090 headless, gets most of that back.

Every number below is a file size on disk, close to what the weights occupy in video memory once loaded. You can check each one on the model card yourself before you spend anything.

The models that fit an RTX 5090 with room left over are the 20B to 32B class at 4-bit. That is the class buyers are usually looking at for office document work.

Qwen3.8-27B is the one to build on. It is 27B dense, ships at 17.4 GB in bartowski's Q4_K_M or 16.5 GB in unsloth's UD-Q4_K_M and has a native 262k context window. It is also a vision-language model. The vision projector ships as a separate 0.93 GB file on bartowski's repo, and the 17.4 GB figure does not include it.

Gemma 4 31B is 19.6 GB at Q4_K_M and shipped in April 2026. Mistral Small 3.2 24B is the smallest of the three at 14.3 GB.

gpt-oss-20b is 21B total with 3.6B active and post-trained in MXFP4, and ggml-org's MXFP4 GGUF is 12.1 GB. The 5090 is a Blackwell card, so it runs MXFP4 at its intended precision rather than emulating it. Llama 3.1 8B is 4.9 GB at Q4_K_M, 6.6 GB at Q6_K and 8.5 GB at Q8_0, so the 8B class runs on this card at near-full precision.

Now the limit, because precision is where the headroom goes. Qwen3.8-27B at Q8_0 is 29.1 GB, which leaves nothing for context. Bartowski's Q5_K_M at 20.9 GB, or unsloth's UD-Q5_K_M at 19.8 GB, is the practical ceiling for that model on one card.

Llama 3.3 70B at Q4_K_M is 42.5 GB of weights, which is over a 32 GB card before a single token of context. There is no honest quantization of a 70B that fixes it.

Dropping the quantization does not rescue it. The same 70B is 37.9 GB at IQ4_XS and 34.3 GB at Q3_K_M, both still over the card. Weights are not the only cost. Llama 3.3 70B has 80 layers with 8 KV heads at head_dim 128, which is 320 KiB per token. Context adds 2.7 GB at 8k and 10.7 GB at 32k on top of the weights.

Qwen2.5-72B is further out of reach at 47.4 GB in Q4_K_M, 39.7 GB in IQ4_XS and 37.7 GB in Q3_K_M. Llama 4 Scout is the smaller of the two Llama 4 models, at 109B total with 17B active. It is 65.4 GB at Q4_K_M, 62 GB at UD-Q4_K_XL and 42.4 GB even at UD-Q2_K_XL. Meta's own guidance for Scout is a single H100 with on-the-fly int4.

If a builder tells you the big model will run, ask which quantization and what context length. Then read the file size off the model card before you sign anything.

The weights are only part of the bill. The KV cache holds the document you feed the model, and it grows with every token. On some architectures it grows four times faster than on others.

Qwen3.8-27B is the cheap one, and deliberately so. Its config.json shows 64 layers of which only 16 use full attention, in a pattern of three linear layers to one full. It has 4 KV heads at head_dim 256. That works out to 64 KiB per token: 0.5 GB at 8k, 2.1 GB at 32k and 8.6 GB at 128k. The model plus 32k of context is about 20 GB, and even 128k lands near 26 GB.

Qwen2.5-32B shows the other case at the same size on paper. It is 19.9 GB at Q4_K_M and runs full attention on all 64 layers with 8 KV heads. That costs 256 KiB per token, which is 8.6 GB at 32k and pushes the total to about 28 GB. At 8k it is fine at about 22 GB, so the model is not the problem; the context length is.

Gemma 4 31B sits between them. Its config.json has 60 layers, 16 KV heads, head_dim 256 and a 1024-token sliding window. Every sixth layer uses full attention, so the 50 sliding layers cost a constant 0.84 GB. The cache is 2.2 GB at 8k, for a total near 22 GB. At 32k it is 6.2 GB, for a total of about 26 GB, which is tight against the 31 GB we plan for.

These cache figures are computed from the published config.json values at FP16, two bytes per element, rather than taken from a blog post.

A model family name tells you nothing about what fits on your card. The current Mistral Small needs four H100s, and the Mistral that runs on a 5090 is the previous generation.

Mistral Small 4 arrived in March 2026 at 119B total parameters with 6B active, under Apache 2.0, with a 256k context window. Mistral's own announcement states a minimum of 4x NVIDIA HGX H100, 2x HGX H200 or 1x DGX B200. The Mistral that actually runs on a 5090 is Mistral Small 3.2 24B, at 14.3 GB in Q4_K_M. Trust the family name here and you download a model several times too large for the machine you just bought.

Llama reads the same way. Llama 4 Scout is 65.4 GB in Q4_K_M, and the only Llamas that fit one card are the 8B class. Qwen and Gemma are the two families where the name and the fit agree. Both publish a dense model in the 27B to 31B range.

These are the current releases as of September 2026, and vendor lineups move. Check the model card and the launch page before you download. The safe habit is to buy against a named model and a named quantization, never against a family name.

Two 5090s give you 64 GB, which runs Llama 3.3 70B at Q4_K_M with 32k of context at about 53 GB. That is the build we quote when the larger models, or several people at once, are the actual requirement.

Our Pro workstation is a single RTX 5090 class card with 128 GB of system RAM, at $9,000 to $11,000. The two-GPU office server, which serves the team over your network, is $16,000 to $20,000. Both are one-time hardware ranges, before any software build.

The honest version of the second card is that it buys capacity and nothing else. It does not make a 27B answer better. If the 20B to 32B class already handles your leases and your closing files, the second card is headroom you will not use.

Two cards do not rescue everything. Llama 4 Scout at 65.4 GB in Q4_K_M is still over 64 GB. The UD-Q2_K_XL build that would fit is a quantization we would not put in front of a client. The reason to go to two cards is a 70B at a real quantization with real context, or several people querying at once.

File sizes and cache arithmetic tell you what loads. They do not tell you how fast it answers, or whether the model is any good at your work.

We did not benchmark tokens per second for any model on this page, so there is no speed figure here to quote. Be careful with anyone who quotes one without saying which quantization, which context length and which machine produced it. A card’s bandwidth sets a ceiling, not a result.

Quality is the other gap. A model that loads with room to spare can still be the wrong model for your scanned leases, your closing files or your intake forms. Output quality depends on the prompt, the retrieval step and the state of your documents, and none of that is visible in a gigabyte figure. We test with your real files rather than quoting a number off a spec sheet.