How much VRAM do you need for local LLMs

Some links on this site may be affiliate links. As an Amazon Associate I earn from qualifying purchases. How this works.

An 8 GB graphics card is worth trying with a small, quantized local model. A 12 GB or 16 GB card gives you more room for model weights and longer conversations. A 24 GB card makes some larger models plausible, but a model download smaller than 24 GB can still exceed its memory budget.

Choose the model and the conversation length first. Then choose the hardware. If you already own a supported GPU, try it before shopping. A model that fits is only a candidate: it still needs to answer your questions well and run at a speed you find useful.

Evidence: sourced research checked 10 October 2026, not a hardware benchmark. The file sizes below come from Qwen’s official repositories. The context-memory figures are calculations, not measured VRAM readings. We have not tested these models on the card sizes listed here, and we make no price, speed or quality ranking from their memory requirements.

Start with the model file

Parameter count alone is not a memory requirement. The exact model and its quantization matter. Quantization stores weights with reduced precision; the file becomes smaller, with possible effects on answer quality that need testing.

Here are three concrete downloads in Qwen’s official Q4_K_M GGUF format. These are examples for planning, not a recommendation that every reader use Qwen.

Here, a GB is 1,000,000,000 bytes and a GiB is 1,073,741,824 bytes. We converted the published byte counts and rounded to two decimal places. Compare the same units when checking your GPU’s reported memory. Do not automatically apply a decimal-to-binary conversion to a graphics card’s advertised “GB” label; check what the hardware and driver actually report.

Download size is not total VRAM use. A llama.cpp maintainer’s allocation explanation describes model-weight memory as usually close to file size, with separate context and compute buffers. The file is a useful starting estimate for weights, not a promise about the final allocation on every backend.

Leave room for more than weights

Four things compete for memory when you run a local chat model:

• Model weights: the stored numbers the model uses to generate answers.
• Conversation memory: commonly a key-value, or KV, cache holding attention information from processed tokens so it can be reused.
• Working buffers: temporary space used by the inference software.
• Other GPU use: the display, driver and other applications also need room.

There is no universal “add 1 GB” rule. Model architecture, backend, context, batching and cache settings change the result. Some caches grow as a conversation grows; others reserve space ahead of time. Hugging Face explains these cache strategies. A short prompt that loads successfully does not establish that a long session will fit.

Use VRAM tiers as starting points

This is a shortlist for testing, not a compatibility chart or a quality ladder. Confirm the actual card, operating system, driver and inference backend using the application’s hardware support documentation. A supported GPU with enough memory still needs a workload test.

Why a 24 GB card can struggle with a 20 GB download

For a worked example, take Qwen3-32B. Its official configuration specifies 64 layers, eight key-value heads and a head dimension of 128, with sliding-window attention disabled.

Assume one sequence and an FP16 KV cache, using two bytes per value. Counting both keys and values gives this calculated cache storage per token:

2 × 64 layers × 8 KV heads × 128 head dimension × 2 bytes

= 262,144 bytes per token, or 0.25 MiB per token.

The formula follows the per-layer key and value tensor dimensions described in Hugging Face’s cache explanation. It is specific to these assumptions, not a formula for every model architecture.

These subtotals exclude working buffers and other GPU use. The last row already exceeds a 24 GiB budget. The first two rows are not fit guarantees. Different cache precision, allocation strategy or CPU/GPU placement changes what is held in VRAM. The Q4_K_M label describes the model weights; it does not mean this example’s cache is also four-bit.

Reserve context for the answer

Context includes more than the text you paste. System instructions, chat history, retrieved material, tool output and generated tokens all take space. With an 8,192-token budget, reserving 2,048 tokens for an answer leaves 6,144 for everything that comes before it. That is arithmetic, not a prediction of answer length or a word-count conversion.

Parallel requests can raise the memory requirement too. Do the first check with one request, then test the concurrency you actually intend to use.

Check the machine you already own with Ollama

Use a local model and a representative task. This is a diagnostic procedure; we have not run it as a benchmark for this article.

1. Record the exact local model, quantization, Ollama version, GPU, driver and operating system. A matching parameter count is not enough to reproduce a run.
2. In an ollama run session, set the context you intend to use. For example, /set parameter num_ctx 8192 requests 8,192 tokens; it is not a universal recommended size.
3. Send a representative prompt, then run ollama ps in another terminal while the model is loaded. Inspect PROCESSOR for the CPU/GPU split and CONTEXT for the allocated context reported by your version.
4. Exercise the full task, including a longer conversation or document and the other applications you normally keep open. A successful initial load is only the beginning of the check.
5. Record the result: allocation, CPU/GPU split, errors and whether the response time and answers are useful to you. Change one setting at a time when comparing.

Ollama documents context inspection and the interactive setting and processor split. Its documentation pages currently describe defaults differently, so this guide does not assume one default for every version. Set and inspect yours explicitly.

On NVIDIA hardware, nvidia-smi provides another view of memory use. NVIDIA distinguishes total, reserved, used and free framebuffer memory. Free memory matters more to a new allocation than the number printed on the box. A snapshot still does not show the peak of a whole session.

If the model does not fit

Try a shorter context if it still serves the task. Otherwise, try a smaller model or a different weight quantization and compare the answers. Neither is a free improvement: you are changing what the application can do.

Cache quantization is a separate setting from weight quantization. Support and quality tradeoffs depend on the backend and model. Read the applicable Ollama cache guidance before changing it.

CPU/GPU offloading can let a model use system RAM as well as VRAM. Software such as llama.cpp supports hybrid inference, but there is no fixed slowdown we can honestly promise. Measure the workload. More system RAM does not simply become more dedicated GPU memory.

Apple silicon uses unified memory, so the operating system, applications and model share a budget. Do not treat the whole advertised capacity as an exclusive model allocation. Our M4 Mac mini research notes cover that different buying choice.

Choose the whole computer after the memory check

If 24 GB of dedicated graphics memory looks appropriate, the used RTX 3090 buying notes cover compatibility and used-card checks. The card still needs a suitable desktop around it.

Use the SER8 versus used RTX 3090 complete-cost comparison to separate an upgrade to a computer you already own from buying everything. For the broader choices, see the home-AI hardware guide.

Sources and method

Research checked 10 October 2026. Qwen’s official Git LFS records give 5,027,783,488 bytes for the 8B file, 9,001,752,960 bytes for the 14B file and 19,762,149,024 bytes for the 32B file. The download table converts those values; the cache table uses the stated Qwen3-32B configuration and FP16 assumptions. Totals were calculated before rounding. Documentation and files can change after this date.

No GPU inference test, power measurement, throughput measurement or quality evaluation was performed for this article. There are no measured fit results or current hardware prices here. Use the calculations to choose what to test, then make a buying decision from your actual workload and complete-system cost.