---
title: How much VRAM do you need for local LLMs
canonical: "https://consideredgear.com/guides/how-much-vram-local-llm/"
pubDate: "2026-10-10T00:00:00.000Z"
updatedDate: "2026-10-10T00:00:00.000Z"
author: Jim Shannon
description: "Work out whether a local LLM fits an 8GB, 12GB, 16GB or 24GB GPU. Real model sizes, context-memory math and an Ollama check before you buy."
---
# How much VRAM do you need for local LLMs
Work out whether a local LLM fits an 8GB, 12GB, 16GB or 24GB GPU. Real model sizes, context-memory math and an Ollama check before you buy.
## How much VRAM do you need for local LLMs?
Start with the exact model file, then budget for conversation memory, working buffers and other GPU use. An 8 GB card is worth trying with a small quantized model; 24 GB makes larger models plausible. Neither is a fit guarantee. Check your chosen context and CPU/GPU split before buying.
Some links on this site may be affiliate links. As an Amazon Associate I earn from qualifying purchases. [How this works.](/disclosure/)
An 8 GB graphics card is worth trying with a small, quantized local model.
A 12 GB or 16 GB card gives you more room for model weights and longer
conversations. A 24 GB card makes some larger models plausible, but a model
download smaller than 24 GB can still exceed its memory budget.
**Choose the model and the conversation length first. Then choose the hardware.**
If you already own a supported GPU, try it before shopping. A model that fits
is only a candidate: it still needs to answer your questions well and run at a
speed you find useful.
**Evidence: sourced research checked 10 October 2026, not a hardware benchmark.**
The file sizes below come from Qwen's official repositories. The context-memory
figures are calculations, not measured VRAM readings. We have not tested these
models on the card sizes listed here, and we make no price, speed or quality
ranking from their memory requirements.
## Start with the model file
Parameter count alone is not a memory requirement. The exact model and its
quantization matter. Quantization stores weights with reduced precision; the
file becomes smaller, with possible effects on answer quality that need testing.
Here are three concrete downloads in Qwen's official Q4_K_M GGUF format. These
are examples for planning, not a recommendation that every reader use Qwen.
| Exact model file | Download size, GB | Download size, GiB |
| --- | ---: | ---: |
| [Qwen3-8B-Q4_K_M.gguf](https://huggingface.co/Qwen/Qwen3-8B-GGUF/blob/main/Qwen3-8B-Q4_K_M.gguf) | 5.03 | 4.68 |
| [Qwen3-14B-Q4_K_M.gguf](https://huggingface.co/Qwen/Qwen3-14B-GGUF/blob/main/Qwen3-14B-Q4_K_M.gguf) | 9.00 | 8.38 |
| [Qwen3-32B-Q4_K_M.gguf](https://huggingface.co/Qwen/Qwen3-32B-GGUF/blob/main/Qwen3-32B-Q4_K_M.gguf) | 19.76 | 18.40 |
Here, a GB is 1,000,000,000 bytes and a GiB is 1,073,741,824 bytes. We converted
the published byte counts and rounded to two decimal places. Compare the same
units when checking your GPU's reported memory. Do not automatically apply a
decimal-to-binary conversion to a graphics card's advertised “GB” label; check
what the hardware and driver actually report.
**Download size is not total VRAM use.** A llama.cpp maintainer's
[allocation explanation](https://github.com/ggml-org/llama.cpp/discussions/9936)
describes model-weight memory as usually close to file size, with separate
context and compute buffers. The file is a useful starting estimate for
weights, not a promise about the final allocation on every backend.
## Leave room for more than weights
Four things compete for memory when you run a local chat model:
- **Model weights:** the stored numbers the model uses to generate answers.
- **Conversation memory:** commonly a key-value, or KV, cache holding attention
information from processed tokens so it can be reused.
- **Working buffers:** temporary space used by the inference software.
- **Other GPU use:** the display, driver and other applications also need room.
There is no universal “add 1 GB” rule. Model architecture, backend, context,
batching and cache settings change the result. Some caches grow as a conversation
grows; others reserve space ahead of time. Hugging Face explains these
[cache strategies](https://huggingface.co/docs/transformers/main/en/kv_cache).
A short prompt that loads successfully does not establish that a long session
will fit.
## Use VRAM tiers as starting points
| Advertised GPU tier | A candidate to investigate | What still needs checking |
| --- | --- | --- |
| 8 GB | A small quantized model, such as the 4.68 GiB Qwen3-8B file above | Available memory after context, buffers and other applications |
| 12 GB | The same small model with more room, or the 8.38 GiB Qwen3-14B file | Whether the chosen context and backend leave sufficient headroom |
| 16 GB | More room around that 14B file | The 18.40 GiB 32B file is already larger than a 16 GiB budget before overhead |
| 24 GB | The 18.40 GiB 32B file becomes a candidate | Context can consume the apparent spare capacity, as the example below shows |
This is a shortlist for testing, not a compatibility chart or a quality ladder.
Confirm the actual card, operating system, driver and inference backend using
the application's [hardware support documentation](https://docs.ollama.com/gpu).
A supported GPU with enough memory still needs a workload test.
## Why a 24 GB card can struggle with a 20 GB download
For a worked example, take Qwen3-32B. Its official
[configuration](https://huggingface.co/Qwen/Qwen3-32B/blob/main/config.json)
specifies 64 layers, eight key-value heads and a head dimension of 128, with
sliding-window attention disabled.
Assume **one sequence and an FP16 KV cache**, using two bytes per value. Counting
both keys and values gives this calculated cache storage per token:
**2 × 64 layers × 8 KV heads × 128 head dimension × 2 bytes**
= **262,144 bytes per token**, or **0.25 MiB per token**.
The formula follows the per-layer key and value tensor dimensions described in
Hugging Face's [cache explanation](https://huggingface.co/docs/transformers/main/en/cache_explanation).
It is specific to these assumptions, not a formula for every model architecture.
| Tokens held in the cache | Calculated FP16 KV cache | File-size proxy for weights + cache |
| ---: | ---: | ---: |
| 8,192 | 2 GiB | 20.40 GiB |
| 16,384 | 4 GiB | 22.40 GiB |
| 32,768 | 8 GiB | 26.40 GiB |
These subtotals **exclude working buffers and other GPU use**. The last row
already exceeds a 24 GiB budget. The first two rows are not fit guarantees.
Different cache precision, allocation strategy or CPU/GPU placement changes
what is held in VRAM. The Q4_K_M label describes the model weights; it does not
mean this example's cache is also four-bit.
## Reserve context for the answer
Context includes more than the text you paste. System instructions, chat
history, retrieved material, tool output and generated tokens all take space.
With an 8,192-token budget, reserving 2,048 tokens for an answer leaves 6,144
for everything that comes before it. That is arithmetic, not a prediction of
answer length or a word-count conversion.
Parallel requests can raise the memory requirement too. Do the first check
with one request, then test the concurrency you actually intend to use.
## Check the machine you already own with Ollama
Use a local model and a representative task. This is a diagnostic procedure;
we have not run it as a benchmark for this article.
1. Record the exact local model, quantization, Ollama version, GPU, driver and
operating system. A matching parameter count is not enough to reproduce a run.
2. In an `ollama run` session, set the context you intend to use. For example,
`/set parameter num_ctx 8192` requests 8,192 tokens; it is not a universal
recommended size.
3. Send a representative prompt, then run `ollama ps` in another terminal while
the model is loaded. Inspect `PROCESSOR` for the CPU/GPU split and `CONTEXT`
for the allocated context reported by your version.
4. Exercise the full task, including a longer conversation or document and the
other applications you normally keep open. A successful initial load is only
the beginning of the check.
5. Record the result: allocation, CPU/GPU split, errors and whether the response
time and answers are useful to you. Change one setting at a time when comparing.
Ollama documents [context inspection](https://docs.ollama.com/context-length)
and the [interactive setting and processor split](https://docs.ollama.com/faq).
Its documentation pages currently describe defaults differently, so this guide
does not assume one default for every version. Set and inspect yours explicitly.
On NVIDIA hardware, `nvidia-smi` provides another view of memory use.
[NVIDIA distinguishes total, reserved, used and free framebuffer memory](https://docs.nvidia.com/deploy/nvidia-smi/index.html#fb-memory-usage).
Free memory matters more to a new allocation than the number printed on the box.
A snapshot still does not show the peak of a whole session.
## If the model does not fit
Try a shorter context if it still serves the task. Otherwise, try a smaller
model or a different weight quantization and compare the answers. Neither is
a free improvement: you are changing what the application can do.
Cache quantization is a separate setting from weight quantization. Support and
quality tradeoffs depend on the backend and model. Read the applicable
[Ollama cache guidance](https://docs.ollama.com/faq) before changing it.
CPU/GPU offloading can let a model use system RAM as well as VRAM. Software such
as [llama.cpp](https://github.com/ggml-org/llama.cpp) supports hybrid inference,
but there is no fixed slowdown we can honestly promise. Measure the workload.
More system RAM does not simply become more dedicated GPU memory.
Apple silicon uses unified memory, so the operating system, applications and
model share a budget. Do not treat the whole advertised capacity as an exclusive
model allocation. Our [M4 Mac mini research notes](/reviews/mac-mini-m4/)
cover that different buying choice.
## Choose the whole computer after the memory check
If 24 GB of dedicated graphics memory looks appropriate, the
[used RTX 3090 buying notes](/reviews/nvidia-rtx-3090-used/) cover compatibility
and used-card checks. The card still needs a suitable desktop around it.
Use the [SER8 versus used RTX 3090 complete-cost comparison](/guides/ser8-vs-used-rtx-3090-total-cost/)
to separate an upgrade to a computer you already own from buying everything.
For the broader choices, see the [home-AI hardware guide](/guides/best-way-to-run-ai-at-home-2026/).
## Sources and method
Research checked **10 October 2026**. Qwen's official Git LFS records give
5,027,783,488 bytes for the
[8B file](https://huggingface.co/Qwen/Qwen3-8B-GGUF/raw/main/Qwen3-8B-Q4_K_M.gguf),
9,001,752,960 bytes for the
[14B file](https://huggingface.co/Qwen/Qwen3-14B-GGUF/raw/main/Qwen3-14B-Q4_K_M.gguf)
and 19,762,149,024 bytes for the
[32B file](https://huggingface.co/Qwen/Qwen3-32B-GGUF/raw/main/Qwen3-32B-Q4_K_M.gguf).
The download table converts those values; the cache table uses the stated
Qwen3-32B configuration and FP16 assumptions. Totals were calculated before
rounding. Documentation and files can change after this date.
No GPU inference test, power measurement, throughput measurement or quality
evaluation was performed for this article. There are no measured fit results
or current hardware prices here. Use the calculations to choose what to test,
then make a buying decision from your actual workload and complete-system cost.
## Buying options, if they fit your setup
Work out your required VRAM and total system cost before shopping. A 24 GB RTX 3090 is one hardware category worth researching, not a promise that your chosen model or computer will work.
- [Browse NVIDIA RTX 3090 24 GB graphics cards (paid link)]() — Check condition, cooling, power supply, physical fit, drivers and return terms.
These are Amazon search references, not verified offers. Prices, availability and exact configurations are unconfirmed. [How affiliate links work](/disclosure/).
As an Amazon Associate I earn from qualifying purchases.