How much memory a local language model needs

A language model running on a laptop or a server with no network needs memory for three things: its weights, a key-value (KV) cache that grows with every token of context, and the working buffers of the software that runs it. The Hugging Face guide to optimizing large language models gives the rule of thumb for weights: a model with X billion parameters needs roughly 2 × X GB in 16-bit precision. The cache is the part that grows with use. For one example model, the same guide computes that a 16,000-token cache needs about 15 GB, about half as much as the weights.

Weights: parameters times bits

Each parameter is a number stored at some precision. The Hugging Face guide gives roughly 4 × X GB for X billion parameters in 32-bit floating point and 2 × X GB in bfloat16 or float16, and notes that the formula is best read as an "at most" figure, because a gigabyte computed with 1,000 differs from one computed with 1,024. For inputs under 1,024 tokens, the guide says, inference memory is dominated by the weights.

Quantized files store fewer bits per weight. The Hugging Face GGUF documentation lists the Q4_K type at 4.5 bits per weight: super-blocks of 8 blocks, each holding 32 weights, with a 6-bit scale and a 6-bit minimum per block. Weight memory in bytes is then parameters × bits per weight ÷ 8. That arithmetic holds up for loading: the Accelerate model memory estimator documentation says its calculation is accurate within a few percent of the actual value.

The key-value cache and context length

During generation, a model keeps the key and value vectors of every earlier token, in every layer, so it does not recompute them for each new token. The Transformers cache documentation gives each layer's key and value tensors the shape batch size × heads × sequence length × head dimension, so memory grows linearly with the sequence.

Per token, the cache therefore holds 2 (a key and a value) × layers × key-value heads × head dimension × bytes per value, multiplied again by the number of sequences held at once. The PagedAttention paper by Kwon and colleagues, presented at the 2023 Symposium on Operating Systems Principles (SOSP), works this for the 13-billion-parameter OPT model: 2 × 5,120 (hidden state size) × 40 (layers) × 2 bytes in 16-bit floating point (FP16) gives 800 KB per token, and up to 1.6 GB for one 2,048-token request.

Many models now share key-value heads among query heads. The Hugging Face guide describes multi-query attention (MQA), with one shared key-value head, and grouped-query attention (GQA), with a small number such as 2, 4, or 8. For the guide's example model, MQA cut a 16,000-token cache from 15 GB to less than 400 MB.

A worked example from a published model card

This example is an illustration of the arithmetic only, chosen because the card publishes every number the formula needs. The Mistral-Nemo-Base-2407 model card describes a model of 12B parameters trained with a 128k context window, with 40 layers, 32 heads, 8 key-value heads (GQA), and a head dimension of 128. The repository's config.json sets the maximum context at 131,072 tokens and the stored type to bfloat16.

• Weights at 16 bits: 12 billion × 2 bytes = 24 GB, using the card's rounded parameter count.
• Weights at Q4_K's 4.5 bits per weight: 12 billion × 4.5 ÷ 8 = 6.75 GB.
• Cache per token at 16 bits: 2 × 40 × 8 × 128 × 2 bytes = 163,840 bytes, or 160 KiB. KiB, MiB, and GiB count in powers of 1,024.

Context length	KV cache at 16 bits, one sequence
4,096 tokens	640 MiB (0.67 GB)
32,768 tokens	5 GiB (5.37 GB)
131,072 tokens	20 GiB (21.47 GB)

At the full context, the cache alone is more than three times the 4.5-bit weights and close to the 24 GB of 16-bit weights. With 32 key-value heads instead of 8, every cache figure would be four times larger.

Runtime overhead

Weights and cache are not the whole bill. For a 13-billion-parameter model on a 40 GB graphics processing unit (GPU), the PagedAttention paper shows about 65% of memory on weights, close to 30% on the dynamic state of requests, and a small amount used briefly for activations. The Accelerate documentation advises that for inference one can expect to add up to an additional 20%, as found by EleutherAI. Attention also needs working memory. The Hugging Face guide computes that the attention score matrices of a 40-head model in bfloat16 take 40 × 2 × N² bytes for N tokens: about 50 MB at 1,000 tokens and 19 GB at 16,000. Flash Attention gives identical outputs with memory that grows only linearly with N, computing the result in chunks in fast on-chip memory.

Memory-mapped loading in llama.cpp

The llama.cpp command-line documentation, as read on its master branch in October 2026, shows how one local runtime handles these parts. Loading defaults to auto, which memory-maps the model file unless a device does not support it. Without memory mapping, loading is slower but "may reduce pageouts if not using mlock," and mlock forces the system to keep the model in random-access memory (RAM) rather than swapping or compressing it.

The context size defaults to 0, meaning the value is loaded from the model, so a long-context model can ask for a long cache unless a smaller size is set. A fit option, on by default, adjusts unset arguments to fit device memory, with a default margin of 1,024 MiB per device and a minimum context of 4,096 tokens. The cache type defaults to f16, and smaller types such as q8_0 and q4_0 are allowed. Flash Attention defaults to auto.

What to check

• What context length is configured, and what cache size does it imply at the cache type in use?
• Does a quoted memory figure cover the weights only, or weights, cache, and runtime buffers together?
• How many sequences are held at once? The cache scales with batch size.
• Is the model memory-mapped, locked in RAM, or neither, and how does it behave when RAM runs short?
• Do the model card's figures match the configuration file? This card lists a "hidden dim" of 14,436, while its config.json gives an intermediate_size of 14,336.

Azcuna's BABEL reads, translates, and searches foreign-language documents entirely on-device, with no cloud and no connection, and keeps every claim tied to the passage it came from.

Sources

1. 1Optimizing LLMs for Speed and Memory · Hugging Face Transformers documentation
2. 2How caching works · Hugging Face Transformers documentation
3. 3Model memory estimator · Hugging Face Accelerate documentation
4. 4GGUF · Hugging Face Hub documentation
5. 5Efficient Memory Management for Large Language Model Serving with PagedAttention · arXiv (Kwon et al., SOSP 2023)
6. 6Model Card for Mistral-Nemo-Base-2407 · Mistral AI, Hugging Face model repository
7. 7config.json (Mistral-Nemo-Base-2407) · Mistral AI, Hugging Face model repository
8. 8llama.cpp/tools/cli (README) · ggml-org (llama.cpp project), master branch as read October 21, 2026

Where this shows up in our work

• ELLMENT, the offline LLM for E-2D crews answers a crew's question from the aircraft's own manuals and cites the page, with no network.
• BABEL, offline multilingual translation and search translates and searches foreign-language documents on-device, every fact tied to its source passage.
• TRACE, the offline S1000D and IETM viewer opens S1000D, MIL-STD-3001 and PDF technical manuals on a tablet with every marking intact.