TheFocus Labs
THEFOCUS.AI LABS MEASUREMENT RM-2026-07-KV

Local inference field report · July 16–17, 2026

Conversation roots
are the cache.

Local KV reuse is not a chat ID. It is an exact token-prefix match against whatever is still resident in a server slot. Stabilize the shared root, warm it once, fork only at the end—and prompt work can drop to roughly a tenth of a cold run. Scramble the front of the prompt, oversubscribe slots, or pick a runtime path that cannot keep the match, and you pay for the whole briefing again. We measured that mechanism on Ollama and on llama-server with the same harness.

One shared context root branching into five agent histories, with reused prefixes in blue and new suffixes in red, before converging into a reducer
~0.1×warm shared-root prompt cost
exactprefix match required
slotslimit concurrent reuse
2runtimes compared

Takeaways first

How to win the local KV cache

One study, two runtimes, same mental model. Nine Ollama configurations across seven scenarios; eight llama-server GGUFs on the shared-root suite with a direct cache_n hit counter. Design the prompt so the expensive part stays identical; size the server so a slot is free to hold it.

One strip below = one request’s context window. Segments are the things you actually put in it; the cache line marks the first byte that differs from the resident prefix. Everything left of the line is nearly free; everything right of it is evaluated from scratch. Toggle the layout and watch where the line lands.

token 0≈ 2,700 shared tokenswindow →
Paid once (cached)2,700 tokens
Paid every request150 tokens
Warm prompt cost≈ 0.05×

System, tools, and briefing are byte-identical at the front of every branch. Only the short task tail is new. This is the layout that produced 0.04–0.05× warm ratios.

request history=matching prefix+new suffix

Rules for local agent systems

  1. Stabilize the root. System prompt + shared briefing must be byte-for-byte identical at the front of every branch.
  2. Warm once, then fork. Pay for the long root on the first request; send short lane-specific tails afterward.
  3. Never put uniqueness first. Lane IDs, timestamps, and tool noise belong after the shared material.
  4. Size slots to the fan-out. If five agents arrive together, you need enough parallel slots—or accept a queue.
  5. Context after slots. Reserve KV for the windows you will actually keep warm; huge unused context is a tax.
  6. Reducers are new prompts. Branch outputs must be pasted in explicitly; caches do not merge.
  7. Trust wall time and cache hits. Client parallel ≠ server parallel. On llama.cpp, read cache_n; on Ollama, compare against a unique-prefix control.
  8. Doubt the runtime before the weights. Nemotron looked cache-blind on Ollama and reused normally on llama-server.
Default five-agent boxGPT-OSS 20B · enough slots · modest ctx

Best practical balance under Ollama (warm parallel continuations ~2.3s at 128K) and still a clean llama.cpp profile.

Cache-heavy sequential workWarm root → fork tails

Gemma 4 26B (Ollama) and GLM 4.7 Flash (llama.cpp, 0.08× prompt ratio) crush after the root is resident.

One deep threadOne slot, large context

Gemma 4 31B at 256K fits; five copies of that window do not. Prefer depth over fan-out.

Nemotron / odd architecturesMeasure on llama.cpp too

Do not treat an Ollama serial/no-reuse result as a model property until a second runtime agrees.

Evidence · Interactive 01 · Root anatomy

Follow one root as it branches, continues, and reduces

The harness sent the full chat history on every request—no conversation ID. Blue segments below are token prefixes available for reuse; red segments are newly evaluated. Step through the topology to see when the match holds and when a reducer becomes a new prompt.

Conversation root reuse diagram An interactive diagram showing a common root branching into five agent conversations and then feeding a reducer.
Current requestWarm the shared root
Reusable prefix≈ 2,700 tokens
New workRoot prompt itself

The first request pays for the root. Keeping the model loaded makes this prefix available to later requests.

request history=matching prefix+new suffix

Interactive 02 · Prefix microscope

Where uniqueness sits in the prompt decides your bill

The server already holds one warm prompt in a slot (top row). Every other row is a new request. Read left → right as start → end of that prompt. Blue means “still identical to the warm copy, so skip the work.” The first token that differs turns the rest red—even if the later text looks the same to you.

A lane ID, a timestamp, or extra whitespace at the front is enough. The cache does not know your intent; it knows bytes from the left.

Evidence · Scenario map

Seven tests, three pressures

Every scenario sends the full prompt history. The difference is what stays identical at the front, whether requests arrive one-at-a-time or together, and whether a later reducer starts a brand-new prompt. Pick a pressure to see the request shape.

Shared root

Can later forks skip the briefing?

Warm one large root, then diverge. Sequential isolates pure prefix reuse; parallel asks whether slots and bandwidth keep up when five forks arrive together.

Blue is the reusable prefix. Red is newly evaluated work. Left → right is time.

Evidence · Measured results

Prefix reuse and elapsed time are different questions

Prompt-evaluation ratio compares prompt processing with that model’s unique-prefix control (lower is better). Wall time is what the client waited for. On llama.cpp we also have cache_n; on Ollama the ratio is the reuse signal. Same scenarios, both stacks.

Strong reuse Contention / partial No useful reuse

Best five-agent default GPT-OSS 20B

On Ollama: 128K context, 12.09 GiB resident, warm parallel continuations in 2.34s. On llama.cpp: still a clean sequential profile (0.12× · 100% cache).

Cache-heavy sequential ~0.08–0.15× after a warm root

Gemma 4 26B hit 0.05× on Ollama; every llama.cpp GGUF reported 100% cache_n on shared sequential forks. Same mechanism, both stacks.

Slots beat wishful parallel Fan-out without lanes recomputes

Oversized Ollama slots and -np 2 with three llama.cpp agents both show the same failure: cold forks re-pay the root while siblings hit cache.

Runtime, not weights Nemotron flips between stacks

Ollama forced one slot and ~0.82–1.37× ratios. The same Nano 4B on llama-server posted 0.10× sequential reuse. Doubt the serving path before the model.

Interactive 05 · Experiment tree

Roots are expensive. Branches are nearly free.

Warm one 2,700-token root, then fork K short tails—ask the same context K different questions. The naive plan pays for the root K times; the shared-root plan pays once. This is the measured mechanism behind the 0.04–0.05× ratios: it turns “one careful question” into “a sweep of experiments” at almost the same price.

Naive: K full prompts
Shared root + K tails
—cheaper than paying for the root every time

Sequential forks reused ~100% of the root on every llama.cpp model tested (0.08–0.15× prompt ratio). The catch: parallel arrivals need slots—fan out wider than your slots and cold forks recompute the root anyway.

Interactive 06 · Slot theater

“Parallel” has four meanings on one Mac

Promise.all only submits requests together. Actual concurrency depends on loaded slots (OLLAMA_NUM_PARALLEL or llama-server -np), the model architecture, and whether memory bandwidth can feed them.

Client parallel, server serial

Five requests arrive together, but a single inference slot serves them one after another. Client wall time accumulates even when prefix reuse is excellent.

Interactive 07 · Compaction

Compression is buying a new root. Time the purchase.

A long agent session, turn by turn. The shaded area is total context (climbing toward the window). Bars are tokens evaluated per request—tiny while the prefix holds. Compacting shrinks the context but rewrites the prefix: one full re-evaluation spike, then cheap turns resume from the new, smaller root. Compact at a natural boundary—ideally right before a fan-out—never in the middle of one.

Total tokens evaluated—
Peak context—
Outcome—
total context resident tokens evaluated this turn window exceeded compaction event

Without compaction the session dies at the window edge—and every turn keeps re-reserving the largest KV allocation of the run.

Modeled illustration (8K window, 2,700-token root, 350-token turns, 1,100-token summary)—drawn to scale of the measured runs, not itself a measurement.

Interactive 08 · Memory budget

Context length is a reservation, not a badge

Our prompts were only about 2,500–2,900 tokens. Reserving 256K across five slots therefore bought no useful benchmark capacity while multiplying KV memory. The sharpest example was dense Gemma 4 31B: five 256K slots projected 108.4 GiB of KV cache before model weights and compute buffers. Pick a model, a context length, and parallel slots: black is weights, each blue block is one slot’s KV reservation, and the double line is the machine.

Weights—
KV reservation—
Total vs 64 GB—

Reservations are modeled from the run’s anchor points (illustrative, not remeasured). The lesson survives the error bars: unused context is a tax, and it compounds per slot.

Choose which scarce resource you are preserving: large context or parallel lanes. Maximizing both is usually an expensive way to process a short prompt.

Complete measurements

Same harness, both runtimes

Absolute wall times are not cross-runtime comparable—the llama.cpp sweep used a shorter prefix and three agents. Compare shape: sequential reuse, parallel contention, and whether a model that fails on one stack recovers on the other. Pick a view.

Shared-root sequential prompt cost vs that model’s cold control. Squares are Ollama; circles are llama.cpp. Left is better. Nemotron Nano 4B is the outlined row—the only clear verdict flip.

0× free0.25×0.5×0.75×1× full cold cost
Ollama (square) llama.cpp (circle) outlined = verdict flipped between runtimes

Rule 8, drawn: doubt the runtime before the weights. An Ollama no-reuse result is not a model property until a second runtime agrees.

Method and limits

What this benchmark does—and does not—say

Machine

Apple M4 Max with 64 GiB unified memory. Ollama 0.32.x and Homebrew llama-server (b9860) on the same box, same harness.

Prompts

Ollama matrix: ~2.5–2.9K tokens × seven scenarios. llama.cpp sweep: shorter ~4K-character prefixes, three agents, unique / shared-seq / shared-parallel.

Metrics

Client wall time and server prompt-evaluation time. llama.cpp adds timings.cache_n; Ollama reuse is inferred from prompt cost vs the unique-prefix control.

Limits

No answer-quality judging. Parallel contention can inflate prompt time. Do not rank absolute wall times across runtimes—prompt budgets differ.

Under Ollama, keep_alive retained the model, not a permanent cache namespace. Under llama.cpp, each sweep run injected a fresh nonce so prior KV state could not fake the unique-prefix control. Both stacks sent complete histories. Results are about resident prefix matching and scheduling on this machine, not a universal ranking of model intelligence.

THEFOCUS.AI LABS · RM-2026-07-KV · LOCAL MEASUREMENT