KV-cache quantization on AWS Graviton4 CPU

Quantizing the KV cache (q8_0) at 8k context: +101.5% prefill, -27.8% decode, -8.9% memory vs f16.

Qwen3-4B-Instruct-2507c8g.4xlarge · Graviton4N=10 · warmup discardedllama.cpp 2d973636eKleidiAI on · fa on · seed 42data @ 1baf1af ↗

delta vs f16/f16 baseline · same context · median of N=10

Error bars: propagated stdev from rep-level spreads. Positive prefill and negative memory are improvements; negative decode is the cost. Blue = better, orange = worse.

The five-config, four-context, six-model matrix is complete on c8g (Graviton4); every band is generated from the canonical results by script, nothing is hand-typed. Every number traces to its exact committed source: commit 1baf1af, the CI runs, and the per-model source files in the methodology section.

model
00start here

Which config for your job?

Fastest prefill at 8k
q8_0/q4_0 K8 V4
+103% prefill vs f16, and the prefill bill drops to $3.96 per 1M tokens.
8kspeedprefill
Biggest memory saving at 32k
q4_0 K4 V4
-35% peak memory vs f16, the most room for long context on small RAM.
32kmemory
Safest quality
q8_0 K8 V8
Lowest divergence from f16 (KLD 0.0037). 8-bit keys stay safe on every model measured.
quality8-bit key
Best all-around at 16k
q4_0 K4 V4
+114% prefill, -31% decode, -23% memory vs f16.
16kbalanced

Computed from the measured cells for this model, the same transparent logic the recommend_config MCP tool uses. Switch models above to rescope.

01the tradeoff surface

Which config should you run?

02context scaling

How the trade moves with context length

03cost economics

The tradeoff, in dollars per million tokens

04quality cliffs

Where quantization breaks retrieval

05cell explorer

Every cell, down to the raw reps

06methodology and reproduction

How these numbers were made

Instancec8g.4xlarge (16 vCPU, 30.7 GB), on-demand, us-east-1
CPUNeoverse-V2 (Graviton4)
OS / kernelUbuntu 24.04.4 LTS / 6.17.0-1019-aws
llama.cpp2d973636e292ee6f75fadcf08d29cb33511f509f (pinned; never upgraded)
Build-DCMAKE_BUILD_TYPE=Release -DGGML_CPU_KLEIDIAI=ON
AccelerationKleidiAI = 1 verified in system_info before every cell; reported as the baseline it is
Flash attentionon for every run including f16 baselines
ModelQwen3-4B-Instruct-2507 Q4_K_M, sha256 3605803b982cb64a...
Run standardN=10 reps where a cell costs under 10 minutes, N=5 minimum, first rep discarded as warmup, median + stdev + CV, seed 42
Measurementone llama-completion pass per rep: prefill and decode throughput from the perf lines, peak RSS from child rusage, KV buffer from the load log

Reproduce clean-room on any arm64 instance (Graviton4 for real numbers). One command builds the pinned llama.cpp with KleidiAI, downloads a small model, runs the harness, and validates the output against the schema. It mirrors CI:

git clone https://github.com/StephenSook/gravitonkv
cd gravitonkv
./scripts/reproduce.sh

Every number traces to its exact committed source. The data on this page was generated from commit 1baf1af; each per-model result file:

Ask the data directly: a read-only MCP server serves these results. Remote endpoint https://gravitonkv-mcp.vercel.app/mcp (paste into Claude's custom connectors), or run it over stdio with npx @gravitonkv/mcp.

CI runs this harness end to end on GitHub's arm64 runners (Azure Cobalt 100, Neoverse N2) and validates the output against the canonical JSON schema on every push. CI proves the harness; it never produces findings: headline numbers come only from Graviton4.

Mechanism notes from reading the pinned source: docs/mechanism-source-notes.md. Related work and citations live in the repository README.