TuneForge

Agent-driven, evidence-based LLM inference tuning for Arm64 cloud platforms.

Point TuneForge at a GGUF model on an Arm Neoverse instance (AWS Graviton, GCP Axion, Azure Cobalt). It runs a disciplined tuning loop — diagnose the bottleneck from hardware counters → change exactly one knob → measure → accept or reject → repeat — and emits two artifacts:

• recipe.yaml — a replayable, verifiable optimization recipe: every accepted change with its hypothesis, measured gain, and full environment provenance. tuneforge apply recipe.yaml re-measures it on your host and verifies the claim.
• report.md — the human-readable narrative: before/after table, the step-by-step reasoning, cost-per-million-tokens analysis, and honest methodology caveats.

The proposer can be a deterministic coordinate-descent search (zero API dependencies) or a Claude-powered performance engineer that reads PMU evidence (backend stalls, cache misses, IPC) and picks the next experiment — with every guardrail (single-variable steps, budgets, noise thresholds, the quality gate) enforced in deterministic code, not by the model.

Real end-to-end session, Google Axion (c4a-standard-16, Neoverse V2), Llama-3.2-3B Q4_0 — full record in docs/labnotes/2026-07-09-generic-baseline-e2e.md:

$ tuneforge run Llama-3.2-3B-Instruct-Q4_0.gguf --llama-cpp ~/llama.cpp \
    --baseline generic --quality-dataset wiki.test.raw --hourly-usd 0.72

#0  kleidiai=True      ->  -4.9% rejected   ← yes, really: see "Field notes" below
#1  mcpu='native'      -> +72.3% ACCEPTED   the portable-binary tax, measured
#2  mcpu='neoverse-v2' ->  -0.3% rejected
#7  threads=16         ->  -3.2% rejected   (default thread policy proven right, not assumed)
#9  ubatch=1024        ->  +0.1% rejected
...
tg_tps: 48.7 → 84.0 (×1.72)   pp_tps: 76.7 → 289.9 (×3.78)   recipe: runs/.../recipe.yaml

tg_tps is token generation throughput (tokens/second while writing the answer) and pp_tps is prompt processing throughput (tokens/second while reading the prompt). They move independently, which is why the tool tracks both.

Measured results (2026-07-09, Google Axion c4a-standard-16)

tier	tg t/s	pp t/s	TTFT*	$/Mtok @ $0.72/hr
Generic portable build — what multi-arch Docker images actually ship	48.7	76.7	6678 ms	$4.10
Upstream default native build	75.9	265.7	1927 ms	$2.64
TuneForge-tuned	84.0	289.9	1766 ms	$2.38

Generic → tuned: ×1.72 generation, ×3.78 prompt processing, TTFT −74%, generation cost −42% — measured end-to-end by a single --baseline generic session, one attributed step at a time (generic and tuned rows above; the native-tier row is from the same-day default-baseline session, see lab notes for both artifacts). (*TTFT derived from prompt-processing throughput; methodology and caveats in the report.)

Not just LLMs — three modalities, one tool, one credibility floor

The same orchestrator drives a second engine (ONNX Runtime) across vision and NLP. What ties them together is the quality gate: the biggest lever (int8) is offered every time, and the gate arbitrates each case on its own merits — sometimes accept, sometimes reject, sometimes only via the right recipe. All measured on AWS Graviton4 c8g.4xlarge; full recipes in recipes/.

modality	model / engine	biggest int8 lever	gate verdict
LLM	Llama-3.2-3B Q4_0 · llama.cpp	×1.83 tg (build + threads + mcpu)	✅ accepted, drift-certified
Vision (CNN)	ResNet50 · ONNX Runtime	×2.11 img/s, static int8-qdq	✅ accepted (−2.6pp top-1, within gate)
Vision (CNN)	MobileNetV4-small · ONNX Runtime	+147% offered, static int8-qdq	❌ rejected (−4.6% top-1 > 3% floor) — ships fp32
NLP (transformer)	DistilBERT SST-2 · ONNX Runtime	×3.08 img/s, int8-dynamic	✅ accepted (90.5% acc, −0.6%)

The NLP row is the interesting one: naive static int8 (the CNN recipe) breaks a transformer — DistilBERT scored 49% (chance) because activation outliers collapse the calibration range to a NaN scale. The tool carries a second, transformer-appropriate int8 lever (int8-dynamic: int8 weights, activations quantized at runtime); the agent picks it by model class, and the gate confirms it at 90.5% accuracy. Same family of levers, three models, three different answers. Nothing but measurement separates them.

Why

CPU LLM inference on Arm64 has a large, under-explored configuration space: KleidiAI micro-kernels, -mcpu targeting, thread count vs. memory bandwidth, micro-batch sizing, KV-cache quantization, core pinning, allocators. Getting it right is worth 1.5–3× throughput — but hand-tuning it takes an engineer-day per model × instance combination, and most deployments simply don't.

TuneForge automates that engineer, and keeps the two things hand-tuning often loses: evidence (every change is justified by hardware counters and measured with median-of-N + stddev discipline) and honesty (a quality gate rejects any "optimization" that trades model quality for speed — perplexity must stay within threshold of baseline).

Quick start (any Arm64 Linux box — tested on AWS Graviton and GCP Axion)

Prefer a guided walkthrough with explanations at every step? Follow the learning path (~15 min demo, ~45 min full session).

Five-minute look — the full pipeline on a toy model, cheapest possible instance (a $0.1/hr spot c8g.2xlarge is plenty):

git clone https://github.com/edenfunf/tuneforge && cd tuneforge
MINIMAL=1 bash scripts/setup_graviton.sh   # toolchain + llama.cpp checkout, no model
. .venv/bin/activate
tuneforge run --demo                        # tiny model, real loop: TUI → recipe → report

Real tuning run — the setup script records install paths in ~/.tuneforge/defaults.json, so only the model is needed:

bash scripts/setup_graviton.sh              # + 1.9 GB model + quality dataset
tuneforge run ~/models/Llama-3.2-3B-Instruct-Q4_0.gguf --hourly-usd 0.63

The quality gate finds its dataset automatically (disable with --no-quality-gate; override anything with the usual flags).

For the Claude-powered proposer:

pip install -e ".[agent]"
export ANTHROPIC_API_KEY=sk-ant-...
tuneforge run ... --strategy agent

Other commands: tuneforge knobs (the catalog and per-host availability), tuneforge env (host fingerprint), tuneforge apply recipe.yaml --llama-cpp ~/llama.cpp (replay + verify a recipe), tuneforge report <run-dir> (regenerate the report).

Supported hosts — what's verified vs. what degrades

Benchmarking needs Linux/aarch64; everything else (tests, knobs, report, recipe inspection) runs on any OS. Where a host can't provide something, you get a warning and a gap in the evidence, never a number we made up.

tier	hosts	what you get
Verified on hardware	AWS Graviton4 (Neoverse V2, Ubuntu 24.04) · GCP Axion (Neoverse V2)	Everything. On Graviton4 that includes full topdown counter evidence, cross-checked against Arm's own topdown-tool. Axion tunes fine but its hypervisor exposes no CPU PMU, so those sessions run without hardware evidence.
Expected to work	Any Neoverse server on Linux/aarch64 (Graviton2/3, Azure Cobalt, Ampere Altra, …)	The full tuning loop. CI runs the documented setup and a real two-step session on Neoverse N2 hardware every push, where KleidiAI measured −5.1% against −4.9% on V2 — the same finding, one generation apart. Topdown slot constants for N2-class cores are not hardware-validated yet (that runner exposes no PMU); if the buckets stop adding up to ~100%, the tool says so and you should treat the percentages as approximate.
Degraded by design	Unknown Arm cores (Raspberry Pi-class, new silicon), hosts without perf/PMU	Tuning still works. Hardware evidence falls back to generic stalled-cycle counters, or is left out entirely, and host-specific knob values like mcpu=neoverse-v2 are only offered where they apply.
Not supported	macOS / Windows / x86 benchmarking	A clear refusal saying why. The pure-logic commands still work.

The setup script targets Ubuntu 24.04 (the images all published numbers come from) and says so explicitly if it finds a different package manager. Models larger than ~80% of host RAM trigger a warning before any benchmark runs — an mmap'd oversized model measures paging, not inference.

How it works

┌────────────┐   propose one    ┌──────────────────────────────────────┐
│  Strategy  │──(knob, value)──▶│        TuningSession (the loop)      │
│ coordinate │                  │  budgets · single-variable · accept  │
│ or Claude  │◀──full history───│  rule · quality gate · persistence   │
└────────────┘  + HW evidence   └──────┬───────────────────────────────┘
                                       │ measure candidate
                                ┌──────▼───────────┐   wraps exact cmd   ┌─────────────┐
                                │ MeasurementEngine│────────────────────▶│ perf stat / │
                                │ warmup+5 runs,   │                     │  Performix  │
                                │ median±stddev    │                     └─────────────┘
                                └──────┬───────────┘
                                       │ build/run/parse
                                ┌──────▼───────────┐
                                │ LlamaCppAdapter  │  (vLLM/ORT: implement one class)
                                └──────────────────┘

Design decisions and their rationale (why the LLM proposes but never controls the loop, why medians, why the quality gate is non-negotiable) are documented in docs/ARCHITECTURE.md.

Field notes: why evidence beats folklore

Day one on real Neoverse V2 silicon, the tuner rejected the single most widely-recommended "Arm optimization" — enabling the KleidiAI micro-kernels — because it measured −14.3% on this host/model/engine combination (and −4.9% again from the generic build tier, its second data point). Both paths were verified engaged (KleidiAI's I8MM kernels vs. upstream ggml's CPU_REPACK kernels, themselves contributed by Arm engineers). Public tutorials still recommend the flag based on 2024-era baselines; upstream moved.

The same discipline cuts both ways: the bring-up session "found" +8.7% from explicit thread pinning, and the very next session on the same host proved it had been warm-up noise against a fresh-boot baseline (retracted in the lab notes, which keep both data points). A tuner you can trust has to be able to retract its own findings.

That is the point of this tool: the fastest path flips with host × model × engine version, faster than documentation can follow. Measure, don't cargo-cult — and keep the negative results, because knowing what not to enable is worth as much as the wins.

Honest numbers, by construction

• Median of ≥5 runs after warmup; relative stddev recorded; unstable configurations get extra runs and are flagged in the report.
• Single variable per step — enforced by the orchestrator, so every gain in the recipe has exactly one cause.
• Quality gate — MODEL-layer changes (KV-cache type, etc.) must keep wikitext perplexity within 3% of baseline or they are rejected, no matter how fast they are.
• Derived TTFT is labeled derived, baseline is the default upstream build (exactly what cmake -B build && cmake --build build gives you), and recipes embed the full environment fingerprint.
• Predictions are graded on mechanism, not just outcome. An agent proposal names the topdown bucket it claims to move, and the orchestrator checks that against the measured counters. "Throughput rose, but the bucket you named never moved" shows up in the report as being right for the wrong reason.
• The tool re-tests its own early verdicts. Greedy search judges each knob against the incumbent of the moment, so a rejection from step 0 was measured on a machine that no longer exists by the end. After converging, TuneForge re-measures the stalest rejections against the final configuration — a confirmed rejection is one that held at two different points in the configuration space, and a flip is a gain greedy order had hidden.

Requirements

• Benchmarking: Linux on aarch64 (Neoverse-based cloud instances or any Arm64 Linux box), cmake + a C/C++ toolchain, optionally perf for hardware evidence.
• The CLI's pure-logic commands and the test suite run anywhere Python 3.10+ runs.

License

MIT — see LICENSE.