Adapting Qwen 3.8 27B to the π Coding Harness on Apple Silicon
Testing harness-hub's RL LoRA for the π agent on a 64 GB M4 Max: why Q4_K_M hits 19.45 tok/s (+34% over Q8) while Q3_K_M regresses on Metal, how to keep .pi project-local with mise, and timing agent loops from llama.cpp Prometheus counters.
Most coding-agent work holds the model fixed and iterates on the harness—rewriting system prompts, compacting tool output, or wrapping brittle retry loops around model quirks.
Hugging Face’s harness-hub recently ran that arrow in the opposite direction with Harness Adapters: taking Qwen3.8-27B and training a lightweight RL + LoRA adapter (harness-hub/Qwen3.8-27B-pi-adapted-GGUF) specifically for the π (pi.dev) coding agent (@earendil-works/pi-coding-agent) and its native tools (read, bash, edit, write, grep, find, ls).
We wanted to see how that weight + harness pairing actually behaves on a developer laptop—not in a cloud cluster, and without polluting global agent config. Everything below comes from the companion workbench repo, The-Focus-AI/qwen-3.8-pi-adapted.
Flipping the arrow: adapting weights to the harness
On the upstream harness-hub/Qwen3.8-27B-pi-adapted benchmarks (measured as pass@1), adapting the model specifically to π’s tool contract moved both completion rate and tool discipline dramatically:
- SWE-bench Pro (Hard 51):
29.4% → 62.7% - Terminal-Bench 2.1:
39.5% → 55.8% - Tool efficiency:
55–62%fewer tool calls per task
That last bullet matters most for local Apple Silicon inference. When single-stream (c1) decode runs at ~14–20 tokens per second, cutting tool-call thrashing in half has a larger wall-clock effect than squeezing another few percent out of GPU kernels.
Model & hardware physics on a 64 GB M4 Max
Inspecting the GGUF metadata reveals why this model has an unusual serving profile:
- Architecture:
qwen35hybrid (gated delta-net / SSM layers interleaved with grouped-query attention), 64 layers,emb = 5120,FFN = 17408, 24 attention heads, 4 KV heads (head_dim = 256). - SSM state parameters:
conv_k = 4,state = 128,groups = 16,dt_rank = 48,inner = 6144. - Dense active traffic: No MoE routing sparsity was observed in GGUF metadata—every decode token sweeps effectively all ~27B active parameters plus the 304 MB F16 π LoRA (
Qwen3.8-27B-pi-lora-F16.gguf). - Concurrency bound: Because of the hybrid SSM layers, SSM state slots—not just KV cache tokens—constrain concurrency alongside prefix reuse (see our earlier study on local KV reuse for agents).
On a 64 GB M4 Max (16c CPU / 40c GPU, 546 GB/s spec LPDDR5X), Q8_0 + LoRA (~28.6 GB HF / ~27.3 GB GGUF) decoded at 13.8 tok/s baseline, implying ~395 GB/s effective Metal bandwidth (~72% of spec) and setting our practical c1 Q4 target around ~20–25 tok/s.
Quantization hill-climb: why Q4_K_M won and Q3_K_M regressed
Rather than guessing a quant, we ran a five-step hill-climb using sybil-solutions/inference-optimisation-skill against llama-server 1.0.0 (b1b770d90, Metal backend, 131,072 context, --reasoning off, π LoRA active, median of 6 c1 runs). Local Q5_K_M, Q4_K_M, and Q3_K_M variants were quantized from Q8_0 using llama-quantize --allow-requantize --leave-output-tensor:
| Config | c1 Decode (tok/s) | vs. Q8 ctl-b | Short TTFT (ms) | Prefill 4K / 32K (tok/s) | Verdict |
|---|---|---|---|---|---|
Q8_0 + π LoRA (ctl-a) | 13.80 | — | 241.1 | 171.6 / 161.1 | Control |
Q5_K_M + π LoRA (local) | 15.18 | +5% | 234.1 | 143.8 / 150.4 | Reject — slower than Q4 |
Q4_K_M + π LoRA (shipped) | 19.45 | +34% | 209.1 | 190.7 / 172.4 | Accept (~78% of Q4 ceiling) |
Q3_K_M + π LoRA (local) | 15.59 | +8% | 218.8 | 185.5 / 166.3 | Reject — Metal Q3 dequant penalty |
Q8_0 + π LoRA (ctl-b) | 14.48 | — | 224.8 | 197.4 / 179.9 | End control |
Two findings stood out right away:
- Smaller weights ≠ faster decode on Metal. Dropping from
Q4_K_MtoQ3_K_Mcost almost 4 tok/s (19.45 → 15.59) because 3-bit K-quant unpacking overhead on Apple Metal kernels outweighs the memory bandwidth saved.Q4_K_Malso posted the fastest short TTFT (209.1 ms) and strong 4K/32K prefill (190.7 / 172.4 tok/s). - Speculative decoding (
mtp/dflash) stays parked. Atc1on dense 27B Apple Silicon setups, draft overhead routinely loses against plainQ4_K_Mdecode, so we kept the server loop simple and predictable.
Quality caveat: Our local hill-climb requantized
Q8_0 → Q4_K_Mto iterate quickly on Metal throughput. For day-to-day use,mise run servepulls upstreamQwen3.8-27B-Q4_K_M.ggufdirectly from Hugging Face (ggml-org/Qwen3.8-27B-GGUF, ~18 GB on disk) instead of double-quantizing.
Precise recipe: zero-global-state workbench (mise + .pi)
One recurring pain with local agent experiments is global configuration bleed. In qwen-3.8-pi-adapted, everything stays pinned inside the repo via mise and PI_CODING_AGENT_DIR:
# Prerequisites once per machine: mise, Homebrew, Hugging Face CLI (`hf`)
git clone https://github.com/The-Focus-AI/qwen-3.8-pi-adapted
cd qwen-3.8-pi-adapted
# 1. Install Node/Bun + run setup hook:
# - brew install llama.cpp (if missing)
# - compute CONTEXT from unified RAM minus model & 16 GB headroom → mise.local.toml
# - install local @earendil-works/pi-coding-agent + .pi/ packages (pi-llama, pi-lens, pi-subagent)
mise install
# 2. Terminal A: auto-download Q4_K_M + F16 π LoRA (if missing) and serve on :9931
mise run serve
Under the hood, mise run serve unsets LLAMA_API_KEY on the server process (so unauthenticated /health probes aren’t rejected) and exposes Prometheus metrics and slot inspection:
llama-server \
-m weights/Qwen3.8-27B-Q4_K_M.gguf \
--lora weights/Qwen3.8-27B-pi-lora-F16.gguf \
--reasoning off \
-c "$CONTEXT" \
--port 9931 \
--alias qwen3.8-pi \
--metrics \
--slots \
--log-file logs/llama-server.log
In a second terminal, talk to local/qwen3.8-pi without touching ~/.pi:
mise run pi # interactive REPL inside this repo
mise run pi:tmp # mktemp -d + symlinked .pi plugins for clean coding tasks
mise run ask -- "Write a streaming Bun CLI against http://127.0.0.1:9931/v1"
mise run stats:watch # poll /metrics + /slots into logs/serve-stats.jsonl
mise run demo:grafana # foreground Prometheus + Grafana dashboard at :3000
Timing the agent from the outside (llama-server /metrics Δ)
π does not report how long a multi-turn task took or how much time went into prefill vs. decode vs. bash tools. To measure agent loops cleanly, mise run challenge -- <name> spins up a symlinked temp workdir, runs pi with --approve --mode json --no-session --tools read,bash,edit,write,grep,find,ls, streams live tool plans through scripts/challenge-stream.py, and diffs llama-server’s /metrics counters before and after (llamacpp:prompt_tokens_total, llamacpp:prompt_tokens_cached_total, llamacpp:tokens_predicted_total, llamacpp:prompt_seconds_total, and llamacpp:tokens_predicted_seconds_total).
Results so far (logs/challenges.jsonl, stage: exploring):
health-and-models(passed,exit_code: 0): Wrote and verified a Bun/health+/v1/modelschecker in 43.01 s wall clock across 4 turns (write: 1,edit: 1,bash: 1). Inference accounted for3,605prompt tokens (17.22 s, 209.34 tok/s) and424decode tokens (21.17 s, 20.03 tok/s)—meaning ~89% of end-to-end task time was Metal inference and ~4.6 s went to tool execution.labs-sqlite-search(in progress, interruptedexit_code: 130): A harder Bun +bun:sqlite+ Ollamaembeddinggemma-2crawl-and-search challenge (pairing with our EmbeddingGemma 2 + SQLite report). Early turns streamed at 220.88 tok/s prefill and 19.69 tok/s decode before interruption.
What stays on the bench vs. what could graduate
Right now this sits squarely on The Lab Bench as exploring:
- Next bench checks: Finish timed end-to-end runs across all five repo challenges (
health-and-models,labs-sqlite-search,pelican-bicycle,stream-chat-cli,dual-endpoint-compare), testflash-attn/ubatchand KV cache dtype flags onQ4_K_M, and A/B the π LoRA againstmise run serve:raw(base Qwen 3.8 without LoRA) andmise run serve:bonsai(Ternary-BonsaiQ2_0, ~7.6 GB). - Candidates for
standards.thefocus.ai: Repo-scoped.pi+mise.local.tomlunified-memoryCONTEXTsizing + external/metricsdelta timing is already clean enough to inform how we wrap local agent harnesses across repos. - Candidates for
courses.thefocus.ai: Pairing harness-adapted RL LoRAs with local Metal serving makes a natural hands-on follow-up alongside Run AI Locally and Fine-Tune a Local Agent.
Neural harness
I'm on my second Claude Max plan. Are we going to go for three? Tokens are the bill for running a new native unit of computing. The harness is to neural nets what the compiler is to source code.
Where A2A Begins and Ends
A2A hit v1.0 in April, and MCP's extension catalog made the two protocols look alike on paper. The actual boundary is stance: compose a transparent workbench, or delegate to an opaque peer.
Agents Get a UI
MCP was the USB port for agents. The 2026-07-28 revision gives that port a screen, with host-mediated widgets in the chat, plus extensions for tasks, auth, and whatever comes next.