The Harness Experiment Q4 2026

Adapting Qwen 3.8 27B to the π Coding Harness on Apple Silicon

Testing harness-hub's RL LoRA for the π agent on a 64 GB M4 Max: why Q4_K_M hits 19.45 tok/s (+34% over Q8) while Q3_K_M regresses on Metal, how to keep .pi project-local with mise, and timing agent loops from llama.cpp Prometheus counters.

Stage EXPLORING

Most coding-agent work holds the model fixed and iterates on the harness—rewriting system prompts, compacting tool output, or wrapping brittle retry loops around model quirks.

Hugging Face’s harness-hub recently ran that arrow in the opposite direction with Harness Adapters: taking Qwen3.8-27B and training a lightweight RL + LoRA adapter (harness-hub/Qwen3.8-27B-pi-adapted-GGUF) specifically for the π (pi.dev) coding agent (@earendil-works/pi-coding-agent) and its native tools (read, bash, edit, write, grep, find, ls).

We wanted to see how that weight + harness pairing actually behaves on a developer laptop—not in a cloud cluster, and without polluting global agent config. Everything below comes from the companion workbench repo, The-Focus-AI/qwen-3.8-pi-adapted.

Flipping the arrow: adapting weights to the harness

On the upstream harness-hub/Qwen3.8-27B-pi-adapted benchmarks (measured as pass@1), adapting the model specifically to π’s tool contract moved both completion rate and tool discipline dramatically:

  • SWE-bench Pro (Hard 51): 29.4% → 62.7%
  • Terminal-Bench 2.1: 39.5% → 55.8%
  • Tool efficiency: 55–62% fewer tool calls per task

That last bullet matters most for local Apple Silicon inference. When single-stream (c1) decode runs at ~14–20 tokens per second, cutting tool-call thrashing in half has a larger wall-clock effect than squeezing another few percent out of GPU kernels.

Model & hardware physics on a 64 GB M4 Max

Inspecting the GGUF metadata reveals why this model has an unusual serving profile:

  • Architecture: qwen35 hybrid (gated delta-net / SSM layers interleaved with grouped-query attention), 64 layers, emb = 5120, FFN = 17408, 24 attention heads, 4 KV heads (head_dim = 256).
  • SSM state parameters: conv_k = 4, state = 128, groups = 16, dt_rank = 48, inner = 6144.
  • Dense active traffic: No MoE routing sparsity was observed in GGUF metadata—every decode token sweeps effectively all ~27B active parameters plus the 304 MB F16 π LoRA (Qwen3.8-27B-pi-lora-F16.gguf).
  • Concurrency bound: Because of the hybrid SSM layers, SSM state slots—not just KV cache tokens—constrain concurrency alongside prefix reuse (see our earlier study on local KV reuse for agents).

On a 64 GB M4 Max (16c CPU / 40c GPU, 546 GB/s spec LPDDR5X), Q8_0 + LoRA (~28.6 GB HF / ~27.3 GB GGUF) decoded at 13.8 tok/s baseline, implying ~395 GB/s effective Metal bandwidth (~72% of spec) and setting our practical c1 Q4 target around ~20–25 tok/s.

Quantization hill-climb: why Q4_K_M won and Q3_K_M regressed

Rather than guessing a quant, we ran a five-step hill-climb using sybil-solutions/inference-optimisation-skill against llama-server 1.0.0 (b1b770d90, Metal backend, 131,072 context, --reasoning off, π LoRA active, median of 6 c1 runs). Local Q5_K_M, Q4_K_M, and Q3_K_M variants were quantized from Q8_0 using llama-quantize --allow-requantize --leave-output-tensor:

Configc1 Decode (tok/s)vs. Q8 ctl-bShort TTFT (ms)Prefill 4K / 32K (tok/s)Verdict
Q8_0 + π LoRA (ctl-a)13.80—241.1171.6 / 161.1Control
Q5_K_M + π LoRA (local)15.18+5%234.1143.8 / 150.4Reject — slower than Q4
Q4_K_M + π LoRA (shipped)19.45+34%209.1190.7 / 172.4Accept (~78% of Q4 ceiling)
Q3_K_M + π LoRA (local)15.59+8%218.8185.5 / 166.3Reject — Metal Q3 dequant penalty
Q8_0 + π LoRA (ctl-b)14.48—224.8197.4 / 179.9End control

Two findings stood out right away:

  1. Smaller weights ≠ faster decode on Metal. Dropping from Q4_K_M to Q3_K_M cost almost 4 tok/s (19.45 → 15.59) because 3-bit K-quant unpacking overhead on Apple Metal kernels outweighs the memory bandwidth saved. Q4_K_M also posted the fastest short TTFT (209.1 ms) and strong 4K/32K prefill (190.7 / 172.4 tok/s).
  2. Speculative decoding (mtp/dflash) stays parked. At c1 on dense 27B Apple Silicon setups, draft overhead routinely loses against plain Q4_K_M decode, so we kept the server loop simple and predictable.

Quality caveat: Our local hill-climb requantized Q8_0 → Q4_K_M to iterate quickly on Metal throughput. For day-to-day use, mise run serve pulls upstream Qwen3.8-27B-Q4_K_M.gguf directly from Hugging Face (ggml-org/Qwen3.8-27B-GGUF, ~18 GB on disk) instead of double-quantizing.

Precise recipe: zero-global-state workbench (mise + .pi)

One recurring pain with local agent experiments is global configuration bleed. In qwen-3.8-pi-adapted, everything stays pinned inside the repo via mise and PI_CODING_AGENT_DIR:

# Prerequisites once per machine: mise, Homebrew, Hugging Face CLI (`hf`)
git clone https://github.com/The-Focus-AI/qwen-3.8-pi-adapted
cd qwen-3.8-pi-adapted

# 1. Install Node/Bun + run setup hook:
#    - brew install llama.cpp (if missing)
#    - compute CONTEXT from unified RAM minus model & 16 GB headroom → mise.local.toml
#    - install local @earendil-works/pi-coding-agent + .pi/ packages (pi-llama, pi-lens, pi-subagent)
mise install

# 2. Terminal A: auto-download Q4_K_M + F16 π LoRA (if missing) and serve on :9931
mise run serve

Under the hood, mise run serve unsets LLAMA_API_KEY on the server process (so unauthenticated /health probes aren’t rejected) and exposes Prometheus metrics and slot inspection:

llama-server \
  -m weights/Qwen3.8-27B-Q4_K_M.gguf \
  --lora weights/Qwen3.8-27B-pi-lora-F16.gguf \
  --reasoning off \
  -c "$CONTEXT" \
  --port 9931 \
  --alias qwen3.8-pi \
  --metrics \
  --slots \
  --log-file logs/llama-server.log

In a second terminal, talk to local/qwen3.8-pi without touching ~/.pi:

mise run pi                       # interactive REPL inside this repo
mise run pi:tmp                   # mktemp -d + symlinked .pi plugins for clean coding tasks
mise run ask -- "Write a streaming Bun CLI against http://127.0.0.1:9931/v1"
mise run stats:watch              # poll /metrics + /slots into logs/serve-stats.jsonl
mise run demo:grafana             # foreground Prometheus + Grafana dashboard at :3000

Timing the agent from the outside (llama-server /metrics Δ)

π does not report how long a multi-turn task took or how much time went into prefill vs. decode vs. bash tools. To measure agent loops cleanly, mise run challenge -- <name> spins up a symlinked temp workdir, runs pi with --approve --mode json --no-session --tools read,bash,edit,write,grep,find,ls, streams live tool plans through scripts/challenge-stream.py, and diffs llama-server’s /metrics counters before and after (llamacpp:prompt_tokens_total, llamacpp:prompt_tokens_cached_total, llamacpp:tokens_predicted_total, llamacpp:prompt_seconds_total, and llamacpp:tokens_predicted_seconds_total).

Results so far (logs/challenges.jsonl, stage: exploring):

  • health-and-models (passed, exit_code: 0): Wrote and verified a Bun /health + /v1/models checker in 43.01 s wall clock across 4 turns (write: 1, edit: 1, bash: 1). Inference accounted for 3,605 prompt tokens (17.22 s, 209.34 tok/s) and 424 decode tokens (21.17 s, 20.03 tok/s)—meaning ~89% of end-to-end task time was Metal inference and ~4.6 s went to tool execution.
  • labs-sqlite-search (in progress, interrupted exit_code: 130): A harder Bun + bun:sqlite + Ollama embeddinggemma-2 crawl-and-search challenge (pairing with our EmbeddingGemma 2 + SQLite report). Early turns streamed at 220.88 tok/s prefill and 19.69 tok/s decode before interruption.

What stays on the bench vs. what could graduate

Right now this sits squarely on The Lab Bench as exploring:

  1. Next bench checks: Finish timed end-to-end runs across all five repo challenges (health-and-models, labs-sqlite-search, pelican-bicycle, stream-chat-cli, dual-endpoint-compare), test flash-attn / ubatch and KV cache dtype flags on Q4_K_M, and A/B the π LoRA against mise run serve:raw (base Qwen 3.8 without LoRA) and mise run serve:bonsai (Ternary-Bonsai Q2_0, ~7.6 GB).
  2. Candidates for standards.thefocus.ai: Repo-scoped .pi + mise.local.toml unified-memory CONTEXT sizing + external /metrics delta timing is already clean enough to inform how we wrap local agent harnesses across repos.
  3. Candidates for courses.thefocus.ai: Pairing harness-adapted RL LoRAs with local Metal serving makes a natural hands-on follow-up alongside Run AI Locally and Fine-Tune a Local Agent.
Newsletter · Practice Before Theory

Get the next dispatch in your inbox.

Field reports, benchmarks, and bench recipes from TheFocus.AI Labs. Irregular cadence — sent when we have real measurements or patterns worth sharing.

Loading archive index…