The Coding Agent Harness Math: 29% to 85%

    Claude Haiku 5.5 finished a 122-step job about 29% of the time inside a homegrown coding agent harness, according to the builder who posted the entire build to r/AI_Agents. The same job, restructured into short phases with fresh context, landed nearly 85%. That gap is the whole case for caring about the harness layer: the software wrapped around the model that owns the call loop, the tool list, the spend caps, and the permission gates. Nothing about the model changed between those two numbers. The layer around it did, and the layer is the part you actually control.

    If you run coding agents on real work and jobs die quietly at step 90, the model is the wrong suspect. The arithmetic usually is.

    What the harness layer actually is

    The build in question is almost embarrassingly small: TypeScript with the Anthropic SDK as its only dependency, running Claude Haiku 5.5. The parts list from the post reads like a checklist a sysadmin would write, not a research paper:

    – A while loop around one model call
    – One tool with an inner tool loop
    – A registry of 5 file and shell tools
    – Caps on both steps and tokens
    – A yes/no gate before file writes and bash commands
    – Context management: prompt caching, cutting tool results over 12,000 characters where they’re made, and server-side compaction at 90k tokens

    That’s the layer.

    The intelligence is rented by the token from whoever trained the model.

    And everything that makes an agent survivable in a real repository is this scaffolding. It’s also the part nobody demos on stage, because “I added a confirmation gate” does not make a keynote.

    My take: when people say coding agents aren’t reliable, they’re usually describing a harness problem and billing it as a model problem.

    The report above is the cleanest demonstration I’ve seen, as the model never changed.

    The math that kills one long loop

    The builder’s numbers: at 99% success per step, a 122-step job run as one long loop finishes about 29% of the time. That’s just compounding. Ninety-nine percent sounds like a grade you’d frame. But multiply 0.99 by itself 122 times and you’re left with roughly 0.29. One wobble anywhere in the chain and the whole run is dead, which is why long agentic jobs fail in a way that feels mysterious when you look at any single step.

    The fix, per the same report, was not a smarter model or a bigger context window. It was phasing: each phase a fresh loop with an empty context, returning a 5-line note to the next phase, with one retry each. Run that way, the job lands near 85%. The builder’s summary is the sentence worth taping to your monitor: “Compaction fixed my cost but not my success rate. Only phases fixed both.”

    Note what didn’t work. The build already had aggressive context management: tool results cut at 12,000 characters, compaction triggered server-side at 90k tokens. Those changes made the bill smaller. They did nothing for completion until the work itself was split up.

    This is where I part ways with the sales pitch. The industry answer to long jobs has been bigger context windows, as if the fix for a 122-step task is more room for it to go wrong in. The reported result points the opposite way: shrink the job, hand off a short note, start clean. I’d bet most “agent reliability” complaints are phase design complaints in disguise.

    Small models drown in harness bloat

    A second thread, on harnesses for very small local models, names Qwen 8B as the test case and diagnoses the same layer from the other end: “8b models fall apart in standard harnesses since the system prompt and 20+ tool schemas eat up their attention budget before they even read the file.” The model isn’t too dumb to code.

    It’s too busy parsing your apparatus to get to the file.

    The proposed fix in that thread was to strip down to 3 tools, `rg`, `read_lines`. And `apply_diff`, add grammar-constrained JSON output, and retry automatically through a linter. The claim: that combination makes 7B-8B models way more usable. A separate project takes context from another angle: Enola builds a deterministic graph of the codebase and exposes it to agents through MCP. So the agent queries structure instead of re-reading half the repo to find it.

    Same layer, two failure modes.

    A frontier model burns money on bloat; a small model fails outright on it.

    For a small operator, this is the whole difference between a local model being a toy and being a tool. And the fix is configuration, not a bigger subscription.

    What to do about it this week

    Here’s the checklist I’d run against any coding agent setup, built from the numbers above:

    – Count the tools. If a small model is staring at 20+ schemas, cut to a handful covering search, read, and edit.
    – Phase long jobs. Fresh context per phase, a short handoff note, one retry each.
    – Put writes and bash behind a yes/no gate before you let anything run unattended.
    – Set step and token caps so a confused loop fails loudly instead of expensively.
    – Truncate noisy tool output at the source; the reported build cut results over 12,000 characters.
    – Prefer deterministic context, like a codebase graph served over MCP, over letting the agent re-read everything.

    The layer is too becoming a product you can swap rather than build. Distill is an open-source (Apache-2.0) coding agent harness and TUI whose author says it supports Grok and Codex subscription logins and takes any OpenAI-compatible model. That’s the direction this whole market is headed: the model becomes the replaceable part. And your phase design, tool list, and permission gates become the asset you keep.

    Run the experiment yourself.

    Take your longest failing agent job, split it into phases with fresh context and one retry each.

    And count completions before and after. If your numbers move anything like the reported 29% to nearly 85%, you just fixed reliability without spending a dollar on a new model. I want to hear what your before and after looks like.

    Leave a Reply

    Your email address will not be published. Required fields are marked *