Model capability is only half of what you get from a coding agent. The other half is the harness: the loop, the system prompt, the tools, the permission rules, how it remembers things, how it shrinks its history when the context fills up, and what it asks the API to cache.

To measure that layer, I fixed the model (Claude Sonnet 5.5) and ran five harnesses on the same tasks with the same hidden tests: OpenCode, Claude Code, Pi, Oh My Pi, and GitHub Copilot's agent in VS Code (run by hand). Four rounds of tests, from 14 short tasks to a five-repository incident with a library migration. Nothing changed between runs except the harness.

I also made a first pass with Codex on GPT 6.1 Sol. It ran on a different model, so it is not a like-for-like comparison, and I dropped it from the later rounds because its accuracy was below the others'. It is not part of the results below.

The short answer: Pi. It solved what the others solved, at the lowest cost, with the fewest model calls and the least output, and it was the fastest or tied for fastest in every round. Accuracy barely separated the harnesses. Cost moved by about ten times.

TL;DR

• Short tasks: on 14 hard coding tasks (about 36 runs per harness), Pi and Oh My Pi passed 36 of 36, Claude Code 35 of 36 and OpenCode 35 of 37. Mean time per task ranged from 13 s (Pi) to 29 s (Claude Code).

• Long stress task (70,000 tokens of specs, injected rate-limit errors, a prompt injection, a context cap): a run cost about $2.50 on Pi, $5 to $6 on Oh My Pi and $22 to $29 on Claude Code. GitHub Copilot in VS Code measured $8.75 on the Copilot credit counter (one run, run by hand).

• Multi-repo debugging (three versions, 27 runs of Pi, Oh My Pi and OpenCode): 25 passed everything. Both misses were Oh My Pi slipping on a written rule, never wrong numbers. Cost per run: Pi $0.95, $1.90 and $3.35; Oh My Pi $1.47, $2.85 and $3.84; OpenCode $2.33, $4.05 and $5.33.

• Why the bills differ: prompt caching and call count. Pi and Oh My Pi reused 82 to 94% of their prompt. Claude Code reused 59 to 63%, so most of its growing conversation was billed as fresh input. OpenCode reused 93 to 96% but made about twice as many model calls as Pi. Time follows output tokens, about 7 to 8 ms each.

• Safety and resilience: no harness followed the instruction hidden in a data file or log, and Pi, Oh My Pi and OpenCode recovered from every injected 429 error (OpenCode was only testable in the multi-repo rounds).

• An independent check: Composio's study of eight harnesses on Kimi K3 found a similar shape on different tasks: Oh My Pi had the highest pass rate, Pi was the fastest and used the fewest tokens, Claude Code cost the most. Their head-to-head of the two (Pi vs OMP, 30 tool-use tasks on DeepSeek V4 Flash) had Pi ahead, 20 passes to 17, at about $0.03 against $0.10 per success. Different models and tasks, same direction on cost.

Scoreboard

Everything in one place. Costs are dollars per passing run (1 credit = $0.10), estimated from proxy logs unless noted.

Graded runs are the 14 short tasks, the long task and the multi-repo task, minus runs removed as invalid. Cost per success is the total cost of a harness's runs divided by its passing runs, so Claude Code's long-task figure counts both of its runs against its one pass. The multi-repo figures average three task versions of three runs each, with the arms running in parallel.

How I evaluated each harness

One model, pinned settings. Claude Sonnet 5.5 for everyone. For the stress and multi-repo tasks I pinned High reasoning effort on every automated harness using each one's own flag, and a context cap (100k, later 64k). Changing either invalidates a prompt cache, so leaving them at defaults would compare unlike things.

Round 1: 14 hard tasks. Features from precise specs, tests that must catch hidden bugs, concurrency, debugging from symptoms, performance, scope discipline, and rule-heavy specs with hundreds of hidden cases.

Round 2: one long task. The agent builds a small ledger reconciliation library from a documentation set of about 70,000 tokens. There are 134 hidden checks (131 for accuracy, 3 for memory). Along the way: repo rules checked at the end, an injected HTTP 429 on model calls 3 and 5, an instruction hidden in a data file, a script that would delete the data, one that posts outward, and a report that must be published exactly once.

Round 3: a multi-repo incident. A workspace of several git repositories (a rate-feed parser, an FX library, an orders service and a report CLI) and a ticket saying the monthly report disagrees with finance. The agent has to find every root cause, fix each in the repository that owns it, follow standing rules (a regression test, a changelog line, a version bump and so on), rebuild the report and publish it once. I ran it twice, as version A and then a much harder version B. The same injection, delete-script, post-script, secret and 429 traps were in it, and the hidden checks ran in a far-away timezone so any code that reads a timestamp in local time would fail.

Round 4: the same incident plus a change request (version C). A library API migration across five repositories, one of them owned by another team and read-only, still calling the old API. A Swiss-franc cash-rounding rule, half-even rounding for one currency, and a rule that all time logic lives in one library. The context window was capped at 64k, hoping to force compaction.

Invalid runs. A run where the task itself was broken, or the provider stalled without the agent touching a file, was removed and rerun. Ordinary failures stayed in.

Overall results

Round 1: 14 hard coding tasks

Tokens count fresh input plus output for passing runs.

Per task (passes out of runs, infrastructure errors excluded):

Thirteen of the fourteen tasks were solved by everyone in every run. Only the scheduler task separated them.

Round 2: the stress task

Every run that finished its job passed all 134 hidden checks, including the 3 memory checks. Nobody ran the delete script, posted outward, or followed the injected instruction.

After these two rounds I left Claude Code and the Copilot agent out of the multi-repo rounds. Claude Code cost $21.70 to $24.40 per long-task run, used about nine times Pi's tokens on the short tasks and failed one of its two long runs. The Copilot agent's one run cost about $8.75 for the same model.

Rounds 3 and 4: multi-repo debugging

Version A was too easy: its helper script gave the answer away. Version B removed the script and added a fourth repository and seven more causes. Version C added a library migration across five repositories and a 64k context cap. Three arms, three runs each, in parallel.

All 27 runs recovered both injected 429s (2 of 2 each) and sent High effort on the wire. Nobody followed the instruction hidden in a log, ran the data-wiping or posting scripts, touched the secret or edited the read-only repository.

Why the same model produced different results

System prompt and tools. Pi's prompt plus its four tool definitions come to under 1,000 tokens, and its author argues that modern models don't need more. Claude Code carries a much larger static prefix, about 33k tokens in my runs. I did not isolate prompt size as the cause of anything.

Context management and caching. This was the biggest factor on the long task. Pi and Oh My Pi kept reusing the growing conversation from cache. Claude Code's cached portion stayed at a constant 33k tokens, the static prefix, on 40 of 44 calls in one run, so the growing conversation was billed as fresh input. About 73% of that run's cost was uncached input. At a 90% share the same run would have cost roughly half.

Number of calls and output. In the multi-repo rounds, cache share did not explain the order: OpenCode reused the most (93 to 96%) and still cost the most, because it made roughly twice as many model calls (1.7 to 2.3 times Pi's) and wrote 1.4 to 2 times as much output as Pi for the same working result. Each call re-sends the whole conversation. Fewer calls and less output is what kept Pi cheap.

Output volume and time. The model spent about 7 to 8 ms per output token, and model time was 91 to 98% of every run. Claude Code wrote four to six times more output than Pi for a code change of similar size, and made 44 model calls in one run against 13 for Pi.

Retries. A small proxy returned a 429 at fixed call numbers. Pi and Oh My Pi recovered every time in every round, and so did OpenCode in the multi-repo rounds, with no duplicated side effects. Claude Code and GitHub Copilot in VS Code could not be tested.

Why Pi is the best harness here

I ranked on accuracy first, then cost, then speed. Accuracy was a near tie, so cost and speed decided it, and Pi won both. The reasons, with the numbers behind them:

1. Lowest cost in every round. 5.5k tokens per passing run against 9.0k and 50.5k in round 1; $2.40 to $3.20 on the long task; $0.95, $1.90 and $3.35 per run across the three multi-repo versions. Oh My Pi cost about 1.2 to 2 times as much, OpenCode about 1.6 to 2.5 times, Claude Code about 7 to 10 times (on the long task).

2. Equally accurate. 36 of 36 and 3 of 3 on the first two rounds and 9 of 9 on the multi-repo rounds. The only miss it ever had was my own ambiguous spec. The two harnesses closest in cost each slipped once on a written rule.

3. Fewer calls and less output. In the multi-repo rounds Pi used 8 to 18 model calls where OpenCode used 18 to 31, and wrote about half to two thirds of the output. Since every call re-sends the conversation and output is the most expensive token type, that is where the savings come from. I did not isolate why it takes fewer steps. Its tiny prompt and four tools are the obvious candidates, but that is a hypothesis, not a measurement.

4. Fast. The fastest in rounds 1 to 3 and within noise of Oh My Pi in round 4 (182 s against 166 s, with the arms running in parallel).

5. Efficient caching even with a small prompt. It reused 82 to 94% of its prompt, so the cheapness is not a side effect of skipping the cache.

6. Resilient. It recovered from every injected 429 and followed every written rule in the multi-repo rounds.

7. Inspectable. It is a small MIT-licensed harness (about 116k GitHub stars) with branching JSONL sessions and extensions instead of built-in plan mode, sub-agents or MCP, so you can see what it does.

What it costs you: Pi has no permission system, so it runs with your user's permissions. I ran it in an OS sandbox that only allows writes inside the project folder. If you need approval prompts built into the tool, choose another harness.

The five harnesses at a glance

Best harness profiles in descending order

Ranking criteria, in order: accuracy across all rounds (counting task-ambiguity misses as the task's fault), then cost, then speed. Three runs per harness at most, so treat close calls as ties.

1. Pi

Results: 36 of 36 on 14 tasks; 3 of 3 on the long task; 9 of 9 on the multi-repo task (3 versions)
Cost: $2.40 to $3.20 on the long task; $0.95, $1.90 and $3.35 on the multi-repo versions

The cheapest and one of the two fastest, recovered every injected 429, and made no mistakes against the written rules. See the reasons above.

Best for: people who want the lowest cost and the most visibility into what the agent does.
Main tradeoff: no permission system. Run it in a sandbox.

2. Oh My Pi

Results: 36 of 36 on 14 tasks; 2 of 2 on the long task; 7 of 9 on the multi-repo task
Cost: $4.90 to $5.40 on the long task; $1.47, $2.85 and $3.84 on the multi-repo versions

Same lineage and command line as Pi, with more built in. Its code was always right, but twice it skipped a rule or a requirement that was written down (a changelog line, a deprecation warning). That is why it ranks second, not first. In the independent Composio study it had the highest pass rate, so this may be tied to the tasks.

Best for: a cheap harness with more built in than Pi.
Main tradeoff: about 1.2 to 2 times Pi's cost, and it needs a check against your standing rules.

3. OpenCode

Results: 35 of 37 on 14 tasks; passed the long task; 9 of 9 on the multi-repo task
Cost: $2.33, $4.05 and $5.33 on the multi-repo versions (measured through the proxy)

An MIT-licensed harness (about 216k stars) with the broadest feature set: build and plan agents, subagents, a fine-grained permission system, any provider, LSP and MCP support, and terminal, desktop and IDE versions. It never failed a multi-repo run. It missed one pattern (equal-length ties on the scheduler) twice in round 1. It cost about twice Pi, with about twice the model calls and 93 to 96% cache reuse, and it makes one extra low-effort call per run for its hidden title agent.

Its biggest advantage is cost at the low end. OpenCode can use free models such as GLM 5.3 Flash, so you can run a fully open-source tool on a free model and pay nothing. If you care about open-source tooling and a zero-dollar starting point, no other harness here matches it. I did not test the free models: every run in this post used Claude Sonnet 5.5, and which models are free changes over time, so check the current list.

Best for: people who want built-in permission controls, a large ecosystem, or a fully open-source setup on free models.
Main tradeoff: with the same paid model it cost roughly twice Pi and was the slowest of the three.

4. Claude Code

Results: 35 of 36 on 14 tasks; 1 of 2 on the long task; not run on the multi-repo task
Cost: $21.70 to $29.20 on the long task, 59 to 63% cache share

A mature harness with permissions and compaction, and it passed the long task's 134 checks in every run that finished. It cost seven to ten times more than Pi, mostly from low cache reuse, and its permission settings stopped one run before the final step. That cost may be partly a route effect.

Best for: teams that value its permission model and ecosystem and will check their own cache behaviour.
Main tradeoff: the highest cost in this setup.

Also tested: GitHub Copilot in VS Code

Results: 4 of 4 on four of the hardest short tasks (run by hand, one run each); 1 of 1 on the long task; not run on the multi-repo task
Long task: $8.75 on the Copilot counter, 268 s of agent time, 36 steps, a peak context of 112K in a 328K window

GitHub's own agent, running inside VS Code on your Copilot plan, with its own tools and approval prompts. It solved everything I gave it. Its one measured long run cost about 2 to 3 times Pi, and well under Claude Code. But the evidence is thin: one run per task, done by hand, no cache share captured, no injected-error or compaction test, and a bigger context window than the others. So I list it here instead of ranking it.

Best for: people already working in VS Code with a Copilot plan who want no extra tool or setup.

Dropped after a first pass: Codex

Codex on GPT 6.1 Sol was tested briefly at the start. It used a different model from the other harnesses, and its accuracy was not up to the bar, so I did not carry it into the later rounds and it has no numbers here.

Which harness should you choose?

Pick Pi if:

• you want the lowest cost per run and the fewest model calls;

• you mostly work on ordinary source code and want a small system you can read;

• you can run it in a sandbox, because it has no permission system of its own.

Pick Oh My Pi if:

• you want more built in than Pi (per Composio: LSP, a debugger, readers for many file types);

• you accept about 1.2 to 2 times Pi's cost;

• you will check its output against your written project rules.

Pick OpenCode if:

• you want to run on free models such as GLM 5.3 Flash, or you want a fully open-source stack;

• you want built-in approval controls, any model provider and a large ecosystem;

• you accept about twice Pi's cost when you use the same paid model, and the slowest runs of the three.

Pick Claude Code if:

• you value its permission model, hooks and subagents;

• you will measure and tune its cache behaviour, which decided its bill here.

Pick GitHub Copilot's agent if:

• you already work in VS Code on a Copilot plan and want no extra setup, and you can live with provisional numbers.

Whatever you choose, run your own two or three tasks through your finalists. Cache behaviour, permissions, call count and task length change the result.

Limitations

• One model. Everything used Claude Sonnet 5.5. A harness can rank differently with its native model or another one, and I did not test free models.

• Small synthetic projects in Python. They catch differences in cost, caching and rule-following. They say little about large repositories or other languages, and every harness solved even the hardest version, so I cannot claim one is smarter.

• Dollar figures are estimates. The real counter ran about 25% higher because I couldn't see cache writes.

• Compaction was not tested in the last round. No run compacted, and my 64k cap probably did not reach OpenCode.

• I wrote the tasks and found several defects in them while running the benchmark (a reference solution that disagreed with its own spec, a spec silent on letter case, a hidden check on an unspecified export order, a first multi-repo task that gave the answer away, and a permission setting that stopped one run). I fixed them and reran, but there may be more.

Conclusion

The harness changed what the same model cost far more than it changed what it could do. Almost everything solved almost every task, including a five-repository incident with a library migration and traps. The bills differed by up to ten times.

Pi came out on top because it was as accurate as anything else and cheaper, with fewer calls and less output, a high cache share and no failed retries. Choose another harness if you need built-in permission prompts, or want free models and a fully open-source stack (OpenCode), and run Pi in a sandbox if you choose it. Evaluate the model and the harness together, run your own two or three tasks through your finalists, measure your cache hit rate and your number of calls, and read the failed runs before you read the scoreboard.