Nine harnesses, one model, and a 17x spread in the bill

Same model. Same tasks. Nine different harnesses driving it. The cost per task runs from $1.05 to $18.34.

That is the whole finding, and the rest of this page is what is wrong with it, because a 17x number with no caveats attached is not worth much to you.

What was actually run

• 9 harnesses across 12 configurations
• 360 trials total, so 30 tasks per configuration (that is arithmetic off 360 over 12, the page does not state it in words)
• One model: Kimi K3
• One runtime: Runta, every run from the same fresh checkpoint

Source: frontierharness.org, read 9 September 2026, along with its Show HN thread the same day.

The full table

Most writeups quote the two ends. Here is all nine, because the harness you already use is probably in the middle.

Harness	Pass rate	Cost per task
Codex	66.7%	$3.47
Claude Code	63.3%	$18.34
DSH Creator	63.3%	$3.28
Pi	60.0%	$2.43
DSH PTC	60.0%	$4.58
DSH Standard	60.0%	$3.46
Oh My Pi	56.7%	$4.75
Kimi Code	56.7%	$3.65
Exo Harness	53.3%	$1.05

The two spreads

17.5x is $18.34 against $1.05. It is the headline, and it needs the most caveats, because the top of it is an outlier for a specific reason (below).

4.5x is Oh My Pi at $4.75 against Exo at $1.05. This is the spread that survives throwing the outlier out entirely, and it is the number worth acting on. Even among ordinary harnesses, the one you pick moves your bill several times over.

What is wrong with it

Four things, in the order they matter.

1. Runta ran the benchmark on the runtime it sells. The page offers $100 in credits to test your own harness there. Nobody independent has reproduced any of this.
2. The Claude Code figure is that harness driving Kimi K3 through a proxy. It is not what a Claude Code subscriber pays. If you read $18.34 as "Claude Code is expensive", you have read the wrong thing.
3. Median, not mean. joshheitzman, in the thread: "your provider will bill you for all your tasks and one can get back to that total from the mean by multiplying by the number of tasks. This isn't possible with the median." Your invoice comes from the mean. A median hides the runs that blew up.
4. One model means this ranks harnesses on Kimi K3, and nothing else. vidarh's point: harnesses built primarily around Anthropic's models never had to deal with Kimi's particular quirks. Swap the model and the order can change.

One more that gets argued both ways. nsingh2 said nobody runs Pi without extensions, so its bare configuration is unrepresentative. infecto replied "I use Pi barebones", and Edward40, who built the benchmark, said he also prefers vanilla Pi over Oh My Pi. Contested, not settled.

The bit almost everyone skips

Exo Harness is the cheap end at $1.05. It is also the worst harness in the table at 53.3%.

Codex passes 66.7% of the time at $3.47. So the 17x spread is partly a price difference and partly a quality difference, and dividing one by the other treats them as the same thing. A cost ratio between a 63.3% harness and a 53.3% harness is not a like for like saving. If you retry every failure, the cheap harness costs more than its sticker.

The harness is a cost variable you probably never tuned, and it moves your bill by more than most model swaps do. That is the thing worth taking away, and it survives every objection above.

Your turn

If you have run the same model through more than one harness, reply and tell me what you saw. Which two, and whether the cost gap looked anything like this. I read every one of these, and it is the only way this gets checked by someone who is not selling a runtime.

---

Where the longer version lives

I keep a room called The Rabbit Hole for the parts that don't fit on a page like this: the full run of numbers, what broke the first time, and the ones that looked good and then didn't work at all.

Two catches, up front, because that's the rule here. It's private, so you have to be inside to read anything. And it's new, so you'd be early rather than joining something busy.

Free to join: skool.com/the-rabbit-hole-3206. The course library is free; optional Premium adds weekly office hours.