In 27,280 coding agent runs, 64% of inference spend went to runs that failedImage: runta.com
Cooling

In 27,280 coding agent runs, 64% of inference spend went to runs that failed

Heat 46 · #36 in AI · Cooling2 reports · 2 sources · First reported Nov 2026 · Updated Nov 20

The story

In an evaluation of 27,280 coding-agent runs, 64.5% of list-price inference spending went to trials that did not pass. The FrontierHarness Eval v1.1, run by Runta with Baseten, consumed 83 billion tokens across 22 harnesses, five open-weight models and 62 tasks, with four attempts at each task. Failures accounted for 52.7% of the wasted spend and timeouts for another 11.8%. The cheapest harness depended on the model: Codex CLI ranked first with GLM-5.3 but only 16th with Kimi-K3. The study also found that failed runs ended with a confident answer 68-82% of the time.
Latest development
same post, same pts/cmt as round 163

How it unfolded

  1. same post, same pts/cmt as round 163
  2. same post, same pts/cmt as round 162
  3. stays active, exactly at the 66.0 line)
  4. snapshot fetch for this item timed out, re-fetched directly from the HN API
  5. studio.aisa.one v1.1 item is an AI-written summary, per AiNews247-mirror precedent not counted as a platform), cross stays 30
  6. search: no independent reporting of the v1.1 finding (thebeat.tech covers v1.0
  7. failed runs end with a confident answer 68-82% of the time
  8. cheapest harness per task changes with the model (Codex CLI 1st with GLM-5.3, 16th with Kimi-K3)
  9. 64.5% of list-price inference cost went to trials that did not pass (52.7% failures + 11.8% timeouts)
  10. Runta blog introducing FrontierHarness Eval v1.1 (with Baseten): 83B tokens across 27,280 agent runs, 22 harnesses x 5 open-weight models (GLM-5.3, GLM Flash, DeepSeek V4.1 Flash, Muse Spark Open, Kimi-K3) x 62 tasks x 4 attempts

The angle

64% of the money spent on AI coding agents goes to runs that fail - and the cheapest setup for one model is nearly the priciest for anotherSuggested format: Short video · for Developers using AI coding agents

Heat

Now 46 · Peak 46 · 64 recorded checks since Nov 20
Full ranking →