How Good Are LLM Agents At Implementation?

Scores at Each Effort LevelHigher effort correlates with more correctness. Top four models form a close cohort, separating from GPT-6 Sol and GPT-6 Luna.

Show the numbers

Score is the share of the 90 hidden tests passed, in %, with each zone-change miss credited back where the run’s final report flagged it. ±95% CI is a t-based 95% confidence interval on the mean. Cost is the mean per run at API prices.
Model	Effort	Runs	Mean	±95% CI	Range	Before credit	Runs credited	Cost/run
GPT-6.1 Sol	low	10	89.8	1.9	85.6–94.4	89.8	0 of 10	$0.52
GPT-6.1 Sol	medium	10	92.2	1.5	88.9–96.7	92.2	0 of 10	$0.67
GPT-6.1 Sol	high	20	93.5	1.0	88.9–96.7	93.5	0 of 20	$0.86
GPT-6.1 Sol	xhigh	20	94.4	1.3	86.7–98.9	94.4	0 of 20	$1.04
GPT-6.1 Sol	max	20	96.9	0.8	94.4–98.9	96.9	0 of 20	$1.47
GPT-6 Astra	low	10	89.0	1.7	86.7–94.4	89.0	0 of 10	$3.27
GPT-6 Astra	medium	10	89.6	1.4	86.7–92.2	89.6	0 of 10	$3.44
GPT-6 Astra	high	10	93.1	1.6	87.8–95.6	93.1	0 of 10	$4.50
GPT-6 Astra	xhigh	10	94.7	0.8	92.2–95.6	94.7	0 of 10	$5.25
GPT-6 Astra	max	10	95.1	1.6	91.1–98.9	95.1	0 of 10	$6.96
Sonnet 5.5	low	10	86.9	1.4	82.2–90.0	85.6	5 of 10	$2.53
Sonnet 5.5	medium	10	89.9	1.9	86.7–94.4	88.2	8 of 10	$3.34
Sonnet 5.5	high	10	90.8	1.2	87.8–92.2	89.3	5 of 10	$5.40
Sonnet 5.5	xhigh	11	92.5	2.1	87.8–97.8	90.5	8 of 11	$10.67
Sonnet 5.5	max	13	94.4	1.4	90.0–98.9	93.3	8 of 13	$19.39
Opus 5.5	low	10	87.1	2.5	83.3–93.3	86.0	3 of 10	$3.71
Opus 5.5	medium	10	88.8	1.7	86.7–93.3	87.9	3 of 10	$6.06
Opus 5.5	high	10	91.6	2.1	84.4–94.4	89.8	6 of 10	$7.18
Opus 5.5	xhigh	10	92.4	1.6	90.0–95.6	91.8	4 of 10	$13.46
Opus 5.5	max	10	93.6	1.7	90.0–97.8	92.1	6 of 10	$21.11
GPT-6 Sol	low	10	71.9	10.4	32.2–80.0	71.9	0 of 10	$0.74
GPT-6 Sol	medium	10	82.6	1.7	77.8–85.6	82.6	0 of 10	$1.40
GPT-6 Sol	high	10	84.2	3.6	71.1–91.1	84.2	0 of 10	$2.18
GPT-6 Sol	xhigh	10	88.4	2.1	84.4–92.2	88.4	0 of 10	$2.44
GPT-6 Sol	max	10	90.7	2.0	86.7–95.6	90.7	0 of 10	$3.24
GPT-6 Luna	low	10	56.7	10.4	40.0–78.9	56.7	0 of 10	$0.09
GPT-6 Luna	medium	10	42.8	16.3	15.6–74.4	42.8	0 of 10	$0.03
GPT-6 Luna	high	10	75.6	7.6	48.9–86.7	75.6	0 of 10	$0.11
GPT-6 Luna	xhigh	10	82.4	9.1	50.0–91.1	82.4	0 of 10	$0.23
GPT-6 Luna	max	10	82.6	5.5	71.1–92.2	82.6	0 of 10	$0.26

Key Takeaways

Opus, Astra, Sonnet, and Sol all made very few obvious mistakes. Points lost by these models were primarily due to making false assumptions about existing code.
Opus did not make any obvious mistakes at any effort level, and Astra only made one.
Sol 6.1 is the most meticulous model, while being hyper-efficient. Among the top four, Opus made the most assumptions.
Claude models sometimes deliberately chose not to fix an edge case, if they felt that the fix would worsen the code-base as a whole.

Methodology

The new benchmark, hob-medium, is based on the V1 benchmark design, but with key improvements on validation strategy. The previous benchmark ran tests on the agent’s implementation, but this was fragile because the agent could implement things differently from what we might expect. The new benchmark validates by simulating the game, and scores whether the game progresses according to expectations.

The benchmark set includes 5 cards from the new Hobbit set: The Eagles Are Coming!, Elrond, Moon-Reader, Gollum, Riddle Master, The Notary Hobbits, and Tom, Bert, and William. These cards were hand-picked because while they aren’t the most complicated cards in the set, they each have tricky edge cases to consider.

Default harnesses were used (Claude Code CLI for Claude models, Codex CLI for GPT models), with no custom skills, and subagents disabled. (Funnily, during a trial run prior to disabling subagents, Sonnet 5.5 on max effort launched multiple Opus subagents, spent $68, and got a perfect score. This run was excluded from the final results.)

Each candidate got 10-20 runs, and scores were aggregated by average.

How a run worksOne run, followed on one cardA real Opus 5.5 run at max effort, on one of the five cards.

01 What the agent gets
Gollum, Riddle Master{1}{B}
Legendary Creature — Halfling Horror
As Gollum enters, choose odd or even. (Zero is even.)
Whenever an opponent casts a spell with mana value of the chosen quality, choose one that hasn’t been chosen —
- Put a +1/+1 counter on Gollum.
- Each opponent loses 2 life and you gain 2 life.
- Draw a card.
3/1
- workspace/
- ├─ engine/a working game engine
- ├─ cards/fdn/276 existing cards to learn from
- ├─ cards/hob/five new cards to write
- │ └─ hob_70/Gollum: the card, notes, an empty class
- └─ RULEBOOK.txtthe full rules of Magic
Five new cards to add to a working engine.
02 What it’s told
The prompt
Implement the selected cards for hob-medium in your Workspace. […] Use card_spec.json as the card definition and existing FDN cards as examples. You may change the engine to support the required mechanics. Keep existing card and engine behavior working.
Then it works alone, with no human help.

03 What it writes

cards/hob/hob_70/card_impl.py


              16
              QUALITIES = ("odd", "even")
            
              40
              class GollumRiddleMaster(Creature):
            
              72
                  def enters_battlefield_with(self, game, event):
            
              73
                      """The odd/even choice is made as Gollum enters …
            
              81
                      self.chosen_quality = choose_mode(
            
              82
                          game, chooser, list(QUALITIES), "Choose odd or even"
            
              83
                      )

43 minutes and $20.85 later, all of its own tests pass.

04 How it’s graded
Gollum’s hidden tests, 6 of 19
- ✓Triggers only on spells of the chosen odd or even
- ✓Each mode can be chosen only once
- ✓Coming back resets its choices
- ✕Counts X in a spell’s mana value
- ✕Comes back from exile without its old counters
- ✓Still draws after Gollum has left
Gollum
15 / 19
All five cards
83 / 90
Existing cards and engine
All pass
Score92.2
Hidden tests it never saw decide the score. With the credit explained below, 94.4.

Model Efficiency

Correctness vs CostModels in the top left are more efficient; Sol 6.1 proves to be the most efficient implementer.

Opus 5.5Sonnet 5.5GPT-6.1 SolGPT-6 Astra

Up: score. Right: share of a $200 plan’s weekly limit per run (log scale).

API cost as a share of the API-priced usage each $200 plan allows a week: Opus 5.5 $2,697, Sonnet 5.5 $2,881, GPT-6.1 Sol $479, GPT-6 Astra $667. That is the plan’s monthly value (Opus 5.5 58.6×, Sonnet 5.5 62.6×, GPT-6.1 Sol 10.4×, GPT-6 Astra 14.5× the fee) over 4.345 weeks. Measured by SemiAnalysis, October 2026, on Claude Max 20x and ChatGPT Pro 200.