Project 01
SilverquiLLM-bench
We hand coding agents brand-new Magic: The Gathering cards and a real game engine. They write the code. Hidden tests decide whether the cards play by the rules.
How a run works
Card text in. Working code out.Hidden tests decide.
Agent setups are multiplying faster than anyone can compare them. Model, harness, prompts: every part is a dial. SilverquiLLM-bench tells you what turning one actually does.
01 A freshly released card
02 An agent writes the code
03 Hidden tests grade it
Illustrative example. The latest round hands agents five freshly released cards and grades the code against 90 hidden tests.
Latest results
Which agent holds up?
Six models on the same five cards and the same hidden tests, 10 to 20 runs each.
Latest results · hob-medium · Oct 2026
Max-effort score, with run-to-run range
| Configuration | Pass rate range | Mean | API $/run |
|---|---|---|---|
| GPT-6.1 Solmax effort | range 94.4% to 98.9% | 96.9% | $1.47 |
| GPT-6 Astramax effort | range 91.1% to 98.9% | 95.1% | $6.96 |
| Sonnet 5.5max effort | range 90.0% to 98.9% | 94.4% | $19.39 |
| Opus 5.5max effort | range 90.0% to 97.8% | 93.6% | $21.11 |
| GPT-6 Solmax effort | range 86.7% to 95.6% | 90.7% | $3.24 |
| GPT-6 Lunamax effort | range 71.1% to 92.2% | 82.6% | $0.26 |
Five freshly released Hobbit cards, 90 hidden tests, 10 to 20 runs per model. GPT-6.1 Sol leads, at about a fourteenth of Opus 5.5’s cost per run. Claude runs that flagged a test contradicting the rules get credit for it.
GPT-6.1 Sol leads
It averages 96.9 out of 100 at max effort, clearly ahead of both Claude models. Sonnet 5.5 now matches Opus 5.5, for less.
Price changes the picture
At API prices a max-effort GPT-6.1 Sol run costs $1.47 and an Opus 5.5 run $21.11, about 14 times as much. On a $200 subscription the gap narrows to about 2.5 times.
Nothing else broke
Every Opus, Sonnet, GPT-6.1 Sol, Astra and GPT-6 Sol run kept all 394 existing-card tests and all 1,437 engine tests passing.
The testbed
Why Magic: The Gathering
Real-world shape
Implementing a card is messy, stateful, long-horizon work: pages of rules, edge cases that interact, board state that shifts every turn. It’s the shape of real software, compressed into one task.
No ceiling
Magic is provably Turing-complete — one of the most computationally complex games there is. There’s no limit to how hard a card can get, so the benchmark won’t run out of headroom as agents improve.
Can’t be memorized
Magic ships new sets constantly. Grading on a freshly released set means the answer can’t come from training data — so the score measures the work itself.
The method
Fair by design
- 01
A fresh container every run
Each run gets its own clean Docker container: the engine, the reference cards, the rulebook and the task. No run can see another’s work.
- 02
Tests the agent never sees
The hidden suite is never mounted into the container. Grading happens afterwards, offline, against the code the agent left behind.
- 03
Three scores, plus the bill
Do the new cards follow the rules? Do the existing cards still work? Does the engine? And what did the run cost?
- 04
The tests get audited too
Benchmark tests are code, and code has bugs. Review agents audit the suite, and every run is re-graded when it changes.
What we’ve learned
Three rounds so far
June 2026
First flight
Ten freshly released cards, 83 hidden tests. A planner-and-implementer setup tied a single agent on accuracy, at the same cost. More machinery isn’t free performance.
Read: Initial FindingsJuly 2026
Fable vs. Opus
Given a detailed plan, Fable 5 tied Opus 4.8 at 1.6 times the price. Sonnet 4.6 sat a full tier below.
Read: Is Claude Fable Really Better Than OpusOctober 2026
hob-medium
The first round on the V2 engine: five Hobbit cards, 90 hidden tests, 335 runs, each re-graded after an audit of the tests themselves. GPT takes the lead, and Sonnet catches Opus.
Take it for a spin.
The benchmark is open. Clone it, run it, and see how your agent setup holds up.