We build softwarethat doesn’t bite.
Instrumentation and seatbelts for coding agents.
Instrumentation
SilverquiLLM-bench
Can a coding agent turn freshly released Magic cards into working engine code? Hidden tests grade every run, and every run’s cost is counted.
Seatbelts
KarnComing soon
Containers for your coding agents, built from a short recipe. Settings pass a closed allowlist, and every image carries a receipt.
What we’re building · SilverquiLLM-bench
Card text in. Working code out.Hidden tests decide.
An open benchmark that tests how well coding agents turn new Magic: The Gathering sets into code. The latest round: five fresh Hobbit cards, 90 hidden tests, and six models.
Latest results · hob-medium · Oct 2026
Max-effort score, with run-to-run range
| Configuration | Pass rate range | Mean | API $/run |
|---|---|---|---|
| GPT-6.1 Solmax effort | range 94.4% to 98.9% | 96.9% | $1.47 |
| GPT-6 Astramax effort | range 91.1% to 98.9% | 95.1% | $6.96 |
| Sonnet 5.5max effort | range 90.0% to 98.9% | 94.4% | $19.39 |
| Opus 5.5max effort | range 90.0% to 97.8% | 93.6% | $21.11 |
| GPT-6 Solmax effort | range 86.7% to 95.6% | 90.7% | $3.24 |
| GPT-6 Lunamax effort | range 71.1% to 92.2% | 82.6% | $0.26 |
Five freshly released Hobbit cards, 90 hidden tests, 10 to 20 runs per model. GPT-6.1 Sol leads, at about a fourteenth of Opus 5.5’s cost per run. Claude runs that flagged a test contradicting the rules get credit for it.
Why it matters
MTG is a brutal stress test for coding agents — long rules, weird edge cases, and state that changes every turn. It’s a thorough yet affordable test for long-running coding agents.
How it’s built
Python, Docker, and pytest, so anyone can run it reproducibly.
More on the way
Notes from the workshop.
Snowfox Builds is just getting started. SilverquiLLM-bench is the first of several tools and experiments — expect more projects and write-ups in the future.
- How Good Are LLM Agents At Implementation?
- Is Claude Fable Really Better Than Opus
- SilverquiLLM-bench: Initial Findings