Benchmark/Published November 16, 2026
An agentic proof-generation benchmark: measuring how efficiently models generate proofs that pass Verus verification, given a program or protocol model and its formal specification
01
The Verus Proof Index incorporates proof-generation correctness and efficiency, computed across 100 problems from the VerusBench, VeruSAGE-Bench, and tlaverus datasets
We tested 25 models as autonomous agents on 100 Verus proof problems: verified systems code, distributed protocols transpiled from TLA+, and smaller programs. Every proof was graded by the Verus verifier and independently checked for integrity violations. Leading models eventually solve most proof-generation problems, so the index also scores efficiency: the tokens and turns needed to generate a passing proof. tlaverus problems come from our internal dataset and have never been public as Verus code; VerusBench and VeruSAGE-Bench are public, so their presence in a model's training data cannot be ruled out. Each model got one run per problem, revising its proof against the verifier's feedback as often as the preset limits allowed, so differences of a point or two can be noise.
| 1 | GPT-6 Astra | OpenCode | 59.1 | 55.8 | +3.4 | 94 | 41 | 1 | $224 | 5m 13s |
|---|---|---|---|---|---|---|---|---|---|---|
| 2 | Claude Fable 5.1 | Claude Code | 58.7 | 58.2 | +0.5 | 97 | 40 | 0 | $483 | 8m 14s |
| 3 | Claude Opus 5.5 | Claude Code | 58.2 | 58.2 | 0.0 | 97 | 64 | 0 | $232 | 6m 44s |
| 4 | Gemini 4 Argon | OpenCode | 56.2 | 51.6 | +4.6 | 88 | 51 | 2 | $54 | 3m 52s |
| 5 | GPT-6.1 Sol | OpenCode | 55.6 | 52.8 | +2.8 | 91 | 40 | 3 | $48 | 5m 44s |
| 6 | Claude Sonnet 5.5 | Claude Code | 53.1 | 55.2 | −2.1 | 94 | 41 | 2 | $245 | 11m 18s |
| 7 | Gemini 3.8 Flash | OpenCode | 52.7 | 54.0 | −1.3 | 90 | 19 | 0 | $96 | 7m 54s |
| 8 | Grok 4.7 | OpenCode | 45.5 | 47.4 | −1.9 | 82 | 25 | 3 | $543 | 13m 08s |
| 9 | Muse Wonder | OpenCode | 44.7 | 46.2 | −1.5 | 78 | 16 | 1 | $334 | 6m 27s |
| 10 | Qwen3.8 Flash | OpenCode | 44.6 | 47.4 | −2.8 | 80 | 7 | 1 | $18 | 15m 09s |
| 11 | DeepSeek V4.1 Flash | OpenCode | 44.5 | 46.7 | −2.2 | 81 | 8 | 3 | $21 | 4m 19s |
| 12 | Hy4 | OpenCode | 43.2 | 46.2 | −3.0 | 80 | 11 | 3 | $34 | 12m 38s |
| 13 | Kimi K3 | OpenCode | 42.8 | 43.2 | −0.4 | 72 | 24 | 0 | $132 | 16m 07s |
| 14 | GPT-6 Luna | OpenCode | 42.4 | 43.9 | −1.5 | 75 | 7 | 1 | $11 | 8m 13s |
| 15 | GLM 5.3 | OpenCode | 39.5 | 42.0 | −2.5 | 70 | 15 | 0 | $127 | 20m 33s |
| 16 | MiMo-V2.6-Pro | OpenCode | 38.5 | 40.8 | −2.3 | 68 | 15 | 0 | $21 | 17m 02s |
| 17 | GLM 5.3 Flash | OpenCode | 37.0 | 38.4 | −1.4 | 65 | 10 | 1 | $17 | 18m 32s |
| 18 | Mistral Large 4 | OpenCode | 33.2 | 36.0 | −2.8 | 60 | 15 | 0 | $117 | 24m 53s |
| 19 | Qwen3.8 Max | OpenCode | 32.4 | 34.2 | −1.8 | 57 | 15 | 0 | $138 | 25m 01s |
| 20 | Mercury 3 | OpenCode | 29.4 | 32.8 | −3.4 | 57 | 8 | 2 | $16 | 3m 39s |
| 21 | Muse Spark 1.3 | OpenCode | 26.9 | 31.4 | −4.5 | 55 | 30 | 0 | $249 | 22m 14s |
| 22 | Beam 1.0 | OpenCode | 24.9 | 26.0 | −1.1 | 48 | 8 | 4 | $23 | 5m 00s |
| 23 | Nemotron 3.5 Super VL | OpenCode | 19.4 | 24.6 | −5.2 | 46 | 3 | 5 | $44 | 11m 36s |
| 24 | Laguna S 2.1 | OpenCode | 18.1 | 22.2 | −4.1 | 41 | 6 | 4 | $17 | 23m 00s |
| 25 | Inkling Small | OpenCode | 7.6 | 10.8 | −3.2 | 26 | 4 | 8 | $79 | 11m 44s |
Select a column heading to sort. Index = correctness + efficiency. First proof: number of problems passed with the first proof submitted. Integrity violations: runs scored as an integrity violation. Cost: total spend recorded across all 100 problems, one run each, at the providers' prices (estimated from token counts where no cost was recorded). Time: median time the model spent generating per solved problem; unsolved problems are excluded. All results were run by Reasonable; none are self-reported.
| Level | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
|---|---|---|---|---|---|---|---|---|---|
| Tokens | 7k | 10k | 15k | 20k | 30k | 40k | 60k | 80k | 120k |
| Turns | 15 | 20 | 20 | 30 | 30 | 40 | 40 | 50 | 50 |
| Problems | 6 | 6 | 13 | 7 | 11 | 12 | 20 | 16 | 9 |
02
The index splits into correctness, scored from proof outcomes alone, and efficiency, gained or lost for using fewer or more tokens and agent turns than each problem's expected budget ().
Median length of a model's proofs (comments and formatting excluded), as a multiple of each problem's expected shortest proof length.
03
What running the Verus Proof Index cost at the providers' prices, with the Pareto frontier ().
Median inference time per solve, on a selected provider per model, with the Pareto frontier ().
04
Scroll sideways for all models