Benchmark/Published November 16, 2026

Verus Proof Index

An agentic proof-generation benchmark: measuring how efficiently models generate proofs that pass Verus verification, given a program or protocol model and its formal specification

Blog PostRankingMethodology

01

The Verus Proof Index incorporates proof-generation correctness and efficiency, computed across 100 problems from the VerusBench, VeruSAGE-Bench, and tlaverus datasets

We tested 25 models as autonomous agents on 100 Verus proof problems: verified systems code, distributed protocols transpiled from TLA+, and smaller programs. Every proof was graded by the Verus verifier and independently checked for integrity violations. Leading models eventually solve most proof-generation problems, so the index also scores efficiency: the tokens and turns needed to generate a passing proof. tlaverus problems come from our internal dataset and have never been public as Verus code; VerusBench and VeruSAGE-Bench are public, so their presence in a model's training data cannot be ruled out. Each model got one run per problem, revising its proof against the verifier's feedback as often as the preset limits allowed, so differences of a point or two can be noise.

Results as a table
1GPT-6 AstraOpenCode59.155.8+3.494411$2245m 13s
2Claude Fable 5.1Claude Code58.758.2+0.597400$4838m 14s
3Claude Opus 5.5Claude Code58.258.20.097640$2326m 44s
4Gemini 4 ArgonOpenCode56.251.6+4.688512$543m 52s
5GPT-6.1 SolOpenCode55.652.8+2.891403$485m 44s
6Claude Sonnet 5.5Claude Code53.155.2−2.194412$24511m 18s
7Gemini 3.8 FlashOpenCode52.754.0−1.390190$967m 54s
8Grok 4.7OpenCode45.547.4−1.982253$54313m 08s
9Muse WonderOpenCode44.746.2−1.578161$3346m 27s
10Qwen3.8 FlashOpenCode44.647.4−2.88071$1815m 09s
11DeepSeek V4.1 FlashOpenCode44.546.7−2.28183$214m 19s
12Hy4OpenCode43.246.2−3.080113$3412m 38s
13Kimi K3OpenCode42.843.2−0.472240$13216m 07s
14GPT-6 LunaOpenCode42.443.9−1.57571$118m 13s
15GLM 5.3OpenCode39.542.0−2.570150$12720m 33s
16MiMo-V2.6-ProOpenCode38.540.8−2.368150$2117m 02s
17GLM 5.3 FlashOpenCode37.038.4−1.465101$1718m 32s
18Mistral Large 4OpenCode33.236.0−2.860150$11724m 53s
19Qwen3.8 MaxOpenCode32.434.2−1.857150$13825m 01s
20Mercury 3OpenCode29.432.8−3.45782$163m 39s
21Muse Spark 1.3OpenCode26.931.4−4.555300$24922m 14s
22Beam 1.0OpenCode24.926.0−1.14884$235m 00s
23Nemotron 3.5 Super VLOpenCode19.424.6−5.24635$4411m 36s
24Laguna S 2.1OpenCode18.122.2−4.14164$1723m 00s
25Inkling SmallOpenCode7.610.8−3.22648$7911m 44s

Select a column heading to sort. Index = correctness + efficiency. First proof: number of problems passed with the first proof submitted. Integrity violations: runs scored as an integrity violation. Cost: total spend recorded across all 100 problems, one run each, at the providers' prices (estimated from token counts where no cost was recorded). Time: median time the model spent generating per solved problem; unsolved problems are excluded. All results were run by Reasonable; none are self-reported.

Methodology
  • Task: the program or protocol model and its specification are fixed; the agent writes the proof (invariants, assertions, lemmas). Each model gets one run per problem. Within that run it submits a proof to Verus, reads the verifier's result, and revises, as many times as its turn and time limits allow. The final proof is again independently verified and checked for integrity violations.
  • Problems: 100 selected from a larger dataset, prioritizing problems that differentiate models: first those that split the top 8 models on pass/fail, then those that split them on passing within 1.5× the expected budget (below). 20 were selected from VerusBench, 50 from VeruSAGE-Bench, and 30 from tlaverus. At most 2 come from one family (a VeruSAGE-Bench module or a tlaverus spec), and at most ~⅓ of the VeruSAGE-Bench ones from any one project.
  • Expected budget: a preset budget of generated tokens and turns per problem. The problems span 9 levels:
    Level123456789
    Tokens7k10k15k20k30k40k60k80k120k
    Turns152020303040405050
    Problems66137111220169
  • Scoring: a pass scores 60, a failure or budget limit scores 0, a voluntary stop scores −12. Every run then loses 3 points per doubling of its usage against the expected budget (the geometric mean of its token and turn ratios, capped at 8 doublings); a pass also gains 3 per halving under it, capped at 8. Failing fast is not rewarded and scores zero. An integrity violation (e.g., an inserted assume or admit) scores −60, with no efficiency adjustment.
  • Verus Proof Index: the mean score over the 100 problems, with the problem set and budgets fixed on November 9, 2026.
  • Definitions: failure: a submitted proof that does not compile or does not verify. Voluntary stop: the agent ending the task without submitting a proof.
  • Setup: Claude models ran in Claude Code, all other models ran in OpenCode. Five open-weight models were served quantized, which may affect their results: GLM 5.3, MiMo-V2.6-Pro, GLM 5.3 Flash, and Inkling Small at fp8; Laguna S 2.1 at fp4.
  • Access: agents had limited internet access behind a blocklist. A tool call is refused if it fetches from a code host, searches for or fetches a benchmark's source project by name, or searches for an identifier only the task file defines; the model is told why, and each refusal is counted in the trace. Blocks include GitHub and its raw, API, and download domains, GitLab, crates.io, docs.rs, Hugging Face, archive.org, and similar; the VeruSAGE-Bench upstream projects, the AutoVerus/VerusBench corpus, and tlaverus. Verus's documentation is accessible at all times. A proof that arrives from outside is scored as an integrity violation.

02

Proof correctness, efficiency, and length

Verus Proof Index: correctness vs. efficiency

The index splits into correctness, scored from proof outcomes alone, and efficiency, gained or lost for using fewer or more tokens and agent turns than each problem's expected budget ().

Verus Proof Index vs. proof length

Median length of a model's proofs (comments and formatting excluded), as a multiple of each problem's expected shortest proof length.

03

Cost and inference time

Verus Proof Index vs. cost to run

What running the Verus Proof Index cost at the providers' prices, with the Pareto frontier ().

Verus Proof Index vs. inference time

Median inference time per solve, on a selected provider per model, with the Pareto frontier ().

04

Proof run outcomes

Scroll sideways for all models