Blog
Eventually Correct: Quantifying Agentic Proof-Generation Capabilities in Verus
Relatively small coding models can now generate machine-checkable proofs for simple, self-contained programs using verification frameworks such as Verus. Despite rapid advances over the past year, verifying properties of complex, real-world systems and protocols remains a harder challenge. Even for simpler tasks, however, the ability to generate a proof does not necessarily translate into a practical formal verification workflow. Repeated cycles of “try, fail, patch, retry” can make the process too slow and costly, and are hardly what software teams want from an LLM editing code whose failures have real consequences.
Our collaborations with partners developing formally verified software and our internal model post-training workflows both depend on evaluating and improving model proficiency across a wide range of formal verification approaches and languages. To address this need, today we’re sharing the first in a series of benchmarking results, evaluating the practical capabilities of coding agents in formal verification of software.

Verus Proof Index25 models, 100 problems
- 01GPT-6 Astra59.1
- 02Claude Fable 5.158.7
- 03Claude Opus 5.558.2
- 04Gemini 4 Argon56.2
- 05GPT-6.1 Sol55.6
- 06Claude Sonnet 5.553.1
- 07Gemini 3.8 Flash52.7
- 08Grok 4.745.5
- 09Muse Wonder44.7
- 10Qwen3.8 Flash44.6
- 11DeepSeek V4.1 Flash44.5
- 12Hy443.2
- 13Kimi K342.8
- 14GPT-6 Luna42.4
- 15GLM 5.339.5
- 16MiMo-V2.6-Pro38.5
- 17GLM 5.3 Flash37.0
- 18Mistral Large 433.2
- 19Qwen3.8 Max32.4
- 20Mercury 329.4
- 21Muse Spark 1.326.9
- 22RBeam 1.024.9
- 23Nemotron 3.5 Super VL19.4
- 24Laguna S 2.118.1
- 25Inkling Small7.6
Higher is betterTop 10 of 25Full ranking and methodologyFull ranking →
Our Verus Proof Index measures a model’s ability to eventually generate verifier-accepted proofs, and how efficiently it does that. We evaluated and ranked 25 recent models as autonomous agents, evaluating them on 100 problems selected from our tlaverus dataset, VeruSAGE-Bench and VerusBench, covering verified systems code and distributed protocol models, not just small programs. Details of the Index and how it is computed are in the Methodology section.
Passing: Necessary, but not sufficient
Across the Index, models solve between 26 and 97 of the 100 problems using Verus, with the median at 75. As the strongest models approach the ceiling in eventual pass rate, our Index scoring goes beyond measuring whether a problem is solved, to account for how efficiently agents reach a successful proof.
If the ultimate goal is a formal verification toolkit that helps build, check, protect, and repair software, at a pace compatible with modern CI/CD workflows, then speed and efficiency are as critical as correctness. Engineers need agents that can complete verification tasks autonomously, reliably, quickly, and cost-effectively.
Therefore we care about questions like:
How often is the first candidate solution correct?
How long does it take to reach a successful proof?
How many tokens and how much money did it cost?
The first-solution pass rate measures how often a model's first candidate solution passes verification, before any revision. Across the compared models, the median first-solution pass rate is just 15 out of 100 problems, and even the best models succeed with their first candidate solutions on just 40-64 of the 100 problems. The strongest models need a median of 22 agent turns per passing solution. On systems-code and protocol verification problems, the same models take roughly five times longer than on small textbook problems; to account for the extra difficulty, we extended agents’ time limit to three hours for protocol problems, against 40 minutes elsewhere.
Our Verus Proof Index does not currently score first-solution correctness; we instead report it alongside the Index score. At current performance levels, it does very little to distinguish between the models. 16 of the 25 models pass fewer than 20 problems with first candidate solution without revising it, so scoring first-solution correctness would compress most models into a narrow band near zero. The Index instead scores efficiency alongside correctness: generated tokens and agent turns against a preset expected budget for each problem.
Accurate or efficient. Rarely both.

The models that are most often right first time are not the most efficient models, and vice versa.
Claude Opus 5.5 has the most first-solution passes (64 of 100), but spends a median 46k generated tokens per solution to get there, and scores near zero on efficiency against the expected budget per problem.
Gemini 4 Argon is the most efficient of the leading models (17k tokens per solution, the best efficiency score), with 51 first-solution passes.
High accuracy comes at very different costs. Claude Fable 5.1 shares the top correctness score with Opus but the cost of running all 100 problems that make up the Index is $483; Argon lands within 2.5 points for $54, GPT-6.1 Sol about 3 points behind for $48.
Cost varies significantly, with little apparent relationship to performance. Grok 4.7 was the most expensive model ($543) and scores 45.5; DeepSeek v4.1 Flash and Qwen 3.8 Flash score within a point of it for $21 and $18.
Token generation varies 8×. The least verbose models generate 16-21k tokens per solution (output, including reasoning), while the most verbose generate over 100k. Claude Sonnet 5.5 generates more than twice what Opus does, for a lower score.
Our interpretation is that models optimized primarily for general reasoning and conventional coding tasks carry those learned problem-solving habits into Verus. Strategies that work well elsewhere, such as generating lengthy reasoning traces or repeatedly revising candidate solutions, can become costly in formal verification, where precise invariants and targeted proof steps can lead to a successful proof on the first try.
Beyond the loop: the case for smaller, specialized models
Today, much of the work toward a passing solution is done by the agentic loop. Every model in the index runs the verifier, reads the errors, and revises the proof. To assess how much support the agentic loop provides, we also tested models in a direct proof-generation setting, without the agent’s file-editing and verifier tools. The verifier was still run externally, and models received its feedback for up to ten rounds of revision. As with the agentic evaluation, we allowed one attempt per problem, testing on the Index’s small-program and protocol problems.
On small-program problems, the loop rescues the mid-tier. Qwen 3.8 Flash drops from 95% to 40% without it, Mistral Large 4 from 95% to 35%, Inkling Small from 45% to 10%. The strongest models score roughly the same, regardless (95%).
On large problems, it rescues almost all models. A protocol problem is 12-33k tokens before a proof is added; without tools the model must return all of it, plus the proof, plus its reasoning, in one response. Claude Sonnet 5.5 drops from 100% to 33%, with two-thirds of its attempts cut off at the output limit; GLM 5.3 drops from 97% to 37%; even Opus 5.5 drops to 83%. VeruSAGE-Bench files reach ~68k tokens, so we did not run them without tools due to their size. Part of this performance gap therefore reflects output constraints, rather than proof-generation capability alone.
The exceptions: GPT-6 Astra (97% with tools, 100% without) and GPT-6.1 Sol (100% either way) do as well or better without tools on protocol problems. These results suggest that some models can achieve high success rates without relying on the full tool-enabled agentic workflow.
These results show how much an agentic loop can compensate for a model's limitations, but also point to an opportunity: models that generate correct, concise proofs with fewer iterations could achieve similar verification success at substantially lower cost.
Generating correct proofs on the first try, with fewer tokens and less reliance on the agentic loop, is a capability no model has yet consistently mastered. It is not obvious that this requires, or even suits, the largest general-purpose models. Proficiency in verification languages, familiarity with their proof idioms, and an understanding of the systems being verified are all important. Model specialization is likely to be valuable here, which is also where the wider field seems to be moving: toward models that trade generality for speed and cost on narrow, highly specific tasks (TypeSafe AI's Jev, which returns typed decisions rather than text, is one example beyond coding).
Smaller, specialized, and open-weight models have a significant role to play in this. While the best open-weight model currently trails the highest-performing models in our Index by roughly 15 points, their inherent efficiency and amenability to specialized post-training could make them valuable for deploying formal verification capabilities in practice.
You can’t train for what you don’t measure
Trusted, practical evaluations are essential for training models to become both proficient and efficient across a wide range of verification languages and toolkits. One of the key challenges is measuring genuine problem-solving capability rather than the ability to reproduce memorized or retrieved solutions.
Benchmarks for this purpose require private datasets and problems. Public benchmarks often have solutions available on GitHub, sometimes in the same repository that a task was derived from,making it easy for agents with web access to find them. During initial testing, we observed a small number of agent runs (70 of ~6,500) fetching or searching for a benchmark’s source before producing a correct solution, so we blocked such lookups. Blocking code hosts closes the obvious route, but agents may still find the same information through mirrors, forks, or renamed identifiers. Our tlaverus problems have never been publicly released as Verus code, substantially reducing the risk of exact-match contamination or retrieval of ready-made Verus solutions.
“Eventually correct” at generating verifier-accepted proofs is where the frontier lives today. We published the Verus Proof Index to measure this capability and highlight the gap to fast, token-efficient proof generation that succeeds on the first try. While proof generation is only one part of the broader formal verification workflow, our results suggest that the playing field may be more level than it appears. We'll share more of our practical benchmarks soon.
How to cite this post
@misc{benko2026eventuallycorrect,
author = {Benk{\H{o}}, Beatrix},
title = {Eventually Correct: Quantifying Agentic Proof-Generation Capabilities in Verus},
howpublished = {\url{https://reasonable.io/blog/verus-proof-index/}},
year = {2026},
month = nov,
organization = {Reasonable},
note = {Reasonable blog post, accessed \today},
}

