SCIENCE UNDER AUTOMATION
SWE-Bench Pro Verified, v3
Repository-level agent evaluation changes materially after removing answer leakage and repairing task definitions.
What this result supports
Passing repository tests does not establish independent requirements discovery, secure deployment or long-term maintenance. No leaderboard percentage is used as an autonomy percentage.
Evidence record: Primary benchmark preprint; revised October 2026. Checked: 2026-10-21.
Where this evidence is used
Computer science
- Interpret a supplied specification: Repository tasks provide the brief; agents can work through it, with uneven reliability.
- Modify code and execute tests: Tool-using agents execute connected coding tasks; failure and benchmark leakage preclude reliable independence.
- Check functional correctness beyond visible tests: The benchmark repairs expose dependence on the quality and separation of evaluation checks.