Implement and test a defined software task
See the requirement-level assessment and its sources.
SCIENCE UNDER AUTOMATION
AI in Computer science: capabilities and evidence
Computing turns specifications into software and tests ideas through executable experiments. Problem selection and verification determine how far those workflows can run unaided.
Autonomy · up to 5.5 / 10
Model interval: 3.2–5.5. Assessed 21 October 2026.
Two equally weighted workflow families: a bounded operational task and an investigation or deployment beyond that task. Inherited subfields use this parent scenario.
Calculation and work mix
Each requirement uses the operational scale from 0 (independent execution unestablished) to 4 (reliable independent execution in the stated setting). A workflow takes its lowest requirement level. The two workflows receive equal weight; the headline is 1 + 9 × weighted level / 4.
These levels are source-linked model assignments, not measurements reported by the papers. “Up to” is the upper model bound, not a probability or a forecast date.
Implement and test a defined software task: 1–2 / 4.
Develop and validate a new computational research result: 1–2 / 4.
Upper bound with 75:25 workflow weights: 5.5. With 25:75 weights: 5.5.
What has AI demonstrated?
Repository-level agent evaluation changes materially after removing answer leakage and repairing task definitions.
- Test
- Primary benchmark preprint; revised October 2026
- Comparison
- Comparators vary by study; see the linked sources.
- System and version
- SWE-Bench Pro Verified, v3 · 2026-10-09
- Evidence checked
- 2026-10-21
Passing repository tests does not establish independent requirements discovery, secure deployment or long-term maintenance. No leaderboard percentage is used as an autonomy percentage.
What does the field require?
Implement and test a defined software task
- Interpret a supplied specification
Autonomy level: 2–3 / 4.
Repository tasks provide the brief; agents can work through it, with uneven reliability.
SWE-Bench Pro Verified, v3 · METR task-completion time horizons - Modify code and execute tests
Autonomy level: 2–3 / 4.
Tool-using agents execute connected coding tasks; failure and benchmark leakage preclude reliable independence.
SWE-Bench Pro Verified, v3 - Check functional correctness beyond visible tests
Autonomy level: 1–2 / 4.
The benchmark repairs expose dependence on the quality and separation of evaluation checks.
SWE-Bench Pro Verified, v3
Develop and validate a new computational research result
- Generate and implement a research idea
Autonomy level: 2–3 / 4.
Idea generation and experiments are connected in the AI Scientist configurations.
Towards end-to-end automation of AI research - Run and analyse computational experiments
Autonomy level: 2–3 / 4.
The demonstrated pipeline runs experiments and analyses outputs within a supplied environment.
Towards end-to-end automation of AI research - Establish validity through independent scrutiny
Autonomy level: 1–2 / 4.
Generated self-review is followed by external review; one accepted workshop manuscript does not establish routine independent validation.
Towards end-to-end automation of AI research
Research tasks
Develop and validate a new computational research result
See the requirement-level assessment and its sources.
Can AI replace the whole workflow?
The results above measure specific tasks. The field assessment also requires completion of the connected workflow, including the requirements listed here. A field-wide replacement percentage has not been established.
Read the assessment method