SCIENCE UNDER AUTOMATION
SocSci-Repro-Bench
Across 221 tasks from 54 papers, Claude Code achieved 93.4% task accuracy and 78.0% paper accuracy; Codex achieved 62.1% and 35.8%.
What this result supports
Original replication materials were provided. Ten tasks concerned identifying absent materials. Prompt framing could induce confirmatory specification search.
Evidence record: Primary preprint; three independent runs. Checked: 2026-10-21.
Where this evidence is used
Social & behavioural science
- Run the supplied research code and data: SocSci-Repro-Bench demonstrates connected computational reproduction with variable paper-level reliability.
- Identify missing inputs and retrieve further resources: Missing-material detection can succeed, but retrieving new replication resources remains difficult.
- Interpret results without confirmatory distortion: Prompt framing affects specification choices; domain checks remain material to interpretation.
- Validate interpretation in the relevant social context: Cross-country errors and prompt sensitivity require contextual scrutiny.