SCIENCE UNDER AUTOMATION
GeoNatureAgent extended benchmark
Nine models were tested over 103 geospatial tasks. Claude Sonnet 4 scored 61.7% ± 0.7% under strict all-checks scoring.
What this result supports
Tasks use a fixed 16-tool API for three indicators in Spain and Portugal. API task success does not establish field sampling or general Earth-science autonomy.
Evidence record: Primary preprint; October extended evaluation. Checked: 2026-10-21.
Where this evidence is used
Earth & environmental science
- Retrieve and combine the appropriate geospatial observations: GNA’s fixed API evaluation exposes incomplete reliability in choosing and combining operations.
- Check spatial validity and model outputs: Tool-task scoring and forecast comparisons support checks within their evaluated domains.
- Design and carry out physical sampling: The evaluated systems consume existing observations and API data; field collection remains external to those systems.
- Choose measurements that distinguish causal processes: Geospatial analysis and prediction assist planning, without establishing an independent field investigation.
- Validate across sites, instruments and conditions: Transfer requires checks beyond the particular forecasting tasks and regional API coverage.