Deploy-first · Scientist-driven
Can AI agents verify good science?
SciVeri-Bench evaluates scientific critiques, evolving rubrics, and judgments of research quality against those of domain scientists.
Read the project overviewWhat SciVeri-Bench evaluates
Scientific outcomes
Papers, experiments & research artifacts
AI evaluator
Judgments to evaluate
Domain scientist
Reference judgments
- Verification tasks
- 3
- Task-proposal opt-ins
- 16
- Contributor institutions
- 11
- Countries represented
- 5
Three scientific verification tasks
Agents examine papers and research artifacts. Scientists provide the reference judgments across multiple rounds.
Verification task 01
Where does the science fall short?
Identify weaknesses in an AI-generated scientific outcome and support each criticism with evidence from the paper or research artifacts.
- Input
- An AI-generated paper and its research artifacts
- Output
- Scientific weaknesses grounded in the paper and research artifacts
The verification task
Illustrative review example
Research outcome
A paper reports a promising material using a single simulation setting.
Critique response
The claim is broader than the evidence: test sensitivity to simulation conditions and compare against an established baseline.
An example of the task format; no benchmark result is implied.
Built with scientists
Propose a question from your field, review agent outcomes, and refine the scientific rubric. Task managers help prepare and run the experiments through OpenReview.
For open-ended research, evaluation evolves with the evidence. Scientists identify weaknesses, revise criteria, and compare the value of different findings.
What makes a good task?
- Open-ended
- Explain why a single scalar score cannot adequately capture the quality of the science.
- Novel
- Propose a question with the significance and research potential of work in a Nature-family journal.
- Feasible
- Describe the data, tools, compute, and practical scope, from a tractable first experiment to a longer-term moonshot.
Our scientific community
- Brookhaven National Laboratory
- Indian Institute of Technology Delhi
- Institut de Biologie de l'École Normale Supérieure (IBENS)
- Korea University
- Massachusetts Institute of Technology
- Shell
- Technical University of Denmark
- University of Colorado Boulder
- University of Florida
- University of Maryland, Baltimore County
- University of Minnesota
Affiliations reported by task-proposal volunteers · Roster as of Oct 1, 2026. Opt-ins do not indicate accepted benchmark tasks.