The Scientist-Verified Benchmark
SciVeri-Bench asks whether AI agents can evaluate what constitutes good science as domain scientists do. We collect open-ended research tasks and the scientific judgments made as their outcomes evolve.
The evaluation target
Research agents produce papers and artifacts. Evaluator agents then make scientific judgments about those outcomes, with domain scientists providing the reference.
TASK 01
Where does the science fall short?
Identify weaknesses in an AI-generated scientific outcome and support each criticism with evidence from the paper or research artifacts.
TASK 02
What did the rubric miss?
Recognize scientific problems that an existing rubric does not cover, then revise its criteria in light of the observed outcomes.
TASK 03
Which outcome is better science?
Compare scientific outcomes for novelty, meaning, and quality of evidence, and assess how an agent’s preferences align with scientists’ judgments.
The benchmark studies agreement with scientists’ verification judgments. Successful execution or a high automated task reward alone does not establish the scientific quality of an outcome. Final evaluation metrics and results will accompany the benchmark release.
How the evidence is collected
Scientists and research agents iteratively improve the investigation. Each round records the critiques, rubric revisions, and preferences needed to study scientific verification.
Domain scientist
An open scientific question, a runnable environment, and an initial rubric.
Research agents
Experiments, a scientific paper, and the artifacts that support its claims.
Domain scientist
Evidence-backed critiques and pairwise judgments of the scientific outcomes.
Scientist + agents
A revised rubric and scientific feedback inform the next research round.
The papers and task-specific artifacts, their scientific weaknesses, and which outcomes offer more meaningful findings. Critiques point to supporting pages, figures, tables, or artifacts.
Run research agents in the prepared environment, collect their outcomes, and make the papers and feedback forms available through OpenReview. The next round receives the revised rubric and the scientific feedback.
Task collection procedure
Proposal review and iterative investigation establish a candidate task. Independent external review is a separate check on its scientific value.
The scientist submits a research question, its significance, required resources, and an initial rubric. Nominated reviewers from the same field help refine its novelty, open-endedness, and scientific meaning.
After proposal review, the task package is prepared for agent execution. Scientists review the resulting papers over multiple rounds and revise the rubric as new weaknesses emerge.
An external scientist in a related field, independent of the proposer and nominated reviewers, assesses the proposed task, its rubric, and the resulting findings.
The review considers whether the work is interesting, novel, and scientifically valuable enough for inclusion in SciVeri-Bench. The detailed external review procedure is still being developed.
The science comes first. A task manager can help translate the proposal into a runnable package.
Design principles
A task’s outcomes cannot be adequately evaluated with a single scalar score. The scientific question leaves room for alternative approaches, discoveries, and expert interpretation.
Scientists propose problems with significance for their field. Review considers whether the direction is scientifically meaningful and has the potential for publication in a Nature-family journal.
Rubrics begin as scientists’ best current criteria. Research outcomes reveal weaknesses that those criteria missed, prompting additions, revisions, and more informative experiments.
OpenReview connects scientists and AI researchers. Scientists focus on questions and judgments; task managers and the harness handle execution and publish outcomes for review.
Help define what good science means in your field.
Looking for the supporting resources? Browse the research task catalog, catalog statistics, or tools and simulation library.
