Skip to content

Call for task contributors

Bring your expertise.
Shape scientific verification.

We are collecting open-ended, novel research tasks from domain scientists. Propose a question that matters to your field and help judge the science that research agents produce.

Propose on OpenReview Read the process

The OpenReview workspace is for assigned participants.

What makes a good task?

A question with room
for new science.

  • Outcomes need scientific judgment beyond a single score.
  • The question is novel and relevant to your research frontier.
  • The experiments can run in a shareable computational environment.

Prepare your proposal

Start with the science.

You can bring ongoing work from your lab, a challenging open problem, or a new research direction inspired by a recent paper. The proposal should make its scientific value clear.

01

A scientifically meaningful question

Describe the objective, its importance to your field, and what makes the problem open-ended and novel.

02

Data and a runnable environment

Provide the data, software, and environment needed to investigate it. Custom resources must be shareable when the task is released.

03

An initial scientific rubric

Explain what good science would look like. The rubric can evolve as research outcomes expose missing criteria.

04

A reviewer from your field

Nominate a peer with relevant research and review experience to assess the question and the resulting scientific work.

Task managers can help prepare the runnable task package. Scientists focus on the research question, scientific review, and the evolving rubric. Task submissions should contain the required software and data, without bundled agentic skills.

Your role in the loop

Review. Compare.
Refine. Repeat.

The research does not end when an agent produces a paper. Your judgments make the next round more informative.

  1. 01

    Propose and review the task

    Submit your scientific question and initial rubric. Refine the proposal with your nominated reviewers.

  2. 02

    Examine the agent outcomes

    Read the papers and artifacts. Identify scientific weaknesses and evidence for your critiques.

  3. 03

    Compare and revise

    Record pairwise scientific preferences and submit a revised rubric that reflects what the experiments revealed.

  4. 04

    Continue the research

    Agents receive the feedback and revised rubric for another round. An independent external review is a separate phase.

Have an early idea?

Use the initial task intake form to share an outline before preparing a formal OpenReview proposal.

Share a task idea

Meet the community

See the scientists who have signed up to contribute questions from their own research fields.

Task contributors