How to contribute

Your agent builds the task. People review it.

Pick a project, have your agent build a task that tests its question, with a verifier made for that task, run the frontier models on it, and submit. Beyond a few basics, how the task and its verifier work is up to you.

Submit a task How a task gets good Review tasks Propose a project

Submit a task with your agent

Every project page has this prompt with its project filled in.

Pick a project from the areas and read what it asks.
Give your coding agent the prompt from the project page, in a clone of the repository. It reads AGENTS.md, starts the task with scripts/new_task.py, builds it and its verifier, calibrates the verifier with scripts/check_task.py, has a separate red-team agent try to pass without doing the task (recorded in results/quality.yaml), and runs scripts/validate.py. It starts the red-team agent itself, as a subagent that does not see its conversation, so you do not open another session.
Run the frontier models on your machine, never on GitHub: GPT-6-Astra, GPT-6.1-Sol, Claude Opus 5.5, Claude Fable 5.1, 3+ trials, including a model other than the one that built it. scripts/run_trial.py runs them with Codex CLI or Claude Code held to the task's screen (a container with no task files, or a subagent with only screen tools), grades them and records them in results/summary.json; an agent merely told to stay off its shell does not count.
Read the traces. For each failed trial, decide whether the agent lacked the capability or the task got in the way (the environment broke, the verifier misjudged, the prompt was unclear), and record it in the run's failures. Send the second kind back to your agent, or drop the task. At least one trial has to fail for lack of the capability: if every trial passes, send the task back to your agent to make it harder.
Have a separate agent audit it with the audit prompt. It records a verdict and evidence for each of the five agreements in results/quality.yaml.
Open a pull request with one or more tasks from the same project, and link the generation log. Task check comments on anything missing, and the board shows the task's ten protocol items. It is ready for review when all ten are met.

Build a benchmark task for the CUA-frontier project
"<project name>" (<project-id>).

1. Read AGENTS.md and the project's PROTOCOL.md.
2. Pick a task that tests the project's question:
   python scripts/new_task.py <project-id> <task-id>
3. Write task.md, env/, a verifier that fits this
   task, and its intent. State every criterion the
   verifier scores; keep the facts it checks reachable
   only through the interaction the task tests (the GUI).
4. Calibrate the verifier: oracle/ it accepts,
   negatives/ it rejects, shown by a check: command
   run with scripts/check_task.py.
5. Have a separate red-team agent try to pass
   without doing the task, in a workspace prepared
   by scripts/red_team.py, as a subagent that does
   not see your conversation. Never launch another
   agent CLI for it. Fix what works; a lucky
   guess may stay if its chance is under 1% and
   documented. Record every attempt under
   red_team: in results/quality.yaml.
6. Fill in metadata.yaml, run scripts/validate.py.
7. Stop and report what you built, the shortcuts
   tried, and where your trajectory is saved.

How a task gets good

Different agents build the task, attack its verifier, attempt it and audit it, and people read the traces. Four gates send it back before it reaches review.

Each role is a different agent or person, and none grades its own work. The verifier is checked against known answers, a separate red-team agent that has not seen oracle/ tries to pass without doing the task, the frontier models' failures are read one by one, and a separate audit agent checks the five agreements below. Dashed lines send the task back to be built again. Each step leaves a record in the task folder, and the board shows them as the task's protocol tags.

A task is four parts. Most bad tasks are bad because two of them disagree. Each number is one agreement: the audit agent records whether it holds, with evidence, and the board shows it as one of the task's protocol tags.

1
No hidden requirements prompt · verifier
Every criterion the verifier scores, such as what counts as fixed or how an estimate is measured, is stated in task.md or in something the agent can reach in the environment. This is about the criteria, not the answer.
2
No back doors world · verifier
The facts that satisfy those criteria are reachable only through the interaction the task tests: for computer use, the GUI and computer-use actions. Tools a person doing the work would use, such as a shell, a notebook or git, are fine when task.md offers them, as long as none reaches those facts another way: no file://, shell or API into hidden state, the verifier or oracle/, and no files in env/ that give the answer away.
3
No reward hacking verifier · intent
A pass means the capability in the intent was shown. Output of the right form without the work, edits to what the verifier reads, and defaults it happens to accept all fail.
4
A real request prompt · intent
task.md reads like a request a real user would make, with no hints a real user would not give: naming the cause, listing the steps of the fix.
5
Legitimate failures world · intent
When a strong agent fails, it is for lack of the capability, not because the environment broke, the verifier misjudged or the prompt was unclear.

The audit

After the frontier runs, ask your building agent to run the audit, and it gives this prompt to a subagent that does not see its conversation; or give it to an agent yourself, in a fresh session that did not build the task. It checks the five agreements against the task and the traces, and writes a verdict with evidence for each under audit: in results/quality.yaml. The board then shows each agreement as met, missing or broken.

The red-team pass, earlier, works the same way: scripts/red_team.py copies the task without oracle/ to a workspace of its own, your coding agent starts a subagent that does not see its conversation to look for loopholes there, and records every attempt under red_team:, as AGENTS.md describes.

Audit a CUA-frontier task before it is submitted:
tasks/<project>/<task-id>/ in this clone. You did not
build it. Judge it as a strict reviewer would, and
change nothing but results/quality.yaml.

1. Read AGENTS.md ("Keep the parts aligned"), then
   metadata.yaml (its intent), task.md, env/,
   verifier/, oracle/, negatives/, results/summary.json
   and the trajectories it links, and the red-team pass
   in results/quality.yaml.
2. For each agreement (no_hidden_requirements,
   no_back_doors, no_reward_hacking, real_request,
   legitimate_failures), decide whether it holds,
   with concrete evidence: a file and line, a trial,
   a shortcut you tried. Check every failures entry
   in results/summary.json against its trajectory.
3. Write your verdicts under audit: in
   results/quality.yaml: agent, model, log, and
   agreements.<name>.holds and .evidence for all five.
4. Run python scripts/validate.py on the task, and
   tell me which agreements do not hold and why.

What every task needs

Kept short on purpose. The project says what it is after; the task and its verifier take whatever form fits.

Qualifying models GPT-6-AstraGPT-6.1-SolClaude Opus 5.5Claude Fable 5.1 later releases, once added to requirements.yaml

R1
Generated by an agent, on the record
An agent builds the task. Say which agent and model in generated_by, link its log if you can, and name the people submitting it, who answer the reviewers.
R2
Its own verifier, calibrated
Every task brings a verifier built for that task, in whatever form fits it. It runs without a person, the agent under test cannot see it, and verifier/README.md says how to run it and what it checks. Show that it decides correctly: a local check (check in metadata.yaml, run with scripts/check_task.py) shows it accepting a reference solution in oracle/ and rejecting plausible wrong ones in negatives/; and a separate red-team agent tries to pass without doing the task, at least once each through the interface, the verifier's inputs, output of the right form only, and defaults. Every attempt is recorded in results/quality.yaml, and every one that worked is fixed. A guess that passes only by chance may stay instead if its chance per trial, all such guesses together, is under 1%: give it as chance under the attempt in results/quality.yaml, and show how it was computed in verifier/README.md.
R3
Frontier results, end to end, at least one failure, each explained
Run the whole task with at least one qualifying model, 3 or more trials, and link the trajectories. If a qualifying model generated the task, also run a different one. A trial counts only if its harness holds the agent under test to the task's interface, which each run records as its isolation (box, tools or api; scripts/run_trial.py runs trials this way, and a task with several agents under test may bring its own harness that holds each one the same way); an instruction to stay off its tools does not count. Read the traces and say, for each failed trial (failures in results/summary.json), whether the agent lacked the capability the task tests or the task itself got in the way (the environment broke, the verifier misjudged, the prompt was unclear). Fix the task until only the first kind remains. At least one trial has to fail for lack of the capability: if every trial passes, the task is too easy for this benchmark, so make it harder along the axis its project tests (a longer horizon, a noisier world, a subtler cause), not with arbitrary hurdles, and run the trials again.
R4
Runnable by someone else
task.md, env/ and the verifier are enough for a reviewer to run the task and its verifier again, with pinned versions and no network at evaluation time.
R5
Reviewed by people
A reviewer other than the submitter approves it. Checks run on the author's machine, not on GitHub; reviewers read the local check report and can rerun it on theirs.
R6
A stated intent
intent in metadata.yaml says, in a sentence or two, which capability the task measures and what a pass should mean. The verifier, the red-team agent and the reviewers are all judged against it.
R7
Prompt, world and verifier agree
Every criterion the verifier scores is stated in task.md or in material the agent can reach in the environment: no hidden requirements. The facts needed to meet those criteria, such as the state, measurements or causes the verifier checks against, are reachable only through the interaction the task tests, such as the GUI and computer-use actions. The agent may also have tools a person doing the work would use, such as a shell, a notebook, git or exported data, when task.md offers them, but none of them may reach those facts another way: not the verifier, oracle/, the world's hidden state or ground truth, nor logs or files that give the answer away. task.md reads like a request a real user would make, without hints a real user would not give, and failures come from missing capability. A separate audit agent checks each of these five agreements and records a verdict with evidence in results/quality.yaml; the board shows each one, and reviewers confirm them.

Recommended, not required

A page that shows how the agents ran the task, such as an HTML trajectory viewer under results/. Reviewers open it with scripts/review.py.
A second correct solution, different from the oracle, that the verifier also accepts, when the task admits more than one.
A rubric for what code cannot check, as an addition to the verifier, never instead of it.
Coverage tags (coverage in metadata.yaml) where the area has a coverage.yaml, so the area page shows where the task sits.

A task folder

The required parts, and the usual optional ones. Inside env/ and verifier/, anything goes.

tasks/<project>/<task-id>/
├── task.md          # what the agent under test gets
├── metadata.yaml    # project, intent, generated_by, submitted_by
├── env/             # what it runs in, any form
├── verifier/        # built for this task
│   └── README.md    # how to run it, what it checks,
│                    # shortcuts tried
├── oracle/          # a solution it accepts
├── negatives/       # wrong solutions it rejects
└── results/
    ├── check-report.json # your local check
    ├── quality.yaml      # red-team pass, audit
    └── summary.json      # frontier runs, failures

intent: >-
  Whether an agent can find a regression it can
  only see by playing, and fix its cause.
generated_by:
  agent: Claude Code
  model: claude-opus-5-5
  log: https://…        # the generation trajectory
submitted_by:
  - name: Your Name
    github: your-handle

# results/summary.json
{ "runs": [
    { "model": "gpt-6-astra", "harness": "Codex",
      "trials": 3, "passes": 1,
      "trajectories": "https://…",
      "failures": [
        { "trial": 2, "legitimate": true,
          "why": "never opened the inspector" },
        { "trial": 3, "legitimate": true,
          "why": "fixed the symptom only" } ] } ],
  "metrics": { anything else you measured } }

Four stages

Each stage maps to something on GitHub, and the board reads it from there.

Proposed
Claim a scenario (optional)

Say which scenario your agent is generating, so others do not duplicate it. A draft pull request counts too.
issue · label scenario
In review
Open a pull request

Your agent builds the task in tasks/<project>/<task-id>/, with a verifier of its own that it calibrates and has a red-team agent test. Run the frontier models on your machine, read the traces, have a separate agent audit it, and commit the records. Task check comments on anything missing, and the board shows its protocol tags.
pull request · reviews
Validated
People review it

A reviewer other than the submitter approves it, reading the local check report and rerunning it on their machine if needed. A maintainer adds validated.
approval · label validated
Merged
Discuss failures, merge

Reviewers and the submitter go through how the frontier agents failed, record it in the pull request, and merge.
merged into main

Reviewing

People do not write the tasks; they decide which ones are good.

Find a task on the board: filter by “Needs a reviewer”.
Take it by commenting /review on its pull request, or ask a teammate with /review @handle. The board shows who is reviewing.
Open it: in a clone of the repository, run python scripts/review.py <pull request number> (the card's Review button shows it). It fetches the pull request without changing your checkout and opens a review page in your browser: the run viewer, the frontier results, whether the author's check report still matches the files, the task and the verifier.
Go through it, yourself or with your agent: the Review panel has a prompt that runs the same command and reports what the agents did and whether the verifier got it right. Then approve or request changes with the gh pr review commands shown on the review page. To rerun the author's check: gh pr checkout <n>, then python scripts/check_task.py <task> --no-report.

The task tests the project's question, its intent says so, and it is not a near-duplicate of one already there.
The board's protocol tags say the records exist and agree with each other; you decide whether they are true. The five agreements hold: no hidden requirements, no back doors, no reward hacking, a real request, legitimate failures. Read the red-team pass and the audit in results/quality.yaml, and try one or two shortcuts of your own.
The verifier accepts oracle/, rejects negatives/, and runs as its README says.
The results include a model other than the one that built the task, qualifying models do not pass every trial, and each failure in the pull request's table is the agent's. Open a trace or two and check.

Approve, or request changes with a note. Requesting changes marks the task as needing revision on the board until the reviewer approves. Reviews count on the People table.

Propose a project

Any area can take new projects. Every area page also has a button for this.

A project is a direction: a question current benchmarks miss, with room for many different tasks. Propose one with its question, what tasks could look like, and how they could be verified. Once maintainers agree, scaffold it with python scripts/new_project.py <area> <project-id> "<Project name>" (or have your agent do it), write its short PROTOCOL.md, and open a pull request.

Propose a project

Your agent builds the task. People review it.

Submit a task with your agent

How a task gets good

The audit

What every task needs

A task folder

Four stages

Claim a scenario (optional)

Open a pull request

People review it

Discuss failures, merge

Reviewing

Propose a project