When a check fails, can the agent tell whether the design or the test is wrong, and fix the right one?
The agent writes RTL or circuits together with the tests that check them, and debugs failing checks, deciding whether the design, the testbench, the constraints or the properties are wrong.
VerilogEval and RTLLM check generated RTL against reference behavior, and AutoChip and
AnalogCoder repair designs under tests the framework runs. CorrectBench repairs testbenches;
Analog Design Bench lets agents write their own testbenches but does not classify the errors
they fix along the way; FVDebug traces some formal failures to testbench errors and missing
constraints, checked by experts rather than repaired and re-verified. None asks the agent to
choose between fixing the design and fixing the test, with both in its hands.
What tasks can look like
Any task that tests the question fits. Some directions:
From a spec, write RTL and its own testbench, simulate, read waveforms, and revise.
Analog: choose a topology, size devices, write measurement benches, and iterate toward specs
such as gain, bandwidth, noise and power; sizing-only tasks compared with an optimizer.
Start from a failing simulation or a formal counterexample, with the fault in the design, the
testbench, the constraints or the properties.
Start from a testbench that is itself wrong, where fixing the test is the right move.
What a verifier here should establish
The design is correct against checks the agent never saw.
Weakening a property or deleting a check fails.
Optionally, how good the agent's own tests are, for example by running them on mutated
designs.
Worth reporting
Design edits and test edits, counted separately, and edits to layers that were not at fault
Simulations used, and the gap to an optimizer on sizing tasks
Generate a task with your agent
Give this to your coding agent in a clone of the repository. It picks a task that tests this project's question, builds it with a verifier of its own, and checks the package. You run the frontier models and open the pull request; people review it there.
Build a benchmark task for the CUA-frontier project "Design and verification" (design-verification).
Repository: https://github.com/kingofspace0wzz/CUA-frontier
1. Read AGENTS.md and areas/chip-eda/design-verification/PROTOCOL.md: what the project asks.
2. Pick a task that tests that question. Start it with:
python scripts/new_task.py design-verification <task-id>
3. Write task.md, env/ and a verifier that fits this task in verifier/, and
state in metadata.yaml (intent:) which capability it measures. Every
criterion the verifier scores must be stated in task.md or the env; the
facts it checks must be reachable only through the interaction the task
tests (for computer use, the GUI): tools you offer, such as a shell, must
not reach them another way.
4. Calibrate the verifier: a reference solution in oracle/ it accepts,
plausible wrong ones in negatives/ it rejects, and a check: command that
shows both; run it with python scripts/check_task.py.
5. Have a separate red-team agent, which has not seen oracle/, try to pass
without doing the task. Prepare its workspace with
python scripts/red_team.py tasks/design-verification/<task-id>
and give the prompt it prints to a subagent that does not see your
conversation (in Codex, fork_turns: "none"). Never launch another agent
CLI (codex exec, claude -p) for it. Fix what works (a guess that passes
by chance may stay if its chance per trial is under 1%, shown in
verifier/README.md), and record every attempt under red_team: in
results/quality.yaml.
6. Fill in the rest of metadata.yaml, including generated_by, and run
python scripts/validate.py tasks/design-verification/<task-id>
7. Stop and report what you built, the shortcuts tried, and where your
trajectory is saved. Do not run the frontier evaluation yourself.
Tasks in this project
Claims, open pull requests and merged tasks, from the board.