PDAgent-Bench and PostEDA-Bench give agents physical-design flows with edits across RTL,
constraints and flow settings, but do not test which layer the agent chose to change. ORFS-agent
tunes the flow with the RTL fixed, TicTacBench revises RTL with the flow fixed, and CLOSER-Bench
proposes comparing them across stages, with its full experiments still pending. Here the agent
has all the layers at once.
What tasks can look like
Any task that tests the question fits. Some directions:
RTL to layout with an open flow: meet timing, area and power, and fix DRC violations.
Problems placed in different layers: RTL structure, constraints, floorplan, flow settings.
Reading layouts and timing reports in the tools' GUIs.
Designs deployed to an FPGA and measured on hardware, where that is available.
What a verifier here should establish
The targets are met without relaxing them, including signoff checks such as DRC and
gate-level simulation.
Worth reporting
Which layer changed, tool runs and wall-clock
Generate a task with your agent
Give this to your coding agent in a clone of the repository. It picks a task that tests this project's question, builds it with a verifier of its own, and checks the package. You run the frontier models and open the pull request; people review it there.
Build a benchmark task for the CUA-frontier project "Physical implementation" (physical-implementation).
Repository: https://github.com/kingofspace0wzz/CUA-frontier
1. Read AGENTS.md and areas/chip-eda/physical-implementation/PROTOCOL.md: what the project asks.
2. Pick a task that tests that question. Start it with:
python scripts/new_task.py physical-implementation <task-id>
3. Write task.md, env/ and a verifier that fits this task in verifier/, and
state in metadata.yaml (intent:) which capability it measures. Every
criterion the verifier scores must be stated in task.md or the env; the
facts it checks must be reachable only through the interaction the task
tests (for computer use, the GUI): tools you offer, such as a shell, must
not reach them another way.
4. Calibrate the verifier: a reference solution in oracle/ it accepts,
plausible wrong ones in negatives/ it rejects, and a check: command that
shows both; run it with python scripts/check_task.py.
5. Have a separate red-team agent, which has not seen oracle/, try to pass
without doing the task. Prepare its workspace with
python scripts/red_team.py tasks/physical-implementation/<task-id>
and give the prompt it prints to a subagent that does not see your
conversation (in Codex, fork_turns: "none"). Never launch another agent
CLI (codex exec, claude -p) for it. Fix what works (a guess that passes
by chance may stay if its chance per trial is under 1%, shown in
verifier/README.md), and record every attempt under red_team: in
results/quality.yaml.
6. Fill in the rest of metadata.yaml, including generated_by, and run
python scripts/validate.py tasks/physical-implementation/<task-id>
7. Stop and report what you built, the shortcuts tried, and where your
trajectory is saved. Do not run the frontier evaluation yourself.
Tasks in this project
Claims, open pull requests and merged tasks, from the board.