Unknown rules

Training environment0 tasks

Does the agent gather information before it acts, and keep and prune hypotheses about rules nobody told it?

Nobody tells the agent the rules, of a game level or of an unfamiliar piece of software. It should probe cheaply, rule out what it sees is false, and act once it knows.

What this project asks

Game benchmarks such as GameWorld mostly test play under known rules. ARC-AGI-3 tests discovering rules, but not for an agent working from screenshots with keyboard and mouse. Real software has the same problem: nobody tells you what a switch, a setting or an unfamiliar control does, and a skilled user tries it, watches what happens, and then decides. Rules can be drawn from generated families and checked exactly, so this is a good training environment as well as a benchmark.

What tasks can look like

Any task that tests the question fits. Some directions:

A game level whose rules (switches, doors, keys, counters, timing) are unknown until probed.
An application with unfamiliar controls whose effect the agent has to work out before using them for a goal.
Probes that cost something, so the agent has to weigh information against risk.
Held-out appearances, parameters, or mechanisms the agent has never seen.

What a verifier here should establish

The goal was reached and the constraints held, from the system's own state.
The rule could be identified from the probes available.
Optionally, whether the agent confirmed the rule before acting on it.

Worth reporting

Probes used against the minimum needed
Success on held-out mechanisms

For example

A door is controlled by switches A and B, and walking to the door steps on B. Flipping A opens the door; stepping on B closes it. Only "exactly one of A and B is on" survives, and the agent tests it by predicting that stepping on B again will open the door.

Build a benchmark task for the CUA-frontier project "Unknown rules" (unknown-rules). Repository: https://github.com/kingofspace0wzz/CUA-frontier 1. Read AGENTS.md and areas/games/unknown-rules/PROTOCOL.md: what the project asks. 2. Pick a task that tests that question. Start it with: python scripts/new_task.py unknown-rules <task-id> 3. Write task.md, env/ and a verifier that fits this task in verifier/, and state in metadata.yaml (intent:) which capability it measures. Every criterion the verifier scores must be stated in task.md or the env; the facts it checks must be reachable only through the interaction the task tests (for computer use, the GUI): tools you offer, such as a shell, must not reach them another way. 4. Calibrate the verifier: a reference solution in oracle/ it accepts, plausible wrong ones in negatives/ it rejects, and a check: command that shows both; run it with python scripts/check_task.py. 5. Have a separate red-team agent, which has not seen oracle/, try to pass without doing the task. Prepare its workspace with python scripts/red_team.py tasks/unknown-rules/<task-id> and give the prompt it prints to a subagent that does not see your conversation (in Codex, fork_turns: "none"). Never launch another agent CLI (codex exec, claude -p) for it. Fix what works (a guess that passes by chance may stay if its chance per trial is under 1%, shown in verifier/README.md), and record every attempt under red_team: in results/quality.yaml. 6. Fill in the rest of metadata.yaml, including generated_by, and run python scripts/validate.py tasks/unknown-rules/<task-id> 7. Stop and report what you built, the shortcuts tried, and where your trajectory is saved. Do not run the frontier evaluation yourself.