Cross-system workflows

Training environment0 tasks

After a side effect with an unknown outcome, does the agent check before it acts again, and keep every system consistent?

Business tasks spread across several systems, where a side effect can fail or time out and a correct-looking final state can hide a wrong process.

What this project asks

OSWorld 2.0 and CUA-Gym cover many workflow tasks, and CUA-Gym notes that a final-state check cannot tell a legitimate operation from a destructive rebuild. It cannot tell "done right" from "done twice" either. Timeouts with unknown outcomes, look-alike customers, and domain rules nobody wrote down are everyday in operations work.

What tasks can look like

Any task that tests the question fits. Some directions:

Refunds, returns, account changes or provisioning across ticketing, orders, payments and inventory.
Faults at set moments: a timeout that actually succeeded, a duplicate webhook, a delayed update, a customer changing the request halfway.
Look-alike entities, and domain rules the task does not state.

What a verifier here should establish

The goal reached exactly once, every system left consistent, and operations in a valid order.
The work went through the interface, not straight into the database.

Worth reporting

Duplicate writes and wrong entity bindings
How much a final-state-only check would overestimate success

For example

A partial refund returns a 504 timeout. Retrying issues a second refund, and the ticket still reads resolved. An agent that checks the payment ledger first sees the refund went through and does not retry.

Build a benchmark task for the CUA-frontier project "Cross-system workflows" (cross-system-workflows). Repository: https://github.com/kingofspace0wzz/CUA-frontier 1. Read AGENTS.md and areas/workflows/cross-system-workflows/PROTOCOL.md: what the project asks. 2. Pick a task that tests that question. Start it with: python scripts/new_task.py cross-system-workflows <task-id> 3. Write task.md, env/ and a verifier that fits this task in verifier/, and state in metadata.yaml (intent:) which capability it measures. Every criterion the verifier scores must be stated in task.md or the env; the facts it checks must be reachable only through the interaction the task tests (for computer use, the GUI): tools you offer, such as a shell, must not reach them another way. 4. Calibrate the verifier: a reference solution in oracle/ it accepts, plausible wrong ones in negatives/ it rejects, and a check: command that shows both; run it with python scripts/check_task.py. 5. Have a separate red-team agent, which has not seen oracle/, try to pass without doing the task. Prepare its workspace with python scripts/red_team.py tasks/cross-system-workflows/<task-id> and give the prompt it prints to a subagent that does not see your conversation (in Codex, fork_turns: "none"). Never launch another agent CLI (codex exec, claude -p) for it. Fix what works (a guess that passes by chance may stay if its chance per trial is under 1%, shown in verifier/README.md), and record every attempt under red_team: in results/quality.yaml. 6. Fill in the rest of metadata.yaml, including generated_by, and run python scripts/validate.py tasks/cross-system-workflows/<task-id> 7. Stop and report what you built, the shortcuts tried, and where your trajectory is saved. Do not run the frontier evaluation yourself.