Teleoperation

Main bet0 tasks

Through a human teleoperation interface, which physical tasks can a general CUA do, and what limits it?

The agent drives robots, vehicles or machines through the same screen and controls a person would use, on everyday remote work and on tasks that move faster than it thinks.

What this project asks

Many machines are already driven from a screen and keys or a gamepad: inspection drones, remote forklifts and cranes, robot arms in labs. VIA has an agent set waypoints for a robot arm from a browser view, and its authors note that slow inference limits it to quasi-static tasks. EmbodiedSWE and LIBERO-Agent skip the human interface, and D2E suggests that keyboard-and-mouse control learned on the desktop carries over to robots. Embodied First Exam runs some robot scenes both paused and on a clock that does not stop, but its coding agents drive them with programs through an API, never a human control surface, and each task fixes how much the clock is slowed instead of sweeping it. This project asks what a general CUA can do through the interface a person uses, from routine remote work to fast dynamic tasks, and where latency, viewpoint and control granularity draw the line. (Real-time decisions, in the Real time area, asks the timing question for decisions made on a screen, a control room's among them; this project goes deeper into driving the machine itself.)

What tasks can look like

Any task that tests the question fits. Some directions:

Everyday remote work: inspect a structure with a drone and report what is damaged, pick and pack orders with a remote arm, move pallets with a remote forklift while people walk through the aisle.
Dynamic tasks: catch a rolling ball, push a moving cart, fly a drone through a gate, from camera views and key presses.
The same scenes with static tasks, as a control.
Disturbances at unpredictable moments.
Different controls (discrete keys, analog gamepad, mouse) and different views (one camera, two, occlusion).
Shared autonomy: the same job driven directly, by setting waypoints, or by supervising the robot's own autonomy and taking over when it goes wrong, as many operator screens allow.
Existing robot scenes, such as Embodied First Exam's or their upstream benchmarks', put behind a human control surface, so the same scene also has a result from a coding agent.
The agent's decision latency as the variable: the same scene paused, slowed down, and at full speed.

What a verifier here should establish

Success from the simulator's state, and for work with a product (an inspection report, packed orders) the product against ground truth.
Collisions, near misses and contact with people, counted from the simulator's state.
The agent used only the camera views and the human controls, never privileged state.
A program-driven run of the same scene, if the task offers one, is a control and is reported separately.

Worth reporting

Success against decision latency
Drop in success under disturbances
Success, time and near misses by mode of control: direct, waypoints, supervised

For example

A ball rolls toward the table edge and someone nudges it at t = 1.1 s. An agent acting every 0.1 s sees the new direction and stops it. One that plans a waypoint every 1.2 s is still aiming at the old path, and the ball falls.

Build a benchmark task for the CUA-frontier project "Teleoperation" (teleop). Repository: https://github.com/kingofspace0wzz/CUA-frontier 1. Read AGENTS.md and areas/physical-ai/teleop/PROTOCOL.md: what the project asks. 2. Pick a task that tests that question. Start it with: python scripts/new_task.py teleop <task-id> 3. Write task.md, env/ and a verifier that fits this task in verifier/, and state in metadata.yaml (intent:) which capability it measures. Every criterion the verifier scores must be stated in task.md or the env; the facts it checks must be reachable only through the interaction the task tests (for computer use, the GUI): tools you offer, such as a shell, must not reach them another way. 4. Calibrate the verifier: a reference solution in oracle/ it accepts, plausible wrong ones in negatives/ it rejects, and a check: command that shows both; run it with python scripts/check_task.py. 5. Have a separate red-team agent, which has not seen oracle/, try to pass without doing the task. Prepare its workspace with python scripts/red_team.py tasks/teleop/<task-id> and give the prompt it prints to a subagent that does not see your conversation (in Codex, fork_turns: "none"). Never launch another agent CLI (codex exec, claude -p) for it. Fix what works (a guess that passes by chance may stay if its chance per trial is under 1%, shown in verifier/README.md), and record every attempt under red_team: in results/quality.yaml. 6. Fill in the rest of metadata.yaml, including generated_by, and run python scripts/validate.py tasks/teleop/<task-id> 7. Stop and report what you built, the shortcuts tried, and where your trajectory is saved. Do not run the frontier evaluation yourself.