Real time

Where we want tasks

• Every decision × Several tempos. Grade every decision as it lands, at several tempos. This is the curve a decision model is judged on, and nobody has drawn it for agents working a screen.
• Service level × Several tempos. Turn up the world's speed or the arrival rate until the agent's service level breaks, and report the tempo it sustains.
• Failing well × Several tempos. Go past the tempo the agent sustains and judge how it fails: what it drops, whether its limits hold, and how fast it catches up once the rush is over.
• Service level × A whole shift. Keep a service level for a whole shift, long enough to fall behind and recover.
• Load × Hand off or shed. More work than anyone could handle: hand off, shed the least important items or ask for help, instead of failing quietly.
• Bursts × Get ready early. A rush that can be seen coming, such as a concert letting out or a sale opening: use the lull to get ready, then keep up.
• Brief events × Where to look. Events shorter than one look: the agent decides where and when to look, which a benchmark that pushes one screenshot a second cannot test.
• Several screens × Where to look. Several apps or tabs on their own clocks, such as a queue, a map and a chat: what to watch, and when to switch.
• Load × Interrupt and resume. Urgent items cut in while the agent is midway through other work; it has to stop, handle them, and pick up where it left off.
• Live queues × What first. A live queue with no time for every item, where the agent decides what to take first.
• Emergency response × What first. Emergency calls that keep coming, where the agent decides which call comes first and which unit goes.
• Control rooms × What first. An alarm flood or a crowded sector, where the agent decides what to handle first.
• Markets and auctions × Who gets what. Work an order over time, deciding how much to trade now and at what price, with position limits held at every moment.
• What first × Choice questions. Triage where each decision is a pick among items on the screen, so decision models can run it as well as agents that see only pixels.

Prior work in the matrix

RB Realtime-Bench

Oct 2026. 50 tasks (20 public): 30 web apps built for it and 20 browser games, five minutes each, one screenshot a second pushed from a single page, pass or fail at the deadline. Wall, paused and token clocks (with a token-rate ablation), low and high effort, screenshots with Set-of-Marks, and choice questions for decision models. The best model passes 44% on the wall clock.

RTG Real-Time Reasoning Gym

ICLR 2026. Freeway, Snake and Overcooked as text environments, with time pressure (a token budget per step) and cognitive load as separate knobs; normalized game score.

WFLS Win Fast or Lose Slow

NeurIPS 2025. A trading simulation that fills the fastest agent first, and Street Fighter from text game state; trades model size and quantization against latency. Daily yield and win rate.

VGB VideoGameBench

Ten 1990s Game Boy and MS-DOS games (three secret) from raw frames. The game keeps running while the model thinks; the Lite variant pauses it. The best models complete under 2%.

GW GameWorld

34 browser games and 170 tasks with metrics read from game state, played with keyboard and mouse or through semantic actions, with a study of real-time play.

GPTNT GPTNT

Two agents defuse Keep Talking and Nobody Explodes bombs against a live countdown in the real game. No model defused a bomb in real time; human players did.

Gaia2 Gaia2

1,120 scenarios in simulated phone apps (email, messaging, calendar, shopping) that run asynchronously while the agent works through tools; a Time split needs actions at the right moment, and other people's messages and app agents with their own policies arrive during a run.

EFE Embodied First Exam

Oct 2026. Coding agents program simulated robots on 1,090 tasks from a shell; some scenes run both paused and on a clock that does not stop, at a slow-down fixed per task.

OSW OSWorld and other static CUA benchmarks

Desktop and web tasks in real software where the screen holds still while the agent thinks; pass or fail on the final state. Time is not part of the task.

Where we want tasks

• Games × Released. Released games at full size and normal speed, such as open-source racing, real-time strategy, rhythm and management games pinned to a build, played for a whole match or session and judged by the game's own score.
• Games × Under a second. Inputs that have to land within a fraction of a second: a racing line, a rhythm note, a parry. A loop that looks once a second cannot play them; an agent on a stream with a fast decision model might.
• Robots × Under a second. A robot whose physics keeps running while the agent decides: catch a thrown object, stop before contact, keep a carried load level. Real-time chunking studies this for trained policies; here an agent commands the robot and the clock does not stop for it.
• Robots × Seconds. Work that moves at its own pace: pick parts off a running conveyor, sort items before they pass, steer a mobile base among people walking by. The agent gets seconds, not a frozen scene.
• Robots × Works alongside. Work beside a simulated person on their own schedule: take a handover when it is offered, give way when they step into the robot's path.
• AI glasses × Steps in. Warn before a mistake can no longer be undone: the wrong part, a skipped step, a hand near a hot pan. A warning counts only if it lands while it can still change the outcome.
• AI glasses × Helps a person. Coach a person through a repair, an assembly or a recipe from what their glasses see: the next step when they reach it, not before they need it and not after they have moved on.
• AI glasses × Branching replay. Real first-person footage filmed with several continuations, so a warning given in time plays the version where the person stops, and real video is judged on what the agent's timing changes, not only on the timing.
• AI glasses × Closed loop. A simulated first-person world with a scripted person who reacts to what the agent says, for situations that cannot be filmed safely or often enough.

Prior work in the matrix

RB Realtime-Bench

Oct 2026. 30 web apps built for it and 20 games adapted from popular open-source games, five minutes each on the wall clock with one screenshot a second; paused and token clocks as comparisons.

OSW OSWorld and other static CUA benchmarks

Desktop and web tasks in real software, where the screen holds still while the agent thinks.

VGB VideoGameBench

Ten 1990s Game Boy and MS-DOS games from raw frames. The game keeps running while the model thinks; the Lite variant pauses it.

GW GameWorld

34 browser games (runner, arcade, platformer, puzzle, simulation) and 170 tasks checked from game state, played with keyboard and mouse or through semantic actions, with a study of real-time play.

WFLS Win Fast or Lose Slow

NeurIPS 2025. Street Fighter on the DIAMBRA platform, played from text game state on the game's own clock, trading model size and quantization against latency. Its trading simulation is in the matrix above.

GPTNT GPTNT

Two agents defuse Keep Talking and Nobody Explodes bombs against a live countdown in the real game. No model defused a bomb in real time; human players did.

LIB LIBERO and other robot benchmarks that step the simulator

NeurIPS 2023. 130 manipulation tasks in four suites, in simulation, for trained policies. The simulator advances one step per action, so the scene waits while the policy decides.

RTC Real-Time Execution of Action Chunking Flow Policies

NeurIPS 2025. A way to run robot policies whose inference outlasts a control step: 12 dynamic tasks in the Kinetix simulator under simulated inference delay, and six bimanual tasks on a real robot. Success and throughput, for trained policies.

VIA VIA, Visual Interface Agent for Robot Control

Jul 2026. Unmodified coding agents drive a robot through a browser interface: they place a virtual gripper over a point cloud and camera views, then call a tool that moves the real one. Six tabletop tasks in simulation and five on a real mobile manipulator, all quasi-static, so the scene waits for the agent.

EFE Embodied First Exam

Oct 2026. Coding agents program simulated robots on 1,090 tasks from a shell; some scenes run both paused and on a clock that does not stop, at a slow-down fixed per task.

HA HoloAssist

ICCV 2023. 166 hours of first-person video from 350 instructor and performer pairs, in which an instructor steps in as the performer works. Benchmarks include mistake detection and predicting the type of the instructor's intervention from the seconds before it. Released under CDLA-Permissive-2.0.

PA ProAssist

EMNLP 2025. Assistant dialogues synthesized from annotated first-person videos, cooking among them, posed as a streaming task: what to say, and when, as the video plays. Automatic metrics, checked against people's judgments.

OI OmniInteract

May 2026. 250 real-world videos with 1,430 response slots, among them gym exercises, shopping and step-by-step guidance, streamed with sound to live audio-visual models that decide whether, when and what to answer, through interruptions. The best model scores 0.368 on its quality-and-timeliness F1.

AR Aroma

UIST 2025. A cooking assistant for blind and low-vision people that joins a recipe video, a wearable camera and the cook's own reports of touch and smell, answers questions and raises alerts on its own; evaluated in a study with eight participants.

OA Object Agents

UIST 2025. A ceiling camera and a vision-language model infer what a person is about to need, and everyday objects on robotic bases move to help them; a system and user study, not a benchmark.