Simulated worlds

What this project asks

Engineers often decide in simulation before anything is built or run: how fast a line can go, which gripper to buy, whether a controller is ready for the real machine. Building a simulation that is good enough for the decision, in a simulator's editor, and calibrating it against a few measurements is skilled work. Models now also build physics simulations from a sentence, and "generate a world, learn in it, evolve the world" is already being done (SimWorld Studio, EmbodiRSI). Checks on whether those worlds obey physics are only starting: GAUGE measures how physics engines and video world models depart from real measurements, and VeriPhy has an agentic verifier check generated video against typed physical obligations, but neither has an agent design experiments inside an interactive world, on a budget, to find what is wrong and where. When a learner sees only reward, the best-scoring trajectory is the one most likely to exploit the simulator. Some Embodied First Exam suites, such as RoboCasa, use procedurally generated scenes, and some of its task instructions list known simulator defects, but it takes both as given: no task there asks the agent to find a defect or to judge whether a high score is an exploit. This project covers both sides: making a simulation that a decision can rest on, and telling when one cannot be trusted.

What tasks can look like

Any task that tests the question fits. Some directions:

• Build a simulation of a setup from a description, a drawing or a video, in the simulator's editor, together with the probes that show it is physically sound.
• Calibrate a simulation against a few measurements, such as friction, damping, stiffness or a motor's response, so that it predicts runs it was not fitted to.
• Decide with the simulation: choose a speed, a part, a layout or a controller setting, checked by running the choice in the hidden system the simulation stands in for.
• Audit a generated world: accept or reject it, and name what is wrong (gravity, contact parameters, time step, units), including generated robot models whose joint limits, inertia or actuator gains are written wrong in their MJCF or URDF, through a model viewer and its measuring tools.
• Audit a learner: decide whether a high-scoring trajectory exploits the simulator or a lenient success check, for example by passing through a wall in a single time step, with documented exploits as seeds.
• Run the generate, learn and evolve loop for a few generations against a fixed reference or real measurements.
• Physics of increasing difficulty: rigid bodies, joints, contact-rich scenes, deformables, fluids.
• A family rather than one world: a generator that draws the hidden parameters, or plants a different set of faults in each instance, with clean worlds among them.

What a verifier here should establish

• A built or calibrated simulation predicts held-out runs of the hidden system within a stated error.
• A decision made with the simulation works when the benchmark runs it in the hidden system.
• Audit verdicts against known faults and known clean worlds, with the root cause, not just a symptom: naming gravity when a wrong gravity makes friction look wrong.
• For loops, performance on a reference the loop never trained on.
• The work was done through the simulator's own editor, viewer and tools, as a person would do it. Reading or writing the world's files or calling an API directly, if the task allows it, is a control and is reported separately.

Worth reporting

• Prediction error on held-out runs against the number of measurements used
• Decisions that held in the simulation and failed in the hidden system
• Faults caught and false alarms, by kind of physics
• Generations until the learner exploits a hole, against until the audit catches it

For example

A gantry crane carries loads across a shop, and the plant wants the fastest move, for any travel from 2 to 12 m, that leaves the load swinging less than 5 cm when it stops. The agent gets the crane's drawing and films of three moves at known accelerations. It builds the crane in the simulator's editor, fits the load's swing (its pendulum length and damping) to two of the films, checks the fit against the third, and shapes the move so the swing cancels as the crane stops. The verifier runs the move in the hidden system over travels across that range. An agent that takes the cable length from the drawing, leaving out the hook and the sling, plans a move that leaves the load swinging.