What this project asks
Much of lab work is done through software: instrument control and method editors, the instrument's own analysis, notebooks, spreadsheets and the lab's records. Lab simulators such as LabUtopia, LabDex and Labimus now give physically and chemically meaningful feedback, but they evaluate carrying out a given procedure. LabOSBench puts GUI agents on eight instrument simulators built for it, from SEM to FIB, and grades each step of the procedure; agents complete many structured steps but struggle with adjustments driven by feedback. ACED-Bench has agents choose interventions against hidden causal graphs, and finds them testing the wrong intervention and going on after the evidence was enough. Embodied First Exam's Embodied Safety Exam includes chemical-handling requests, judged on how the robot handles the hazard; no task in Embodied First Exam grades which experiment to run next, or a numeric estimate against hidden truth. Here the agent owns the investigation: it plans the experiments, runs them on simulated instruments through their software, analyses what comes back, decides what to run next, and reports a result (a property, a best recipe, a decision on a lot) that the benchmark checks against ground truth it never shows. The proposal puts this second, after the engineering track, and it may turn out to be the richer problem.
What tasks can look like
Any task that tests the question fits. Some directions:
- Characterize: estimate an unknown property, such as a flow curve, a glass transition or a tensile modulus, to a stated tolerance.
- Optimize: maximize a yield or a property over recipe and process settings, with a budget that an informed strategy meets and blind search does not.
- Qualify and release: decide whether a lot or a method meets its specification, and record it the way the lab requires.
- Work across the lab's software: a method editor, the instrument's analysis, a notebook or a spreadsheet, the lab's records, where the result is only right if the pieces agree.
- Readings that mislead: drift, a calibration that is off, or a method that measures the wrong thing, so the agent has to notice before it reports.
- Instruments beyond rheology: thermal analysis, a tensile tester, spectroscopy, a titration or another wet-chemistry bench.
- A family rather than a single world: a generator that draws the hidden property or fault for each instance.
What a verifier here should establish
- The reported result against ground truth the agent cannot see: the estimate's error, the objective reached, the decision on the lot.
- Where the agent hands in a method or a recipe, the benchmark runs it again and gets the claimed result.
- Safety limits and the budget held throughout.
- The experiments were run through the lab's own software, as a person would run them. A command-line or API path to the same lab, if the task offers one, is a control and is reported separately.
Worth reporting
- Error against the number of experiments used
- Real readings against hidden or shuffled readings, at the same budget
- Where a reading misleads, how often the agent catches it before it reports
For example
A chemist wants the highest yield from a batch reaction by choosing its temperature and catalyst loading. There is starting material for 12 runs, and the reactor must stay below 90 °C, where a side reaction runs away. Yield rises with both settings until the product starts to break down, so the best conditions lie on a narrow ridge. The agent sets up each run in the reactor's control software and reads the yield from the chromatography report. A 4 × 3 grid spends all 12 runs and finds 71%. An agent that spreads out its first runs, fits a model to them and spends the rest near the predicted peak finds 84%, against a best possible 86%. The verifier runs the agent's final recipe in the hidden lab, checks the reactor's temperature log against the limit, and counts the runs.