What this project asks
Most engineering design happens in software: a part is modelled in CAD, prepared for a printer or a machine in a slicer or a CAM tool, and often checked in a simulator before anything is made. CADWorld shows general CUAs can operate FreeCAD, but its best agent completes 17.5% of tasks against 87.0% for an expert, and evaluation stops at the saved file: nothing is built. CADEngBench judges parts by how they behave (manufacturability rules, parameter perturbations, FEA in CalculiX) and P3D-Bench by their geometry and structure, but in both the model writes the part as code and nothing is made or tested; ScienceVision asks questions about supplied CAD assemblies without changing them. Other work covers pieces of the job on their own: revising a design from engineering analysis (Physics-in-the-Loop, Hephaestus-CCX), setting up an analysis or repairing a motion plan (FEABench, AssemblyBench), revising control after construction (BuildArena), adapting to changed physics (PACE-Bench). Embodied First Exam's RoboEngineer suite comes closest from the robotics side: a coding agent retunes the numbers of a fixed controller from deployment telemetry against a hidden plant, but neither the robot nor the controller's structure can change. This project asks for the whole job, through the tools an engineer uses: from a requirement to a design that works when the benchmark builds and tests it, at a good trade-off of weight, cost and time to make, revised from the tests when the first attempt falls short. The proposal suggests starting here, where the physics is easiest to control.
What tasks can look like
Any task that tests the question fits. Some directions:
- Design to a requirement with trade-offs: a bridge spanning 80 cm that holds 5 kg, as light as it can be; a bracket that meets a stiffness limit and prints in a set time.
- Design for a process: a part that has to print without supports, or be machined with the tools a shop has, so the slicer or the CAM tool is part of the job.
- Brackets, trusses, vehicles, enclosures, linkages, fixtures, assemblies, and parts whose physics is not structural: a heat sink that has to keep a chip cool, a nozzle or manifold that has to feed every outlet evenly.
- Requirements that change: a heavier load, a cheaper material or a second load case arrives after the first design works, and the design has to follow.
- Failures to diagnose: the design, how it was carried out (assembly motion, control inputs, test setup, how a measurement was read) or the agent's assumptions about the world can each be wrong, including pairs of tasks with the same symptom and different causes.
- Feedback at different levels, from pass or fail to full telemetry, and a small test budget so the loop cannot turn into brute-force search.
- A deployment loop like RoboEngineer's, applied to designs: a hidden test rig, noisy telemetry, a few deployments charged before they run, and a re-test on cases the agent never saw.
- A family rather than a single instance: one scenario with a generator that draws the requirement, the material or the hidden cause for each instance.
What a verifier here should establish
- The built design meets the original requirement in simulation, including load cases or conditions within the requirement that the agent did not test. Passing by relaxing a load, a support or an acceptance condition fails.
- Where the task scores weight, material, cost or time to make, the score is measured on the design as built, against a reference design.
- Any design that works counts, however the agent got there. Where a task asks the agent to name a cause, the named cause is confirmed, for example by replaying the design under a reference procedure.
- The graded run went through the CAD tool, the slicer or CAM tool where the task uses one, and the lab's screens, as a person would use them. A command-line or API path to the same lab, if the task offers one, is a control and is reported separately.
Worth reporting
- Requirement met, and quality against the reference design, after n tests
- One-shot designs against designs revised from tests, and real feedback against hidden or shuffled feedback
- Where a cause is planted, design changes made when only the execution or the world was wrong
For example
A wall bracket has to carry up to 15 kg anywhere along a 200 mm arm with less than 1 mm of sag, printed in PETG in under four hours. A plain L sags too far, and arms thick enough to stop it take most of a day to print. A gusseted bracket is stiff enough, but printed standing up it splits between layers. An agent that keeps the gusset and prints the bracket on its side meets both limits. The verifier slices and prints the design in simulation, then loads it with 15 kg at points along the arm the agent did not test.