Machine programming

What this project asks

Most machines in a plant run a program written in the vendor's own tool: a robot arm through its teach pendant or offline programming software, a CNC machine through toolpaths made in CAM, a conveyor or sorting cell through PLC logic. Writing those programs, making them fast enough, and keeping them running through new parts and changed cells is a large share of manufacturing engineering, and it is done on screens. Embodied First Exam has coding agents program simulated robots in Python from a shell, and grades the outcome of each run. PLCWorld runs LLM-written Structured Text against a closed-loop simulated plant and scores task success and safety violations separately; the program is written as code. Here the agent works in the tool a technician uses, and the benchmark runs the program on a simulated cell: the work counts when the cell makes the right parts, at the rate the job needs, with no collision and no safety violation.

What tasks can look like

Any task that tests the question fits. Some directions:

• A new job from a work instruction: a pick-and-place, machine-tending or palletizing routine in the arm's pendant or offline programming tool (for example PolyScope in Universal Robots' URSim), from a drawing of the cell and the job's targets.
• A CNC part: from a drawing and the stock, toolpaths in a CAM tool (for example FreeCAD's CAM workbench) that cut the part in tolerance, without gouges or broken tools, within a machining time.
• A PLC line: ladder or structured-text logic (for example in OpenPLC) for a conveyor, a sorting station or a filling line running against a simulated plant, including start-up, stops, jams and the restart after an emergency stop.
• Making a working program better: a shorter cycle, fewer moves that cut nothing, tool wear against machining time, without losing parts or breaking a safety rule.
• A changeover: a new part, a new gripper or fixture, or a second product on the same line, so a program that worked has to be changed and proven again.
• Commissioning and safety: zones, reduced speeds near people, interlocks, and what the cell does when a sensor fails.
• Debugging: a run collides, gouges or jams, because of the program or because the cell is not what its documents say, such as a fixture mounted off, a wrong tool length or a sensor wired inverted.
• A family rather than a single instance: one cell with a generator that draws the part, the layout or the fault for each instance.

What a verifier here should establish

• The final program, run by the benchmark on the true, hidden cell, does the job: the right parts, in tolerance, at the required rate, with no collision and no safety violation, including on part or layout variants that the work instruction covers but the agent did not try.
• Where the task scores cycle time, machining time, tool wear or energy, the score is measured on the hidden cell, against a reference program.
• Where a task asks the agent to name a fault, the named fault is confirmed by running a reference program on the true cell.
• The graded run went through the programming tool (a pendant, a CAM tool or a PLC editor), as a person would use it. Writing the program as code through an API, if the task offers one, is a control and is reported separately.

Worth reporting

• Success and cycle time against the number of simulated runs used
• Safety violations, including those that did not stop the job
• The same task through the programming tool and as code, the way Embodied First Exam's agents work

For example

A UR5e tends a press: it takes a blank from a fixture, loads the press, waits for the stroke and puts the finished part on a conveyor. The job needs 120 parts an hour, and the work instruction says the arm must slow to 250 mm/s inside the operator's zone by the press door; the cell's safety configuration does not enforce that yet. A program taught point by point from the drawing works, but makes 95 parts an hour. Blending the moves and opening the gripper while it approaches the next blank gets 128. Cutting through the operator's zone at full speed gets 140 and fails. The verifier runs the program on the hidden cell for an hour of simulated time, counts good parts, and checks the arm's speed inside the zone.