-BenchA benchmark for assembling LEGO sets in CAD from the official building instructions.
Preliminary · results not yet published
-Bench
The official building instructions, a STEP file for each distinct piece, and a parts sheet that pairs every file with a picture and a quantity.
The model under test follows the instructions, writes code that places each piece, and renders its own intermediate results. It is given a single attempt.
The exported STEP file is compared against the original CAD model of the set according to which pieces are in contact.
Comparing the positions of pieces directly would penalise a correctly built model whenever a hinge, wheel or boom happens to be set at a different angle from the reference. Lego-Bench instead records which pieces are in contact with one another. This can be drawn as a diagram in which every piece is a dot and every pair of touching pieces is joined by a line, and the submission is scored by how closely its diagram matches that of the reference.
Results from the runs completed so far. Each row is a single attempt by one model on one set, scored as described above. The protocol is still being refined, and these figures should be read as early observations rather than final rankings.
| Model | Set | Correct contacts | Extra contacts | Score | Time | Cost |
|---|---|---|---|---|---|---|
| gpt-6-luna · max reasoning | 60464 City F1 Race Car | 37 / 95 | 408 | 0.00 | 64 min | $4.94 |
| gpt-6-luna · high reasoning | 30465 Technic Helicopter | 33 / 129 | 201 | 0.00 | 12 min | $0.14 |
| gpt-6-luna · high reasoning | 60464 City F1 Race Car | 17 / 95 | 29 | 0.00 | 46 min | $1.89 |
| gpt-5-mini · high reasoning | 60464 City F1 Race Car | 0 / 95 | 0 | 0.00 | 9 min | $0.19 |
Correct contacts are counted out of the contacts in the reference. The score subtracts one expected contact for every extra contact, so a submission with many pieces passing through one another scores zero even when some of its contacts are correct.
The benchmark currently contains twelve sets, ranging from 53 to 1,608 pieces. Further sets are being prepared.
