GitHub - composo-ai/DishwasherBench: A dishwasher-loading game on a physical simulation of one real machine, built to benchmark language models and playable in a browser. The game inside is called Spotless.

DishwasherBench is a dishwasher-loading game on a physical simulation of one real machine, built to benchmark language models and playable by people in a browser. The game inside is called Spotless, and the packages are named @spotless/*. A player loads a 3D model of a Bosch SMV4HAX40G with rigid-body physics. A wash on its Eco 50 programme is then simulated (jets, droplets, run-off, soil and detergent chemistry), and the player gets points for every item that comes out clean. On every level after the first, not everything fits, so the player has to choose what to load and how. This is an early release, version 0.1.0. The simulation is calibrated on one test wash of the real machine. A score shows whether a player packed well and put cups, glasses and pots in mouth down. It does not yet show whether plates and cutlery were stacked so that they wash clean. The section What a score can be trusted for says so in full. What is solid today: loading is deterministic rigid-body physics, every episode is a record that anyone can verify by replaying it and by washing its load again, and the same game is played by people in a browser and by models through MCP. What is rough is how the wash scores plates and cutlery. Built and maintained by Composo. Many documents under docs/ are working records, kept as they were written; docs/README.md explains how to read them. Loading the first level in the browser. A dinner plate is in hand over the lower basket, and "Suggest spots" has numbered the free seats, best first. The panel on the right holds the exact pose a program would send. Contents What works today What a score can be trusted for Requirements Quick start Play in the browser Run a scripted player Run a model Results so far How it works Limitations and roadmap Where things are Contributing Licence, notices and disclaimer How to cite What works today The machine. One Bosch SMV4HAX40G as data and geometry, from its owner's photographs and tape readings (packages/machine): both baskets with their tine rows, shelves and rests, the spray arms and the Eco 50 programme. Loading. A catalogue of items and soils, deterministic rigid-body loading (Jolt), and a session layer with the actions and queries of the game (place, remove, set_rack, list_slots, check_load and the rest). The wash. A simulation at two tiers. The official tier gives the scored result; the fast tier is a coarser preview. A cold official wash of the largest level takes 4.9 minutes on the reference machine, a MacBook Pro (Apple M3 Pro, 18 GB), best of three, and 5 to 8.5 minutes with another job running beside it, against a budget of 30. A course of seven levels, numbered in order of difficulty (docs/benchmark/levels.md), each with saved reference loads of three scripted players (careful, careless, random). Six pass the benchmark's gates on the reference loads; the tutorial does not (its careless load loses too little). Three ways to play. The page (apps/web), an MCP server for agents (packages/mcp), and a harness that runs scripted players and models and writes verified episode records (packages/bench). Checks. 48 loading phenomena from the literature as executable pairs of washes (docs/research/standards.md section 9), and a calibration tool that scores the simulation against a record of a real wash, fits parameters and ranks candidate test washes (the stored ranking is of an earlier parameter set and has not been run again). Two models have played, through an OpenAI-compatible endpoint: six episodes on three levels at the official tier (Results so far). No person has played for a baseline, and the path for Anthropic's API has not been run against the live API. What a score can be trusted for The full account, with every number and where it comes from, is docs/validation/README.md. In short, the simulation has been checked against four things: One test wash on the real machine. Four egg-yolk plates on Eco 50. A prediction written before the result got the order of the plates and the form of the residue right, and was ten to thirty times too optimistic on the amount left on the two oven-dried plates. The calibration that followed moved one number: the share of the long soak during which the pump runs, from 0.5 to 0.8 (anywhere from 0.59 to 1). The record that the prediction came first is in the private development history, which is not published, so that order cannot be checked from this repository. Published class means of a sister machine (Bosch SMV6ECX51E, eco programme, reference detergent): cutlery within 0.1 of a score point, crockery 0.2 low, glass 0.1 to 0.5 low. This is a fit, not a prediction: the soil and chemistry defaults were set so that these numbers come out. 48 loading phenomena with evidence in the literature: 17 pass, 21 fail in size or sign, 1 is a faulty scenario and 9 need mechanisms that are not built. Its own book-keeping. Water is conserved exactly, and the same input gives the same result on one machine. The fast tier agrees with the official tier on 89 % of the verdicts of a standard 13-place load and on 97 % on the levels. A score today shows whether a player packed well and put cups, glasses and pots in mouth down. It does not yet show whether plates and cutlery were stacked so that they wash clean. A verdict can be trusted for A verdict cannot be trusted for A glass, mug or pot left mouth up or on its side: it comes out not clean Plates fallen over or leaning on each other: they come out clean, which is most probably wrong A blocked detergent tray, for protein soils (egg, yoghurt, mince, milk) A spray arm held still by an item: the plates above it keep their verdicts A bowl or a spoon that truly lies in another Cutlery crowded into one cell, and spoons as the loading physics usually nests them: clean Egg yolk and other protein soils, on an item that is far from the threshold of clean Items near the threshold; tea, coffee and wine; milk dried on glass; fat, which comes clean even with no detergent About three quarters of the gap between a careful and a careless reference load is hollow pieces put in the wrong way up. The soils in those pieces (smoothie, tomato soup, ketchup, guacamole) rest on values from the literature that no test wash has checked. Not modelled at all: drying, spots and film on glass, items moved by the jets, and soil settling on clean pieces. Only Eco 50 is calibrated and scored. Requirements DishwasherBench has been run on one machine: a MacBook Pro (Apple M3 Pro, 18 GB) under macOS 27. Every figure in this repository was measured on that machine. Linux and Windows are untested. Nothing is known to rule them out, and a report of what happens there is welcome (docs/ROADMAP.md). Node 20.19 or newer in the 20 line (.nvmrc pins 20.19.0). An older Node is refused at install. Node 22 has not been tried. pnpm 9 (package.json names 9.4.0; corepack enable picks it up). A GPU that Dawn can open. The wash traces water on the GPU through WebGPU from Node (the webgpu package, version 0.6.2, which carries prebuilt Dawn binaries). On the reference machine that is Metal. This version did not load on macOS 14.7 on the same hardware. Google Chrome or Chromium, for the pictures an agent is sent and for the scripts that drive the page. The harness looks for it in the usual install locations; set SPOTLESS_CHROME to the path of the program to choose another. Playing by hand needs only a desktop browser, and scripted players need none (--no-render). About 680 MB for the clone (a download of about 290 MB; the files themselves are about 385 MB, mostly photographs, screenshots and episode records) and about 420 MB for node_modules. 1 to 2 GB of memory for an official wash of a level (2.4 GB was seen on the standard 13-place load), and free ports 8787 (game server) and 8788 (MCP server). Results are repeatable bit for bit on one machine. Across machines, operating-system builds and versions of the webgpu package they may differ in the last bits, which is why every result names the stack it was computed on (docs/research/spike-gpu-compute-macos27-recheck.md). Quick start git clone https://github.com/composo-ai/DishwasherBench.git cd DishwasherBench corepack enable # once: makes pnpm 9.4.0 available pnpm install --frozen-lockfile # about 15 seconds the first time There is no build step for the packages: they are consumed as TypeScript source and run with tsx. Only the page is built, by the play command below. The shortest thing that shows a result is to wash a saved load from the command line: pnpm --filter @spotless/wash-runner wash-load tutorial-careful --tier fast # under a minute pnpm --filter @spotless/wash-runner wash-load tutorial-careful --tier official # about 5 minutes the first time It prints the points and, for each item, the verdict, the score and what is left on it. The saved loads are in packages/loader/loads/ (<level>-<player>.json). The first wash of a load works out where the water goes and keeps that in .spotless/exposure-cache, so a second wash of the same load takes seconds (pass --no-cache to switch this off). To check that everything runs on your machine: pnpm typecheck # the root and every package; a few seconds pnpm test # every package; 8 to 17 minutes on the reference machine How long pnpm test takes depends on what else the machine is doing. One test is skipped by design. The tests run real washes on the GPU. A wash refuses to run when the parameter set is stale. The set the game washes with (anchored-v7.owner-capsule, in packages/calibration/anchors/) records the state of the simulation source and of the machine data it was made for. After a change to either, the wash stops and lists what differs. CONTRIBUTING.md explains the rule and how to make the set current again. Play in the browser pnpm --filter @spotless/web play This builds the page and starts the game server. Open http://127.0.0.1:8787 (add --port <n>