The first part of our Food World Model (FWM) blog post series explained how our FWM works. Given a food ingredient pan and a candidate pick, our FWM predicts the correct pick weight and the surface left behind in the pan. Part 2 of this series will explore where the data for our FWM originates from. This matters because collecting real-world training data is one of the hardest parts of building our FWM, and because that data helps us address two fundamental challenges in the field of world models: ensuring predictions accurately reflect physical reality (i.e., physical grounding) and enabling models to generalize to conditions they haven’t encountered before.
The problem of physical grounding
Much of the current debate in the field of world models is about architecture. Latent state-space models, token transformers, diffusion, joint embeddings, video foundation models... each camp has a view on what a model should predict and how. We find this debate interesting but mostly beside the point for robotics.
The more difficult question is what a prediction is checked against. A video model can generate a future that looks plausible but breaks the laws of physics (e.g., conservation of mass, object permanence, or causality). Evaluations in our recent survey show that models do this precisely when conditions move outside of what they were trained on. For a robot acting in real-world environments, breaking the laws of physics is not an option, and food is especially challenging to handle without causing this problem. Cooked rice, shredded chicken, and curry sauce are granular, cohesive, deformable, and temperature-dependent. No physics engine can model them accurately, and no amount of video training data can teach a model about their physical properties (e.g., what a scoop of diced onions weighs and how a particular curry sauce spills over the edge of a scooping utensil).
As a result, we need to evaluate a world model for food manipulation on something it can’t fake. We picked a weight scale to evaluate our FWM’s performance. Every prediction our model makes is a prediction of a measured weight outcome. Our FWM predicts an ingredient pan’s topology will look like after a candidate pick, and it only improves when the weights it predicts move closer to the numbers our production robot hardware reads. That’s what physical grounding means in practice, and it’s why a small model with a weight scale underneath beats a large one with a camera alone.
A robot fleet as a measuring instrument
To physically ground our FWM with measured outcomes, we need an extremely large number of measurements across food ingredients, portion sizes, placement requirements, factory settings, shift runs (as even the same ingredient changes day by day), customers, countries, conveyors, and hardware versions. Only one place can provide this data: our fleet of production robots.
Each pick-and-place action a robot makes is a small experiment. The depth camera looks at the ingredient pan, the robot records its pick action, the weight scale records the resulting pick weight, and finally, the depth camera records what is left behind in the pan. This experiment runs every few seconds on every robot throughout every shift, 150 million times from our first customer deployment until today. We used this dataset to train our FWM, and it grows as our business grows.
Turning a fleet into a flywheel
Our fleet of robots produces the training data needed to train our FWM, but good training data alone is not enough to build a good world model. Most robotics companies turn logs into a model once, by hand, and then the model stays static for a while. To prevent models from becoming stale, we don’t need to speed up data collection, which happens at the pace of production. We need to speed up AI model training: new logs arrive, we rebuild the data corpus, retrain our model, score it against data it has never seen, and eventually deploy it on our robots. If each additional training run costs what the last one did, you own a dataset. If each run costs less than the last, you own a flywheel.
When we worked on reducing the cost of each training run, we used four principles to help us accomplish our goal:
- We don’t compute anything twice. Every step of a training run is a pure function of its inputs, and we store its result under a fingerprint of those inputs. When new data arrives, we retrain only the affected parts of our model. A retrain is just that: a “re” train. What doesn’t need to change remains untouched.
- Every result is reproducible. Any number in any report traces back to the exact code and the exact data that produced it, and we can regenerate it with one command. We realized early on that a wrong cached result is the worst possible outcome our system could produce. Why? Because it’s silently wrong science. Most of our infrastructure design is based on preventing wrong cached results.
- Our pipeline needs to be honest about its data. A session that yielded fewer usable pick-and-place actions than expected is a fact that needs to be recorded, not a failure that needs retrying. Whether a data corpus is good enough to train our AI model is a question that concerns the whole corpus. One robot missing a pick is ordinary; 30 robots missing a pick is a fault, and only something that sees all 30 of those instances can tell the difference.
- A training run can happen without us. Both data extraction and model training can run on spot capacity that scales to zero when idle, survive being preempted, and finish whether or not anyone’s laptop is open. Training runs should start as soon as the data is ready, instead of waiting for someone to kick it off.
One training run on all our fleet data
We recently completed a training run on every log our fleet has ever uploaded, in one pass, from one version of the code. We didn’t know how much usable pick data our fleet held because whether a pick is usable depends on every sensor stream surviving, and the only way to find out was to try. The data corpus that resulted from this experiment is more than 10 times larger than the one our food world model from part 1 of this blog series was trained on, and it’s reproducible with one command. Every additional training run is based on this one, and its cost is proportional to the new information it provides.
Every new training run we complete encompasses every food ingredient our robots have ever handled across every production facility. The question we care about is one that part 1 of our blog series left open: how well our model handles an ingredient it has never seen before. In part 1, we explore this question for nine new ingredients. Our data corpus now lets us answer this question more comprehensively.
What’s next in the field of world models
The field is arguing about which architecture understands physics. We think it’s whatever architecture stands on a scale, is fed by a fleet, and gets retrained every time that fleet experiences something new. The architecture itself will keep changing. The training flywheel is what we’re building to last.
