How to Train Your Robot

The enormous advance in AI capabilities over the last several years has in large part been driven by the availability of vast amounts of training data. Gwern’s influential 2020 essay “The Scaling Hypothesis,” written after the release of GPT-3, argues that recent advances in AI have been the result of scale: bigger models, bigger datasets, and more compute spent training them. Gwern writes:The scaling hypothesis regards the blessings of scale as the secret of AGI: intelligence is ‘just’ simple neural units & learning algorithms applied to diverse experiences at a (currently) unreachable scale. As increasing computational resources permit running such algorithms at the necessary scale, the neural networks will get ever more intelligent.Likewise, pseudonymous AI researcher Nostalgebraist wrote in 2024 that “the entire current landscape dates back to the realization that ML models improve steadily and predictably as you train them on more data. And that everything since then consists of training bigger models on larger datasets, enabled by years of rapid improvement in hardware.” Epoch AI found that since the early 2010s, the size of training datasets for AI language models has doubled about every six months. The biggest AI models today are trained on tens of trillions of tokens.As more and more effort gets devoted to applying modern AI to robotics, an important question is “where will the training data come from?” Language models could be trained on huge existing corpuses of text. GPT-3, for instance, was trained on about 300 billion tokens’ worth of data scraped from the internet. In 2024 Epoch AI estimated the total existing stock of quality, accessible text data is around 300 trillion tokens. But there’s no equivalent stock of good, public robot training data. Figuring out where the data for robotics will come from and how much of it we’ll need has become a key problem for the field.Training an AI model, whether it’s an LLM or a model used for controlling robots, fundamentally requires lots and lots of examples of something being done correctly (or enough knowledge of what a correct action looks like that you can score an attempt). You feed a model some set of inputs, you compare the model’s output to the actual, correct output, and you use the difference between the two to modify the parameters of the model to reduce this difference via some flavor of gradient descent. (We went into this process in some depth earlier this month.)With text, a training example can be just an existing chunk of text; you take the chunk, feed the model part of it, and have the model predict what comes next in the sequence. One chunk of text can thus generate many training examples by feeding the model different portions of the text.Via Opus 5.5To train a robot, an example is typically some sort of recording of a task being accomplished or attempted. There are a variety of different methods used to create these recordings and turn them into training data.Historically, one of the most common methods for creating these recordings is called “teleoperation.” With teleoperation, a person manipulates some controls that are directly connected to a robot, and uses them to move a robot through some particular task (folding a shirt, picking up and moving objects, and so on). The image below, for instance, shows a person teleoperating a pair of robot arms, using them to pour water from a pitcher into a glass.As the person performs the task, data of the robot’s state is recorded, along with images from cameras that are recording the task (often mounted to the robot itself), which are combined to create a training example. An example might consist of feeding a model some data on the robot’s state and what the cameras can see at some particular point in the sequence. The model will try to predict what the next robot actions are (in the form of something like joint angles), and then compare that to what the actual actions were. This sort of training strategy, using examples generated by an expert (a human) and then training a model to mimic them, is known as behavioral cloning, and it’s part of the broader category of “imitation learning.”Via Opus 5.5.The major benefit of this method is that the data it generates very closely matches the robot’s actual movement capabilities, because it’s produced from the actual robot you want to train, and it remains widely used today. You sometimes see videos or pictures from huge data collection factories, filled with people doing nothing but teleoperating robots to complete various tasks, and you sometimes see teleoperation accidents when the person forgets that they’re controlling the robot and suddenly moves in a way that the robot can’t handle.But teleoperation has drawbacks. The biggest one is probably expense: to get the data, you need to have an actual robot, along with a person to sit there and manipulate it, which makes it hard to scale this method to produce very large amounts of data. Another drawback is that it’s somewhat clunky for a person to control a robot this way; they often can’t get any sort of tactile feedback from the object being manipulated to guide their motions, so this won’t necessarily produce really good data on dexterous manipulation. The clunkiness can also make it hard to collect good teleoperation data, and as I understand it, only a fraction of total teleoperation data collected ends up being useful for training.It’s also possible to generate training data using an existing robot by having it move around on its own, recording its actions, and using some sort of system to categorize these behavioral sequences as successful or not. This sort of strategy, where the robot moves autonomously and you score its actions in some way, is known as “reinforcement learning.” Google was doing something like this around 2016 with an “arm farm”: a series of robot arms moving autonomously, with success scored by whether an arm picked up or grasped an object or not.This sort of arm farm doesn’t seem to be widely used today, but using real-world deployed robots will probably eventually be the largest source of robot training data. You often hear robotics folks talk about the need for a “data flywheel,” where you have robots deployed in the real world, you use the data from them to train even better models, resulting in more robot deployments, resulting in better models, and so on.Robotic AI company Physical Intelligence, for instance, is experimenting with this sort of “deploy a robot to do real work, and use it to gather data to improve the model” proto-flywheel at a chocolate factory near its offices. But while this might work eventually, right now there’s a very small number of AI-controlled robots doing anything resembling real work. Real-world data also tends to be extremely repetitive — robots doing the same task over and over again in very similar conditions — which makes it much less useful for training. So for now most data needs to come from other sources.Another very common strategy for generating robotic training data is simulation: instead of running a physical robot in the real world, run a digital one in a simulated world instead. This is also commonly done with reinforcement learning, in which you have the robot try to navigate some simulated space, after which you score it on how well it does. Another flavor of this is the teacher/student paradigm where you have a “teacher” model that has full access to all the data within the simulated environment, and a “student” model that can only see what the real robot would see (images from its cameras and its own internal state), and you try to get the student to mimic the teacher.The benefits of simulation are that it’s much cheaper and faster to simulate a robot than it is to deploy one in the real world, and that you can get very large amounts of data by simulating the behavior of thousands of robots. Figure AI taught its humanoid robot to walk via reinforcement learning on thousands of simulated robots. OpenAI taught a robotic hand to solve a Rubik’s Cube via the equivalent of 13,000 years of simulated experience. By 2019, Waymo had driven 10 million miles in the real world, but 10 billion miles in simulation.Via Figure.But simulation also has drawbacks. The biggest one is that the quality of the data it produces is a function of the accuracy of the simulation. For some things, such as locomotion (having a robot move through some physical space), simulation is fairly accurate, because the behavior is mostly well-characterized enough that we can represent it digitally. So you can train a quadruped robot to navigate some terrain, or a bipedal robot to dance, or a drone to fly along some path in simulation, and the simulated environment will be a reasonably good approximation of the real world. But for things such as object manipulation (particularly for soft or deformable objects), it’s much harder to build a simulation that accurately captures real-world behavior, and using it as a source of training data is much more difficult. This is generally referred to as the “sim-to-real gap.”Because of simulation inaccuracies, one popular simulation technique is called “domain randomization.” Since you don’t know what various parameters like friction will be in the real world, you decide to vary the parameters randomly when running the simulation, in the hopes of creating a training envelope that encompasses the parameters that will be encountered in the real world. When OpenAI trained its robotic hand to solve a Rubik’s Cube, for instance, it continuously expanded the range of parameters being tested, forcing the model to learn to solve the cube in more and more difficult environments, in the hopes of giving it capabilities that would generalize to the real world. The robotics team there used this strategy specifically because accurate physical modeling was too difficult:The biggest challenge we faced was to create environments in