Berkeley researchers build humanoid that taught itself to walk

UC researchers Ilija Radosavovic and Bike Zhang wondered if "reinforcement learning," a concept made popular by large language models (LLMs) last year, could also teach the robot how to adapt to changing needs. To test their theory, the duo started with one of the most basic functions humans can perform - walking.

Transformer model for learning

The researchers started in the simulation world, running billions of scenarios in Isaac Gym, a high-performance GPU-based physics simulation environment. The algorithm in the simulator rewarded actions that mimicked human-like walking while punishing the ones that didn't. Once the simulation perfected the task, it was transferred to a real-world humanoid bot that did not require further fine-tuning.

The learning system's core is a deep-learning model, now popular as a transformer. The model has been extensively used in LLMs to predict the next element in a series of data sequences. Since the researchers were interested in walking, they deployed a causal transformer that learns from observations and actions and predicts the consequences of actions instead.

This is how the humanoid adjusts its actions in a new landscape, even though it has never encountered it.

we have trained a humanoid transformer with large-scale reinforcement learning in simulation and deployed it to the real world zero-shot pic.twitter.com/WzOIMQXTaD

— Ilija Radosavovic (@ir413) December 10, 2023

How well does the humanoid walk?

Over the past year, the humanoid has strutted around the UC campus, navigating through its patches of grass and roads. Interestingly, the researchers noticed that the humanoid had picked up "emergent" traits that weren't intentional parts of the algorithm.

For instance, the humanoid swings its left arm while putting its right foot forward or takes smaller steps to balance itself while walking down the slopes, much like how humans do. While this might not sound revolutionary, it is still impressive, considering that it does not possess any sensors that help it perceive its environment.

This is likely the reason why the robot cannot navigate steps for now. But its extensive learning ability has made it resilient enough to maintain its balance, even when the researchers poked it with large sticks or threw exercise balls at the robot.

There are areas for improvement as well. The gait of the humanoid isn't as refined as that of a human and comes across as jerky. When the metal bot's foot hits harder material like concrete or asphalt, it resembles a military march. The worst, however, is the robot does not know it will run into an obstacle until it hits it.

These are things that the researchers plan to work on next. While it might be far away from being an ideal factory or home bot, it is still a good start for a self-learning humanoid.