Reinforcement Learning and Sim-to-Real Explained Using MicroDuck
Reinforcement Learning and Sim-to-Real Explained Using MicroDuck
“Have you ever seen a robot duck waddling along?”
That little robot duck waddling along on social media and Reels? That’s Microduck. Despite its cute appearance, this robot learned to walk through countless trials and errors in a virtual world.
But walking well on a computer is a completely different matter from not falling over in real life.
Walking on two legs is harder than you might think. When you lift one foot, your center of gravity shifts, and an impact occurs the moment your foot touches the ground. If a person were to design this from scratch, they would have to calculate everything—from joint movements to the body’s tilt—step by step.
That’s why recent research in robotics is asking a different question: Instead of telling a robot the “correct” way to walk, can we let it figure out how to walk on its own? Microduck, an open-source bipedal robot developed by Pollen Robotics and Hugging Face, is a good example.
How Does Microduck Move?
Microduck is a small bipedal robot that stands about 25 cm tall and weighs about 800 g. To prevent itself from falling over despite its small size, it must quickly correct its posture before losing balance. Therefore, it repeats the process of “assessing its body’s condition and determining the next joint movement” 50 times per second—or every 0.02 seconds.
The neural network responsible for making this decision is called the “policy.” The policy is trained in a simulation using a reinforcement learning algorithm called PPO (Proximal Policy Optimization), then saved in a standard file format called ONNX and transferred to the actual robot.
The policy takes 61 numbers as input at any given moment and outputs 14 numbers. Of these inputs, 48 represent the robot’s current state—such as joint angles or body tilt—while 13 are commands, such as “move forward at this speed.” The 14 outputs are target values that determine where each joint should move.
So how do we teach walking using this approach? Instead of giving a specific answer like “Lift your left foot by X degrees,” we provide a reward when the child successfully maintains the target speed or avoids falling.
If I give him enough rewards, will he learn to walk?
“We give points instead of correct answers”
The basic structure of reinforcement learning is simple. When a robot takes an action, the environment returns a score, and the robot adjusts its policy to achieve a higher score.
For example, the closer you walk to the target speed, the higher your score; if you fall or your posture becomes unstable, points are deducted. (This is just an example to illustrate the principle—it is not Microduck’s actual reward function.)
But here’s where things get interesting. The behavior that earns the highest reward isn’t necessarily the gait we want. Instead of walking naturally, robots sometimes find shortcuts that let them score points more easily. This is called reward hacking.
If we only award points for walking fast, people might lose their balance; if we emphasize stability alone, they might not even want to move at all. After all, what we’re looking for isn’t a high score, but a gait that’s useful in real life.
Why We Train 4,096 Virtual Ducks Instead of One Real One
Since we can’t teach a robot by making it fall over millions of times in real life, the training takes place in a simulation. The 4,096 “ducks” here are all virtual—they exist only inside the computer—so there’s no need to worry about them breaking, no matter how many times they fall.
The Microduck tutorial makes these 4,096 virtual ducks walk simultaneously on a GPU and uses their collective experiences of falling and getting back up to refine a single walking policy. It’s as if thousands of ducks are collectively gathering the experience of a single duck falling thousands of times. Similarly, there is research showing that the four-legged robot ANYmal learned to walk on flat ground in less than four minutes.
What Does PPO Change?
PPO is an algorithm that refines a policy based on accumulated experience. The key is to ensure that the policy does not change too abruptly with each update.
Since the experience we have now was gained by navigating under the current policy, that experience will no longer be valid if the policy changes significantly. If we try to make drastic changes, the policy could collapse—just like a robot that was walking steadily but ends up falling over. That’s why PPO sets an upper limit on the magnitude of changes and makes improvements incrementally.
Of course, just using PPO well doesn’t guarantee that a robot will walk well. Perception, action, reward, and the realism of the simulator are all important.
Why do I walk just fine in the virtual world, but trip in real life?
“The Gap Between Virtual and Reality: Sim-to-Real”
Thousands of virtual ducks have started walking well. Does that mean we’re done once we transfer them to real robots? Not quite—the most challenging step, “Sim-to-Real,” still lies ahead.
In a simulation, a motor moves precisely to the desired angle, but in reality, motors are subject to friction and force limitations. Commands may arrive late, and battery voltage fluctuates constantly. Due to the small gap between meshing gears—known as gear backlash—the joints may follow slightly behind. When balancing on one leg, even these small differences can cause the robot to fall over.
So, the tutorial first uses BAM (Better Actuator Models) to make the virtual motor more closely resemble a real motor. Based on data measured from actual servo motors, it represents friction more accurately and accounts for backlash as well.
We also slightly vary the weight, floor friction, battery voltage, and command latency for each virtual duck, and even add terrain and external impacts. This is called Domain Randomization. The goal is to train the ducks to withstand various conditions, rather than becoming accustomed to just one virtual world.
“Now it’s your turn.”
You might think, “Oh, so all it takes is reinforcement learning for a robot to walk,” but in reality, there are many more steps involved. Reward design, rapid data collection, realistic motor models, learning in various environments, and precise deployment all need to come together seamlessly.
Of course, there are still challenges ahead. The current version of Microduck has learned to walk according to specific training objectives, so whether it can walk well on unfamiliar terrain or in unfamiliar situations is another matter entirely.
In the past, conducting sophisticated gait research required specialized hardware and a dedicated simulation environment. Now, with small open-source robots and publicly available training code, anyone can follow the “process of learning in simulation and validating in the real world” firsthand.
Tags
- #artificial-intelligence
- #ai-papers
- #huggingface
- #microduck
- #sim-to-real













