Reinforcement Learning and Sim-to-Real with Microduck
Sent Nov 12, 2026
Microduck: How to Build a Walking Robot Using Reinforcement Learning
Have you ever seen a tiny robot duck waddling around on social media or Reels? That’s Microduck. Despite its cute appearance, this robot learned to walk through countless trials and errors in a virtual world. But walking well inside a computer and not falling over in the real world are two completely different things.
💡 This Week’s Newsletter Summary
- Walking via reinforcement learning: Microduck learns a policy that takes inputs such as joint states and movement commands and outputs joint targets. The publicly available policy interface accepts 61-dimensional observations and outputs 14-dimensional actions.
- Training Thousands of Instances Simultaneously: The original Microduck RL repository uses mjlab (MuJoCo Warp) and PPO. Meanwhile, the NVIDIA and Hugging Face tutorials run as many as 4,096 simulation environments simultaneously in Isaac Lab.
- The Key Is Translating It to the Real World: Just because you’ve learned to walk in a virtual world, does that mean you’ll walk well in the real world? There are still hurdles to overcome, ranging from motor friction to command latency, gear backlash, and fluctuations in battery voltage.
When you see a small robot walking on two legs, you can’t help but wonder: How does it maintain its balance with such a tiny body? When it lifts one leg, its center of gravity shifts, and the moment its foot touches the ground, an impact occurs. If it moves a joint even slightly out of place, it will fall right over.
What if a person were to design each step of a gait? They would have to calculate everything—from joint movements to foot contact, body tilt, and changes in speed. But recent advances in robot learning raise a different question: Instead of telling the robot what the “correct” gait is, could we let the robot figure out how to walk on its own?
There’s an example that illustrates this point. It’s Microduck, an open-source bipedal robot developed by Pollen Robotics and Hugging Face.

What Microduck Looks Like
How Does Microduck Move?
Microduck is a small bipedal robot that stands about 25 cm tall and weighs about 800 g—small enough to fit on a desk. To walk without falling over with such a small body, it must quickly correct its posture before losing its balance. That’s why Microduck repeats the process of “assessing its current state and determining the next joint movement” 50 times per second—or every 0.02 seconds. The neural network policy is responsible for making these decisions.
So where does this neural network learn how to walk? Instead of using a real robot, it learns in a simulation using a reinforcement learning algorithm called PPO (Proximal Policy Optimization). PPO is a learning method that gradually refines the neural network based on the results of trial and error; we’ll take a closer look at the details later. Once training is complete, the neural network is saved in ONNX format and transferred to the actual robot. ONNX is a standard file format that allows a neural network to run as-is on other devices or programs, regardless of the framework used to train it.
The “policy” we’re talking about here isn’t anything complicated. It’s a function that looks at the robot’s current state and decides how it should move next.
Microduck’s public policy interface uses 61-dimensional observations and 14-dimensional actions. Simply put, this means the policy takes 61 numbers as input every 0.02 seconds and outputs 14 numbers. Of the 61 inputs, 48 represent the robot’s current state—such as joint angles or body tilt—while the remaining 13 are commands specifying movement, head, and body posture, such as “move forward at this speed” or “position the head and body in this way.” The 14 outputs serve as target values that determine where each joint should move. Note that the total number of motors does not necessarily have to match the number of actions directly output by the policy.

The process by which the observed robot state is translated into joint movements through a policy
The process by which the observed robot state is translated into joint movements through a policy
So how do we teach walking using this kind of policy? Instead of giving a specific answer like, “Lift your left foot by X degrees and turn your right foot by Y degrees,” we provide a reward when the child successfully maintains the target speed or avoids falling.
If I give them enough rewards, will they learn to walk?
The basic structure of reinforcement learning is simple. When a robot takes an action, the environment assigns a score, and the robot adjusts its policy to earn a higher score. For example, consider a robot that walks at a target speed: the closer it walks to that target speed, the higher its score; the greater the deviation, the lower the score. Additionally, points are deducted if the robot falls or its posture becomes unstable. However, this is merely an example to illustrate the principle and does not represent Microduck’s actual reward function.
But here’s where things get interesting. The behavior that yields the highest reward isn’t necessarily the gait we want. Instead of the “natural gait” intended by humans, the robot may find a shortcut that maximizes its mathematically defined reward. This phenomenon is called “reward hacking.”
That’s why the process of determining how to score certain behaviors is trickier than you might think. If we only award points for walking fast, the robot might lose its balance; if we place too much emphasis on stability, it might refuse to move at all. After all, what we’re after isn’t a high score in and of itself, but a gait that’s useful in the real world.
📌Reward hacking: This is a phenomenon in which a policy exploits the gap between the designer’s intended purpose and the reward function. Even if a high reward is obtained, the behavior may not be useful.
Why We Train 4,096 Virtual Ducks Instead of One Real Duck
But what if we actually had to teach a robot by making it fall over millions of times? That would take a long time and be dangerous. Every time it fell, the motors or frame could get damaged, and someone would have to pick the robot back up each time. That’s why pre-training in a simulation is necessary. The 4,096 “ducks” I’m about to talk about are all virtual ducks inside a computer. They can fall over as much as they want without worrying about bumping into each other or getting damaged.
This is where a key idea comes into play: instead of training robots one at a time, we train thousands of them simultaneously. A study by Rudin et al., titled “Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning,” reported that by training thousands of virtual robots in parallel on GPUs, they were able to train the ANYmal robot’s flat-ground walking policy in less than 4 minutes and its rough-terrain walking policy in about 20 minutes.
However, this training time is based on the results from the ANYmal paper and does not represent Microduck’s training time. So, how long does Microduck take? Unfortunately, the official documentation does not specify an exact training time. However, according to the Microduck documentation from Isaac Lab, the default configuration for rough-terrain walking involves running 16,384 virtual environments on a single GPU for up to 50,000 training iterations. The actual time required varies significantly depending on GPU performance and settings.

An environment for running multiple virtual robots in parallel
Microduck also utilizes the same parallel training concept. In the training example, it can run 4,096 virtual environments simultaneously. A separate tutorial released by NVIDIA and Hugging Face in October 2026 demonstrates how to implement this parallel training using a combination of Isaac Lab 3.0, Newton, and RSL-RL. Each of these three components plays a distinct role.
What are everyone’s roles?
-
Isaac Lab: It is NVIDIA’s open-source robot learning framework. It serves as a “training ground” that defines the robot, the environment, and observations and rewards, and ties the entire learning process together.
-
Newton: It’s an open-source physics engine that runs on a GPU. It acts as a “set of physical laws” that simultaneously calculates the joints, collisions, and friction of thousands of virtual ducks.
-
RSL-RL: This is a reinforcement learning library developed by the RSL Lab at ETH Zurich. It acts as a “coach” that refines policies by running PPO based on experience gathered from simulations.
In other words, the structure is such that when Isaac Lab sets up a training environment, Newton quickly computes the physics within it, and RSL-RL uses those results to learn a policy.

Microduck’s Obstacle Environment Simulation
It’s okay if the simulator used differs from project to project. The key point is that we gather the experiences of thousands of robots each falling and getting back up to improve a single locomotion policy.
What Does PPO Change?
Now, let’s see how we can use that experience to refine the policy. The algorithm we’ll use here is PPO. PPO designs the objective function so that, while seeking a better policy based on the experience gained from the current policy, the policy doesn’t change too drastically in a single update.
Why should we avoid sudden changes? The experience data we currently have was collected while the robot was navigating under the current policy. If the policy changes drastically all at once, this data will no longer accurately represent the robot’s behavior under the new policy. If we rely on that same data and make drastic changes anyway, a robot that was walking steadily could suddenly start falling over repeatedly—and the policy could collapse. Once this happens, the robot accumulates only experiences of falling, making it difficult to recover. That’s why PPO chooses to improve the policy “little by little, but steadily.”
To achieve this, PPO sets an upper limit on the magnitude of changes. First, it assesses whether each action was better or worse than expected, and then adjusts the policy so that better actions are performed more often and worse actions are performed less often. However, if the probability of performing a certain action changes by more than a certain percentage compared to the previous policy, no additional reward is given even if the policy is adjusted further. This approach is called a “clipped objective.” Of course, this does not guarantee that the policy will never undergo significant changes.

Intuitive Concept Map of the PPO Training Phase
So, will a robot walk well just by using PPO effectively? Not exactly. It’s also important to consider what states to observe, what actions to generate, what rewards to provide, and even how realistic the physics simulator is.
Why do I walk just fine in the virtual world, but trip in real life?
Thousands of virtual ducks have started walking properly. Does that mean we’re done once we transfer the policy to real robots? Not quite—the most challenging step, Sim-to-Real, still lies ahead.
In a simulation, the motor moves precisely to the desired angle, but in reality, servo motors are subject to friction and torque limitations. Commands may arrive late, and battery voltage fluctuates constantly. Due to the slight gap between meshing gears—known as gear backlash—the joint may follow slightly behind when changing direction. When balancing on one leg, even this small delay can cause the robot to fall over.
That’s why the NVIDIA and Hugging Face tutorials first use **Better Actuator Models (BAM)** to make virtual motors more closely resemble real motors. BAM is a library that, based on data measured from actual servo motors, represents motor friction more accurately than the simulator’s default model. It also accounts for backlash.
In addition, the system trains the robots by slightly varying factors such as mass, center of gravity, ground friction, battery voltage, and command latency for each robot, while also changing the terrain and subjecting them to external impacts. This method of intentionally diversifying the training environment is called “Domain Randomization.” The goal is to train the robots to withstand various conditions rather than becoming accustomed to just one virtual world.

The Difference Between Simulation and Reality
📌 Sim-to-Real Gap: The difference between the physical, sensor, and latency characteristics represented by a simulator and those of the actual hardware. This is one of the main reasons why policy performance in a virtual environment does not translate directly to the real world.
How is the learned gait incorporated into an actual robot?
Once training is complete, does PPO continue to run inside the robot? No, it doesn’t. During training, PPO collects data from the environment and updates the neural network, but after deployment, the robot does not retrain at every control cycle; instead, it infers the policy it has already learned.
In Microduck, you can export the trained policy as an ONNX file. The key here is that the robot reads the state and executes actions in the same way it did during training. For example, if a neural network learned to interpret the body’s tilt in a specific sequence and with specific units, but the actual robot interprets that input differently, the joints might move in unexpected ways.

Because it allows users to experience the entire process firsthand—from simulation training to deploying policies onto actual robots—Microduck is more than just a cute, ready-made product. It’s also an open-source research platform where you can train and modify walking policies yourself. Developers can test and share their own policies, and the official tutorials provide guidance on sharing and evaluating policies through Hugging Face Hub and Microduck Arena.
What the Little Duck Showed Us and What Remains
When you see Microduck, you might think, “So, if you just use reinforcement learning, the robot will walk.” But in reality, there are many more steps involved. In fact, even a seemingly basic movement like walking requires a combination of factors—including reward design, rapid data collection, realistic motor models, learning across various environments, and a precise deployment interface—all of which must work together seamlessly.
Of course, there are still challenges ahead. The current version of Microduck has learned to walk according to a given learning objective within a defined observation and action interface. Whether it can walk well on terrain or in situations it hasn’t encountered during training is another matter entirely. The published tutorial also suggests starting with walking and then expanding to more diverse terrains and perception challenges.
Still, one change seems clear. In the past, conducting sophisticated gait research required specialized hardware and a dedicated simulation environment. Now, using small open-source robots and publicly available training code, we can directly observe the process of “learning in simulation and validating in the real world.”
Does a robot need a command to move its joints, or thousands of trials and errors, to take a single step? Microduck seems to demonstrate just how much engineering goes into the process between those two extremes.