Skip to content
NewsletterOct 1, 2026

← Past issues

Why is there a shortage of robot data?

Sent Oct 1, 2026

How Robotics Models Overcome Data Droughts

When reading tech articles, you often come across the statement, “There is a severe shortage of data needed to develop robot AI.” But what exactly is this “robot data” everyone’s talking about, and why are global tech companies pouring astronomical amounts of money into it while touting the importance of data?

💡 This Week’s Newsletter Summary

  • Fundamental Limitations of Robot Data: Control information—such as joint angles and motor torque—is not available anywhere on the web, making large-scale scraping impossible.
  • Limitations of Existing Solutions: Teleoperation, in which a human operator demonstrates the task directly, lacks scalability, and simulators have not yet overcome the “sim-to-real gap.”
  • Next-Generation Data Engine: Researchers are redesigning data pipelines by using generative AI to automatically build virtual environments, convert human movements into robot data, and reconstruct 2D video as 3D (Real-to-Sim).

Why There Is a Lack of Data, Unlike with LLMs

The secret behind the LLM’s remarkable growth has ultimately been the “power of data.” In accordance with the scaling laws demonstrated by OpenAI, it has grown by absorbing the trillions of text and images scattered across the internet. Recently released VLA (Vision-Language-Action) models, such as OpenVLA, have also begun to demonstrate the potential of physical AI by learning from approximately 970,000 robot episodes.

Compared to text (15 trillion tokens) or images (6 billion), the robot data (2.4 million episodes) is woefully insufficient for building a general-purpose AI.

However, compared to language or vision models, this amount of data is woefully insufficient—it’s a mere drop in the bucket. No matter how much data you scrape from the entire internet, information such as “joint angles” or “motor torque”—which is actually needed to make a robot move—isn’t found anywhere on the web.

The Most Intuitive Solution: Human Instruction

The most intuitive way to give a robot training data is for a person to demonstrate the actions directly. This is called teleoperation. It involves a person wearing VR headsets, a joystick, or a motion suit to directly control the robot on a one-to-one basis and accumulate motion data.

**📌 Teleoperation: ** A remote control method that uses equipment such as VR headsets or joysticks to directly operate a robot on a one-to-one basis and collect the precise motion data needed for robot training.

An example of teleoperation, in which a person manually collects data remotely.

However, this approach has a critical bottleneck. Since collecting one hour’s worth of data requires a person’s time to be tied up for exactly one hour, it consumes a massive amount of labor and costs. Furthermore, since each laboratory uses different sensors, grippers, and control modes, it is difficult to ensure data compatibility, and even tracking and evaluating precise 6D poses (position and rotation) remains a challenging problem.

In that case, couldn’t we just generate the data using a simulation?

Realizing the limitations of deploying people one by one, the research team turned their attention to simulators—virtual worlds. They hoped that by running robots indefinitely in a virtual space to generate data on their own, they could achieve unlimited scalability. However, the task of translating the physical world into a virtual one also presented significant challenges.

Let’s consider a single motion that seems very simple: “picking up a cup and moving it to the side” (Pick & Place). To build a virtual environment, we must define countless environmental variables one by one—not only the robot’s shape but also the object’s mass, size, coefficient of friction, and even the gravitational settings. On top of that, from the moment the robot arm approaches and closes its gripper to the moment it lifts the object, the system must perform dense calculations at very short time intervals (milliseconds) to account for the subtle contact and slippage between the arm and the object.

The Franka Emika Panda robot running in the Mujoco simulator

The problem is that even after creating a virtual world through such massive computations, a huge discrepancy—the “Sim-to-Real Gap”—still arises between the virtual and real worlds. Due to errors in the virtual world’s physics engine and rendering, problems often arise when transferring the results to the real world; in particular, it is nearly impossible for current simulators to perfectly replicate the subtle movements of non-rigid objects—such as fabric or tissue paper—or liquids.

Would it be possible to scrape online videos?

So, wouldn’t it be possible to train the robot using the hundreds of millions of everyday videos that flood YouTube and the web? Unfortunately, videos alone aren’t enough to actually make a robot move. This is because videos are merely visual pixel data and contain absolutely no control information—such as joint angles or motor torque—that is crucial for controlling a robot’s motors. While text and images can be obtained in large quantities through web scraping, we run into a fundamental limitation: the action data needed to directly drive a robot simply does not exist anywhere on the web.

The Next-Generation Data Engine That Changes the Game

Ultimately, the research team began redesigning the data collection pipeline from scratch.

The first breakthrough is “self-sufficient simulation.” Moving beyond the traditional method of manually creating virtual environments, technologies such as RoboGen and AutoRT have emerged that utilize generative AI and large language models (LLMs) to automatically generate an infinite variety of elements—from 3D environments to objects, robot tasks, and control policies. This has established a system that autonomously generates training data within a virtual space without human intervention.

AutoRT’s automatic data generation mechanism: Environment exploration ➔ Task generation (LLM) ➔ Filtering ➔ Automatic collection ➔ Infinite loop

The second breakthrough is “technology that translates human actions into robot data.” To overcome the inefficiencies of humans directly controlling robots, UMI (Universal Manipulation Interface) technology is being widely used: when a person holds a portable gripper in their hand and moves an object, the system precisely converts this motion into robot data. This effectively and intuitively bridges the “embodiment gap” between humans and robots.

UMI Technology: Extracting Robot Data Directly from Just Handling a Portable Gripper

Going a step further, through Real-to-Sim (digital twin) technology—which restores ordinary 2D videos, of which there are countless on the web, into interactive 3D virtual spaces—we are transforming even previously discarded online video resources into training data for robots.

“Not ‘more,’ but ‘smarter’”

If the era of language models was a battle over “how much data can be scraped from the internet,” the era of robotic AI is a battle over “how intelligently useful control signals can be generated beyond physical limitations.” Won’t a true “ChatGPT moment” finally arrive in the robotics ecosystem on the day that AI capable of independently building virtual environments and data engines that translate human behavior into robot language are perfected?


Ultimately, there is only one key point.

Given our limited resources, which data should we wisely invest them in?

Leave feedback on this issue