Why robot data is scarce, unlike language data
The secret behind the LLM’s remarkable growth has ultimately been the “power of data.” In accordance with the scaling laws demonstrated by OpenAI, it has grown by absorbing the trillions of text and images scattered across the internet. Recently released VLA (Vision-Language-Action) models, such as OpenVLA, have also begun to demonstrate the potential of physical AI by learning from approximately 970,000 robot episodes.
However, compared to language or vision models, this amount of data is woefully insufficient—scarce in the extreme. No matter how much data you scrape from the entire internet, information such as “joint angles” or “motor torque”—which is actually needed to make a robot move—isn’t found anywhere on the web.
The most intuitive fix: let a human show it
The most intuitive way to give a robot training data is for a person to demonstrate the actions directly. This is called teleoperation. It involves a person wearing VR goggles, using a joystick, or wearing a motion suit to directly control the robot on a one-to-one basis and accumulate motion data.
However, collecting one hour’s worth of data requires tying up a person’s time for an hour, which entails enormous manpower and costs. Furthermore, since each laboratory uses different sensors, grippers, and control methods, data compatibility is difficult, and tracking and evaluating precise 6D poses (position and rotation) remains a challenging problem.
Then why not generate the data in simulation?
Realizing the limitations of putting people on every task one by one, the research team turned their attention to a virtual world—the simulator. They hoped that by running robots indefinitely in a virtual space to generate data on their own, they could achieve unlimited scalability. However, the task of translating the physical world into a virtual one also presented significant challenges.
Even for a simple “pick-and-place” motion, you must manually set variables such as an object’s mass, coefficient of friction, and gravity to build a virtual environment. Even the slightest contact and slippage that occur when picking up an object must be calculated in detail, down to the millisecond (ms).
The problem is that even when we create a virtual world like this, a huge discrepancy—the Sim-to-Real Gap—still arises between the simulation and reality. Due to errors in the physics engine, problems often arise when the simulation is applied to the real world; in particular, it is nearly impossible for current simulators to perfectly replicate the subtle movements of non-rigid objects, such as fabrics or liquids.
The data engine that changes the game
Ultimately, the research team began redesigning the data collection pipeline from scratch. The first breakthrough is “self-sufficient simulation.” Moving away from the traditional method of manually creating virtual environments, technologies such as RoboGen and AutoRT have emerged that use generative AI and large language models (LLMs) to automatically generate an infinite variety of elements—from 3D environments to objects, robot tasks, and control policies. These technologies have established a system that autonomously generates training data within a virtual space without human intervention.
The second breakthrough is “technology that translates human actions into robot data.” To overcome the inefficiency of humans manually controlling robots one by one, researchers are developing UMI (Universal Manipulation Interface) technology, which precisely converts the movements of a portable gripper into robot data. This effectively and intuitively bridges the “embodiment gap” between humans and robots.
Not more data, but smarter data
If the era of language models was a battle over “how well and how much data could be scraped from the internet,” then the era of robotic AI could very well become a battle over “how intelligently useful control signals can be generated beyond physical limitations.” On the day that AI capable of independently building virtual environments and data engines that translate human behavior into robot language are perfected, won’t a true “ChatGPT moment” finally arrive in the robotics ecosystem as well?
Tags
- #artificial-intelligence
- #ai-papers
- #physical-ai
- #robot
- #robot-data
- #vla
- #sim-to-real










