Teaching a robot a new task by teleoperating it hundreds of times works, but it is slow and expensive, and it only ever shows the robot the environments its operators could reach. People, meanwhile, perform the same tasks millions of times a day in kitchens, workshops, warehouses and building sites. Recording that activity — carefully — is one of the most practical ways to give robot learning systems the breadth they lack.
This article looks at how human activity data is actually used in robotics, which properties make it transferable to a robot, and where its limits are.
Three ways robots learn from people
1. Pretraining perception
The most widespread use is indirect. Visual encoders pretrained on large amounts of human activity video learn features that emphasise hands, objects and interactions. When those encoders are fine-tuned on a smaller set of robot data, policies tend to learn faster and generalise better than encoders trained on generic images, because the features already encode what matters for manipulation.
2. Learning task structure
Human episodes segmented into steps teach a model what a task is made of: which sub-actions occur, in what order, and what the scene looks like when each one is complete. That knowledge powers task planners, progress estimators and reward models — components that tell a robot whether it is on track, not just how to move.
3. Direct imitation through retargeting
The most ambitious use maps human motion directly onto a robot. Hand trajectories and grasp poses extracted from video are retargeted to a gripper or dexterous hand, then refined with a smaller amount of on-robot data. This route depends heavily on data quality: if hand pose cannot be recovered reliably from the footage, nothing downstream can fix it.
What makes human data transferable
Not every recording of people working is useful to a robot. The properties that decide transferability are fairly consistent across projects.
- Viewpoint. First-person footage shares the geometry of a robot’s head or wrist camera far more closely than footage from a fixed camera across the room. See Egocentric Video vs Third-Person Video for the trade-offs.
- Visible contact. Hands and the manipulated object need to be in frame and in focus at the moment of grasp and release.
- Complete episodes. Clips that start mid-reach or end before the object is placed teach fragments, not tasks.
- Natural pace. Collectors who rush or perform for the camera produce motion a robot should not copy.
- Documented rigs. Lens, field of view and mounting position need to be known so footage can be undistorted and compared.
Real activities beat staged ones
It is tempting to collect demonstrations in a lab, with tidy benches and a fixed set of objects. Those datasets are easy to annotate and almost useless for generalisation. Real environments contain clutter, wear, variable lighting, other people and objects nobody planned for. A model that has seen a sewing station covered in fabric offcuts, or a storage floor with crates stacked at odd angles, is better prepared for deployment than one that has only seen a clean demonstration table.
Real work also contains the behaviour that matters most for robustness: regrasping after a slip, adjusting an object that landed badly, pausing to look for a tool. These recoveries are rare in scripted data and common in genuine activity.
The embodiment gap
Human hands are not grippers. A person can pinch, roll and slide an object between fingers in ways a two-finger gripper cannot. Human data therefore rarely transfers one-to-one. What transfers well is the higher-level content: where to look, which object to approach, the order of steps, approximate approach direction and contact location. What transfers poorly is fine finger motion — unless the target robot has a dexterous hand.
The practical consequence is that human activity data is most valuable when it is combined with a modest amount of robot data on the target embodiment, rather than used as a replacement for it.
Designing a human activity dataset for robotics
- Start from the deployment task and environment, not from whatever footage is easiest to collect.
- Define an action vocabulary with written boundary rules before annotation begins.
- Choose capture rigs that match the robot’s sensing as closely as practical — field of view, mounting height, depth if the robot uses depth.
- Plan environment coverage deliberately: several sites per task, not one site recorded at length.
- Agree acceptance criteria — what makes an episode usable — and check every delivery against them.
Our guide on how to scope a first dataset walks through the scoping conversation in more detail.
Conclusion
Real-world human activity is the largest untapped source of manipulation experience available to robotics teams. Used well — first-person, complete, natural and documented — it shortens the path from a policy that works in the lab to one that works on site. Used carelessly, it adds volume without adding capability.
Mvaak collects first-person recordings of real tasks across homes, workshops and industrial sites, segmented and checked for robotics use. Learn more on our Physical AI data page or read What Is Egocentric Data and Why Does Physical AI Need It?.
Written by Mvaak — AI Data Services.
