Multimodal AI
Why first-person video is a different kind of input
Egocentric footage preserves attention, approach and sequence — spatial context that third-person capture quietly discards.
3 min read
Solutions · Robotics & Physical AI
From real-world action to AI data.
How first-person demonstrations become structured, quality-checked training data for systems that act in the physical world.
How it works
From Real-World Action to AI Data.
Select a stage to see what happens to a recording on its way to becoming training data.
Stage 01 of 05
Human Demonstration
A trained, consented person performs the task in a real environment, following a written brief.
Outputs
- Capture brief
- Task catalogue
- Consent record
- brief
- pick_up_cup · household kitchen
- include
- one mis-grasp and recovery
- consent
- worker-level · signed
Stage 02 of 05
Capture
A calibrated first-person rig records video, motion and hand data on one clock.
Outputs
- Video
- IMU / motion
- Calibration
Synchronised at source
one clock · drift checked
RGB
IMU
Hand pose
Sub-action
reach
grasp
lift
flip
recover
place
consent_id ✓licence flags ✓provenance ✓LeRobot · RLDS · MCAP
Stage 03 of 05
Annotation
Each episode is labelled to a written guideline — actions, objects and hand pose.
Outputs
- Action segments
- Object labels
- Hand keypoints
reach4.2–4.9s
grasp4.9–5.3s
lift5.3–5.8s
Stage 04 of 05
Quality Assurance
Automated checks and human review catch problems before anything is delivered.
Outputs
- QA results
- Review notes
- Diversity report
Batch QA · five gates
- 1Field check pass
- 2Automated integritysync · drift · blur · exposure pass
- 3Human reviewgraded against the rubric pass
- 4Diversity scoringgap → next capture briefgap found
- 5Acceptance samplingblind re-grade of a random sample sampling
Stage 05 of 05
Training-Ready Dataset
Structured data in your format, with provenance attached to every episode.
Outputs
- LeRobot · RLDS · MCAP
- Episode manifests
- Guideline
- corpus_batch_03/
- episode_0142/
- video/— rgb · stereo
- imu/— clock-aligned
- annotations/— sub-actions · hand pose
- manifest.json— provenance · consent · licence
- diversity_report
- annotation_guideline
LeRobotRLDSMCAPraw + manifest
Explore our data
One recording, three views.
Switch between raw footage, annotations and structured task data to see what turns video into training data.
- Raw footage
- What the camera recorded — useful, but not yet learnable.
- Annotated
- Hands, the object and the current action are marked.
- Structured data
- The same moment as a machine-readable record.
Annotated: hand keypoints, the object and the current action are marked. · Illustrative example
Quality that matters
Problems are caught before delivery.
Quality is built into every stage — and documented, so you can see exactly what was checked.
Structured collection
Every hour traces to a written brief and acceptance rubric.
Annotation
Labelled to a written guideline, with ambiguity escalated.
Validation
Automated checks, human review and blind sampling.
Consent & provenance
Worker-level consent and a manifest with every episode.
Batch review · ep_0142
Illustrative example
- What is checked
- Every required segment, object and keypoint in the guideline must be present.
- Example result
- One segment missing an outcome marker.
- If it fails
- Returned to the annotator with the guideline reference, then re-reviewed.
Who we serve
Built for teams training AI on the real world.
Robotics & Physical AI teams
Human demonstrations of real manipulation tasks.
AI & ML companies
Training and evaluation data specified around model failures.
Research labs
Well-documented datasets with consent and provenance.
Data operations leads
Extra capacity without renegotiating quality.
Related reading
All articlesGenerative AI
Synthetic data is a complement, not a replacement
Generated data is genuinely useful for coverage and for privacy-constrained domains. It is not a substitute for the real-world anchor.
2 min read
Dataset Quality
What actually makes training data usable
Volume is the easiest property to measure and the least predictive of whether a dataset improves a model. Here is what tends to matter instead.
3 min read
Start a project
What Does Your AI Need to Learn?
Tell us about your data requirements, and let’s explore the right approach.