A collection team can come back from a week in the field with terabytes of footage and still have nothing a model can train on. Raw video is a recording. A training-ready dataset is a structured, documented, checked artefact that a training pipeline can load without manual cleanup. Most of the cost and most of the risk sit between the two.
This article walks through the stages of a typical video data pipeline, with a focus on egocentric video for robotics and computer vision, and what can go wrong at each one.
Stage 1: Ingestion and integrity
Files arrive from many devices, often in batches, sometimes with interrupted uploads. The first stage registers every file, verifies checksums, extracts technical metadata (resolution, frame rate, codec, duration) and links each file to a capture session: who recorded it, where, on which rig, under which consent record. Anything that cannot be linked to a session is quarantined rather than processed. Problems missed here propagate through every later stage.
Stage 2: Synchronisation
If a session includes more than one stream — video plus IMU, stereo pairs, multiple cameras — the streams must be placed on a shared timeline. Device clocks drift, recordings start at slightly different moments and frame rates can vary. Synchronisation uses hardware timestamps where available and alignment signals (a visible clap, a sharp motion visible in both video and IMU) where not. The result should be verified on a sample of every session, because a small offset silently corrupts every model trained on the data.
Stage 3: Normalisation
Footage is converted to consistent formats: a standard codec and container, a fixed frame rate if the training stack requires one, and lens calibration files attached so wide-angle footage can be undistorted. Normalisation should never discard the original; derived files are reproducible, originals are not.
Stage 4: Privacy and compliance
Real-world footage contains bystanders, screens, documents and locations. Depending on the project and jurisdiction, faces or other identifying details may need to be blurred, segments removed, or audio stripped. Consent records are checked against each session. This stage belongs early in the pipeline: annotating footage that later has to be removed is wasted work.
Stage 5: Episode segmentation
Continuous recordings are cut into episodes: one instance of one task, with a defined start and end. Episode boundaries need written rules — does the episode begin when the hand starts moving or when the object first appears? — because inconsistent boundaries produce inconsistent training examples. Episodes that are incomplete, heavily blurred or off-task are flagged here.
Stage 6: Annotation
Within each episode, annotators add the labels the model needs. For robotics and egocentric video these typically include:
- Temporal sub-action segments (reach, grasp, lift, move, place, release) on a single timeline.
- Object labels, as boxes or masks on keyframes or tracked across frames.
- Hand annotations: left/right, contact state, sometimes keypoints.
- Natural-language descriptions of the task or each step, for vision-language training.
- Episode-level attributes: success or failure, environment, objects involved.
Each label type needs its own guideline section. The guideline is the single most important control on quality — see Annotation guidelines are the real deliverable.
Stage 7: Quality assurance
QA checks that the data matches what was agreed. It operates at several levels: automated checks (no gaps in timelines, segments within episode bounds, no orphaned labels), sampled human review against the guideline, agreement measurement between annotators, and acceptance testing of each delivery against agreed criteria. Failed items return to annotation with a reason. We look at QA in depth in The Importance of Data Quality Assurance in AI Training.
Stage 8: Packaging and delivery
Finally, data is exported in the format the client’s training stack expects — for robot learning, formats such as LeRobot, RLDS or MCAP are common — together with a manifest, the annotation guideline version, calibration files and a datasheet describing collection conditions, known limitations and licensing. A dataset without documentation is hard to trust and harder to reuse.
Where pipelines usually fail
- Synchronisation assumed instead of verified.
- Episode boundaries left to individual judgment.
- Privacy review done after annotation, forcing rework.
- Guidelines that change mid-project without re-checking earlier batches.
- No versioning, so nobody can say which labels a model was trained on.
Conclusion
Turning raw video into training-ready data is an engineering process with distinct stages, each with its own failure modes. The teams that get the most from video data treat the pipeline as a product: documented, versioned and checked at every step.
Mvaak runs this pipeline end to end for egocentric and video datasets — capture, segmentation, annotation, QA and delivery. See Egocentric Video Data and Physical AI data, or start with What Is Egocentric Data and Why Does Physical AI Need It?.
Written by Mvaak — AI Data Services.
