Occlusion is where video datasets quietly break

Bounding a fully visible object is not difficult, and it is not where video annotation goes wrong. The trouble starts the moment something passes in front of it.

Three defensible answers, one dataset

When a car is half behind a lamp post, an annotator has to choose. Box only the visible pixels? Box the full extent including the hidden part, inferred? Split into two boxes? Drop the frame entirely?

Every one of those is defensible. That is precisely the problem — in the absence of a written rule, different people pick different answers, and often the same person picks differently on a tired afternoon. The model then learns a boundary definition that shifts unpredictably with occlusion, which is the exact condition it most needs to handle.

Identity across the gap

The second decision is harder. An object leaves frame, or is fully hidden for twenty frames, then reappears. Is it the same track?

This is not a question about the world. It is a question about what you want the model to learn. If re-identification matters, the track should persist and the gap should be marked. If it does not, a new identity is cleaner. Either is fine. What is not fine is annotators deciding case by case, which produces tracks that fragment for reasons no model can infer.

Consistency over precision

There is a persistent instinct to make each individual box as tight as possible. It is the wrong optimisation. A box drawn two pixels loose every single time is a rule the model can learn. A box drawn tight on Monday and loose on Thursday is noise dressed up as precision.

Review has to look across frames

Frame-level review misses the entire class of problems described above. Every individual frame can look correct while the sequence is incoherent — a track that drifts, an identity that swaps, a boundary definition that tightens as the annotator gets more confident.

• Review sequences, not sampled frames
• Check identity continuity across every occlusion event
• Watch for boundary drift over the length of a long clip
• Confirm that tracks end for a stated reason, not because attention lapsed

None of this is exotic. It is the difference between a dataset that trains a tracker and one that trains a detector which happens to have been applied to video.

Small objects and motion blur

Two adjacent problems produce the same kind of silent inconsistency. Below some size, an object is a few ambiguous pixels and different annotators will disagree about whether it is there at all. Under fast motion, the boundary genuinely is not a boundary.

Both need an explicit rule. A minimum size below which objects are not annotated is arbitrary but checkable; leaving it to judgement produces a dataset where small-object recall is essentially random.

Interpolation is not free

Most annotation tools will interpolate boxes between keyframes, and it saves a great deal of time. It also produces confident-looking annotations on frames nobody examined, which drift whenever motion is non-linear.

Interpolation is fine as a starting point and a problem as a deliverable. The rule worth setting is how far apart keyframes may be given the motion in the clip, and that reviewers check interpolated frames rather than only the ones a human placed.

What to specify before annotation starts

• Visible-pixels or full-extent boxing under occlusion, with a worked example of each
• The occlusion duration after which a track ends rather than persists
• A minimum object size, in pixels, below which objects are not annotated
• Whether truncation at the frame edge is treated the same as occlusion
• Maximum keyframe spacing, and whether interpolated frames are reviewed