A computer vision model can only be as consistent as the labels it learns from. Architecture and training recipe matter, but when two teams train similar models on the same images and get very different results, the explanation is often in the annotation: how boxes were drawn, how ambiguous cases were resolved, and whether those decisions were applied the same way every time.
This article looks at how annotation quality affects computer vision models in practice, which kinds of error do the most damage, and what a high-quality annotation process looks like.
Labels are the specification
Supervised models do not learn the task you had in mind. They learn the mapping your labels describe. If annotators sometimes include a shadow inside a bounding box and sometimes do not, the model learns that boxes have an uncertain edge. If “cup” and “mug” are used interchangeably, the model learns a muddled boundary between them. Every annotation decision is, in effect, a line in the specification of the model.
The errors that matter most
Systematic errors
Random noise — a box a few pixels off in either direction — is partly averaged out by training on many examples. Systematic error is not. If one annotator always draws boxes tight and another always draws them loose, or if a class is consistently missed in dark scenes, the model faithfully learns the bias. Systematic errors are the most damaging precisely because they look like signal.
Missing labels
An object present in an image but not labelled is taught to the model as background. For detection models this directly suppresses recall: the model is penalised for finding the very objects it should find. Missing labels cluster in hard cases — small, occluded or partially visible objects — which are often the cases the model most needs to learn.
Inconsistent class boundaries
Most taxonomies have grey zones. Is a person holding a phone to their ear “talking” or “holding phone”? Is a half-open drawer “open”? Without written rules for these cases, different annotators decide differently, and the model sees contradictory examples for the same visual pattern.
Temporal errors in video
In video, labels have a time dimension. An action that starts three frames late in one clip and three frames early in another teaches a model a blurred notion of when actions begin. For tasks such as temporal segmentation or action anticipation, boundary consistency matters as much as spatial accuracy.
What high-quality annotation looks like
A written guideline with examples
Good annotation starts with a guideline that defines every class, every attribute and every boundary rule, illustrated with positive and negative examples and the hard cases already found. The guideline is a living document; it changes as new edge cases appear. We argue in Annotation guidelines are the real deliverable that it is the most valuable artefact a labelling project produces.
Calibration before production
Before labelling at volume, annotators label the same sample and compare results. Disagreements reveal unclear guideline sections, which are fixed before production starts. This is far cheaper than discovering inconsistency after thousands of items are done.
Measured agreement
Inter-annotator agreement — IoU for boxes, boundary distance for segments, kappa-style statistics for classes — is the most useful early indicator of label quality, because it can be measured without ground truth. See Inter-annotator agreement: what it tells you.
Multi-stage review
A second reviewer checks a defined share of every batch, with closer review of difficult classes and new annotators. Rejected items go back with a reason, and recurring reasons feed back into the guideline.
Matching annotation type to the model
- Classification needs consistent class assignment, especially for near-duplicates.
- Object detection needs complete coverage and a consistent box-tightness rule.
- Instance and semantic segmentation need precise edges; budget for it, as it costs far more per item.
- Keypoints and hand pose need visibility flags so occluded points are not guessed silently.
- Tracking needs stable identities across frames, including through occlusion.
- Temporal segmentation needs boundary rules expressed in observable events, not intent.
Conclusion
Annotation quality is not a cosmetic property of a dataset. It sets the ceiling on what a computer vision model can learn and determines how predictably it behaves on hard cases. The process that produces quality — a clear guideline, calibration, measured agreement and structured review — is well understood. The difference between projects is whether it is actually followed.
Mvaak labels video, audio and text to a written guideline with multi-stage review. See Video annotation for detection, tracking and temporal segmentation, or read The Importance of Data Quality Assurance in AI Training.
Written by Mvaak — AI Data Services.
