Annotation guidelines are the real deliverable

There is a moment on most annotation projects where someone notices that two people have labelled the same example differently. The instinct is to treat this as a quality incident — someone was careless, retrain them, move on. Almost always it is something else: the guideline did not actually say what to do, and two reasonable people filled the gap differently.

Disagreement is rarely a people problem. It is usually the schema telling you it is underspecified.

What a guideline is for

A guideline is not a training document. It is the specification of what correct means for this dataset, in enough detail that two people who have never met produce the same output on the same input. That is a high bar, and most guidelines fail it not through vagueness in the obvious cases but through silence on the awkward ones.

The obvious cases write themselves. Nobody needs instruction on whether a clearly visible car is a car. The guideline earns its cost on the third of examples where something is partly hidden, partly ambiguous, or sits between two categories.

Write it from the edge cases inward

A guideline written from first principles tends to describe the ideal case at length and the difficult ones not at all. A better method is to annotate a small sample first, deliberately surface the disagreements, and write the document around those.

• Take one or two hundred examples that are representative, not easy
• Have two people annotate them independently, with no discussion
• List every disagreement and resolve each one explicitly
• Write the resolution into the guideline as a worked example, not a rule in the abstract
• Re-run on fresh examples until disagreement is rare and boring

This feels slow. It replaces a week of guideline-writing with two weeks of guideline-writing and sample annotation. It also removes the far more expensive alternative, which is discovering the ambiguity at volume and re-annotating.

Worked examples beat prose

"Include the full extent of the object including occluded portions where they can be reasonably inferred" is a sentence that will produce three different behaviours. Two annotated images — one showing what to do, one showing what not to do — will produce one.

Where a rule is hard to state, show it. Where a rule has an exception, show the exception next to the rule rather than in a separate section nobody reads.

It should ship with the data

A dataset delivered without its guideline is difficult to extend and impossible to audit. Six months later, when someone wants to add a category or investigate why the model behaves oddly on a particular case, the only record of what the labels meant is the labels themselves.

Treating the guideline as part of the deliverable changes how it gets written. It stops being internal scaffolding and becomes an artefact that has to stand on its own — which is exactly the property that makes a dataset extensible.

Version it like code

A guideline changes during a project — that is a sign it is working. The problem is that examples annotated under version one and version three are not interchangeable, and nothing in the data itself records which rule applied.

Stamping each batch with the guideline version it was produced under costs nothing at the time and is the only thing that makes selective re-annotation possible later. Without it, the choice on discovering a mistake is to redo everything or to live with an inconsistency you cannot locate.

Who should write it

Guidelines written entirely by the team who will train the model tend to be conceptually clean and operationally impossible — they assume knowledge the annotator does not have. Guidelines written entirely by the annotation team tend to be practical and subtly misaligned with what the model needs.

The version that works is drafted by whoever understands the model, tested by whoever will apply it, and revised until the second group stops asking the first group questions.