When annotators disagree, look at the schema first

Text annotation has a particular failure mode: the task looks easy enough that nobody specifies it carefully, and the disagreement that follows gets attributed to annotator quality rather than to the label set.

Random disagreement versus structured disagreement

The distinction matters more than the overall agreement number. Disagreement scattered evenly across a corpus usually does indicate attention or training problems. Disagreement concentrated on one kind of example indicates something else entirely: the schema does not cover that case.

So the first diagnostic is not "how much did they disagree" but "where". Group the disagreements. If a cluster appears, you have found an underspecified definition, not a careless annotator.

Boundaries are decisions

Consider tagging an organisation in "the former Bank of England governor". Does the entity include "the"? Does it include "former"? Is "Bank of England governor" a role rather than an organisation?

There is no fact of the matter. There is only a decision, and the decision changes what the model learns about span boundaries. Left to individual judgement, it changes per annotator — and the model learns an average of several incompatible rules.

The middle carries the weight

Clear examples teach a model little it could not have inferred. The examples between two labels are where a dataset earns its cost.

This has a practical consequence for how guidelines are written. Most of the document should be about the awkward middle. If your guideline spends its length carefully defining the categories and one paragraph on exceptions, it is inverted.

Escalate rather than guess

An annotator who encounters a genuinely ambiguous example has two options: make a judgement call and move on, or flag it. The first is faster and produces a dataset with invisible inconsistency. The second is slower and produces a schema that improves.

• Make flagging cheap — a single field, not a separate process
• Resolve flags in batches so decisions are made consistently
• Write each resolution into the guideline as a worked example
• Re-annotate the affected examples rather than leaving the old decision in place

The other-category trap

Almost every schema grows a catch-all: other, miscellaneous, unclear. It is genuinely useful, and it is also where evidence goes to hide. Examples that should have prompted a new category get filed under other, agreement looks healthy, and the gap in the label set never surfaces.

The fix is to treat that category as a queue rather than a destination. Review what accumulates in it periodically; if a coherent group appears, it has earned a label of its own.

Ambiguity in the source, not the schema

Some examples are genuinely ambiguous — a sentence that two fluent readers understand differently, a message whose sentiment depends on context nobody has. No schema resolves those, and forcing a single label onto them teaches the model a confidence the data does not support.

Better to mark them. An explicit ambiguous flag lets you exclude them from training, keep them for evaluation, or model the uncertainty deliberately. All three are defensible. Silently picking one label is not.