Inter-annotator agreement, and what it actually tells you

Inter-annotator agreement is the closest thing dataset work has to a universal quality metric, which is exactly why it gets misused. A single number gets quoted, a threshold gets set, and the far more informative structure underneath it goes unexamined.

It measures the schema as much as the people

Low agreement is routinely read as an annotator problem. Sometimes it is. More often it means the label set cannot be applied consistently, which is a property of the schema and not of the people applying it.

The useful reframing: agreement measures how well-specified your task is. Treated that way, a low score early in a project is not bad news. It is a map of what to fix.

Look at the distribution, not the average

Two projects can report the same overall agreement and be in completely different states. In one, disagreement is spread thinly across everything — a training or attention issue. In the other, agreement is near-perfect except on one category where it collapses — a definitional issue affecting a specific, fixable part of the schema.

High agreement is not automatically good

Agreement can be high for uninteresting reasons. If one label covers ninety percent of examples, two annotators who both default to it will agree most of the time while telling you nothing about the cases that matter. Chance-corrected measures help, but the more direct check is to look at agreement on the minority classes specifically.

Agreement can also be high because annotators have converged socially — sitting together, resolving cases by discussion, drifting toward a shared interpretation that is not written anywhere. That produces a consistent dataset and an unwritable guideline, and it stops working the moment a new annotator joins.

Measure it more than once

Agreement measured once at the start of a project tells you about the guideline as it was then. Definitions drift, new edge cases appear, annotators get faster and more confident in ways that are not always improvements.

• Take an agreement sample at the start, and again at intervals through the work
• Break results down by class and by annotator
• Investigate structured disagreement as a schema problem first
• Report the figures at delivery, alongside the escalated cases

Done that way, agreement stops being a gate to pass and becomes what it is genuinely good for: a continuous read on whether the definition still holds.

Which measure, and why it matters less than people think

Raw percentage agreement is easy to explain and overstates performance when one class dominates. Chance-corrected measures adjust for that and become unstable when a class is very rare. For span-level or bounding-box tasks, neither applies cleanly and an overlap-based measure is more informative.

Choose one, state it, and keep it. The comparison that matters is against your own earlier measurements on the same task, not against a threshold borrowed from a different domain.

Adjudication is where the value is

Measuring agreement and then discarding the disagreements wastes most of the exercise. Those examples are a ready-made list of the places your definition is weakest.

Work through them deliberately: decide each one, write the decision into the guideline as a worked example, and re-annotate anything already affected. Done consistently, the agreement figure rises for the right reason — the task got clearer, not the annotators more compliant.