Measuring annotation quality: IoU,
consensus, and multi-pass QA

Every team that outsources data labeling eventually asks the same question: how good is this data, really? The uncomfortable truth is that "it looks accurate" is not an answer you can put in a report, tie to a service-level agreement, or use to compare one annotator against another. Quality has to be measured, and measured with the right instrument for the task — a bounding-box job, a segmentation mask, and a text-classification pass each fail in different ways and demand different math.

This is a practical tour of the metrics that matter: how to score geometric overlap with IoU, how to measure whether two humans actually agree using kappa, how to fold many opinions into a single defensible label with consensus, and how to wire all of it into a multi-pass QA loop that catches errors before they reach a model. It is tool-agnostic, though we will point to how we run it in practice at H2L India, where our deepest experience is in agritech imagery — labeling Dutch field data that trains autonomous laser weeders and the POTECTOR crop-disease robot — and where the same measurement discipline carries directly into medical, autonomous-vehicle, and text work.

IoU: the workhorse for spatial labels

Intersection over Union (IoU), also known as the Jaccard index, is the default way to compare two spatial annotations. You take the area where the two shapes overlap (the intersection) and divide it by the total area they cover together (the union). The result is a number between 0 and 1: a perfect match scores 1, no overlap scores 0. It is the same idea whether the shapes are bounding boxes, polygon masks, or the accepted radius around a keypoint.

IoU is unforgiving in a useful way. A box that is close but consistently shifted, or a polygon that clips the edge of a leaf, is penalized in proportion to how much area it gets wrong. That makes it far more honest than a raw "correct / incorrect" flag. In evaluation pipelines a threshold turns the continuous score into a decision — a prediction that clears, say, 0.50 IoU against ground truth is counted as a true positive, and anything below is a miss.

The important discipline is picking the threshold to match the cost of the task. A weed-detection box that just needs to steer a spray nozzle can live with 0.5. A polygon tracing the exact margin of a diseased leaf — where the whole point is measuring how far the disease has spread — should be held to a much tighter bar. Choose the number before you start scoring, not after you see the results.

Kappa: measuring agreement on categories

IoU answers spatial questions. But a huge share of annotation is classification: is this a weed or a crop, is this lesion benign or malignant, is this review positive or negative. Here the naive metric — percent of times two annotators picked the same label — is misleading, because two people flipping coins will still "agree" half the time by pure chance.

Cohen's kappa fixes this by discounting the agreement you would expect from chance alone. It compares two annotators on categorical labels and returns a score where 1 is perfect agreement and 0 is no better than random. Fleiss' kappa generalizes the same idea to any number of annotators labeling a shared set of items, which is what you usually have on a real team. A widely used reading of the numbers looks like this:

• 0.81 – 1.00 — almost perfect agreement
• 0.61 – 0.80 — substantial (a common minimum bar for shipping)
• 0.41 – 0.60 — moderate; usually a signal your guidelines are ambiguous
• Below 0.40 — fair to slight; stop and fix the instructions, not the people

Consensus and best-peer scoring

Once you have several people labeling the same item, you need a rule for turning their opinions into one answer. For categories, majority vote is the simplest form of consensus. For spatial labels there is no "majority," so we compute pairwise IoU across the overlapping annotations and look at how tightly they cluster. High mutual overlap means the item is easy and the label is trustworthy; low overlap flags an item that genuinely needs a senior reviewer.

A refinement we rely on is best-peer (consensus IoU) scoring: rather than judging an annotator only against a single gold answer, score each person's shape against the best-matching peer shape on the same object. This separates two very different failures — an annotator who is sloppy (low agreement with everyone) versus an item that is objectively hard (everyone disagrees). In our own Label Studio pipeline we route roughly every third image to triple overlap so this consensus signal exists as a matter of routine, not as a special audit.

Gold standard: the anchor you still need

Consensus tells you whether annotators agree with each other. It does not tell you whether they are collectively right — a whole team can share the same misconception. That is what a gold-standard set is for: a bank of items labeled and adjudicated by your most expert reviewers, treated as ground truth. Silently seeding gold items into normal work lets you measure each annotator's IoU or kappa against a known-correct answer, catch drift early, and calibrate new hires objectively. Team-consensus IoU and gold IoU are different measurements answering different questions, and a serious QA program tracks both.

Multi-pass QA and adjudication

Metrics are only half the system; the other half is the workflow that acts on them. Multi-pass QA means an annotation is never trusted on a single human's say-so. A typical loop:

Adjudication is where the value compounds. Every resolved disagreement is a chance to tighten the written instructions so the same edge case never costs you twice, and to track results per annotator over time — turning quality from a one-off audit into a trend you can manage. That feedback loop, not any single metric, is what actually moves quality.

Which metric, when

No single number captures quality. Use IoU for anything spatial and set the threshold to the cost of being wrong. Use Cohen's or Fleiss' kappa for categorical labels so chance agreement does not flatter you. Use consensus and best-peer scoring to find the genuinely hard items, and a gold standard to make sure the whole team is anchored to truth. Wrap the lot in a multi-pass loop with real adjudication, and quality stops being a gut feeling and becomes something you can report, defend, and improve.

If you are weighing how to set thresholds, structure a gold set, or design a QA loop for your own data — in agritech or in a new domain you are building out — we are happy to compare notes. Get in touch and tell us what you are trying to measure.

Frequently asked

What IoU threshold should I require for my labels?

It depends on the cost of being wrong, not on convention. A detection or counting task that only needs to locate an object can accept around 0.5 IoU, while a segmentation task where the boundary itself is the measurement — a disease margin or a lesion edge — should be held much tighter. Decide the number before you start scoring so it reflects the task, not the results you happened to get.

Why use Cohen's kappa instead of simple percent agreement?

Because two annotators picking from a small set of categories will agree a large fraction of the time purely by chance, and raw percent agreement gives them full credit for that. Cohen's kappa (for two annotators) and Fleiss' kappa (for more) subtract out expected chance agreement, so a high score genuinely means the labels are consistent rather than lucky.

What is the difference between consensus IoU and gold-standard IoU?

Consensus IoU measures how well annotators agree with each other, which flags hard or ambiguous items. Gold-standard IoU measures how well they agree with an expert-adjudicated correct answer, which catches shared misconceptions the whole team might hold. They answer different questions, so a strong QA program tracks both rather than choosing one.