Radiology image annotation: boxes, segmentation,
and the QA clinical AI demands

A model that mistakes a weed for a crop wastes a few seconds of laser time. A model that misses a lung nodule can miss a cancer. That gap in consequence is the single most important thing to understand about radiology image annotation: the labelling task looks superficially like any other computer-vision pipeline — draw boxes, paint masks, place points — but the standard of evidence behind each label, and the quality assurance wrapped around it, belongs to a different discipline entirely. Clinical AI is trained on ground truth, and in medicine "ground truth" is a contested, carefully-adjudicated thing.

H2L Robotics India is a multi-sector annotation studio. Our deepest domain is agritech, where we label the field imagery that trains autonomous laser weeders and the POTECTOR crop-disease robot on real Dutch farms, running three-pass overlap and consensus-IoU QA on every batch. Medical imaging is a domain we are building into deliberately, because the annotation craft transfers but the rulebook does not. This article is a practitioner's map of what radiology annotation actually involves — the label types, the measurement standards, and the QA that clinical models genuinely require — written to explain the approach, not to claim a track record we are still earning.

The label types, and when each one earns its keep

Radiology is not one annotation task. The right label geometry depends on what the downstream model needs to do — detect, delineate, measure, or orient — and choosing wrong wastes annotator hours and caps model performance. The main families:

A well-scoped medical project usually combines these. A chest-CT lung-cancer pipeline might use image-level labels for triage, boxes for candidate generation, segmentation for the confirmed nodule, and RECIST diameters for treatment follow-up — four distinct annotation contracts on the same volume, each with its own guideline and QA rule.

RECIST and the measurement discipline

Measurement annotation deserves its own note because it is where casual labelling breaks down. RECIST 1.1 (Response Evaluation Criteria in Solid Tumours) is the standard oncology uses to decide whether a tumour is responding to treatment, and it is unforgiving about definitions. A lesion is only measurable if it is at least 10 mm in longest diameter on CT; a lymph node is only measurable on its short axis and must be at least 15 mm. Up to five target lesions (two per organ) are chosen, their longest diameters summed, and change in that sum over time drives the response category — complete response, partial response, stable, or progressive disease.

For an annotation team this means a "measure the tumour" instruction is nowhere near enough. Which slice? Which axis? Does this node qualify? Is this lesion a valid target or should it be non-target? An annotator who does not internalise the standard will produce measurements that are individually plausible and collectively useless for training a longitudinal model. This is exactly the kind of domain rulebook we already build for agritech crop protocols — the subject changes, the discipline of a precise, testable guideline does not.

Ground truth is earned, not drawn

Here is the deepest difference from general computer vision. In most annotation work, the annotator is the source of truth: a well-trained labeller looking at a cat produces the correct label by definition. In radiology, the image often does not contain enough information to settle the question, and the annotator's opinion is not automatically ground truth. The truth may live in the pathology report, the biopsy result, the surgical finding, or the patient's later outcome.

Serious clinical datasets exploit this. Findings can be extracted from the free-text radiology report (with all the noise natural-language labelling introduces), image-level opinions can be gathered from multiple readers, and where it exists, reference-standard confirmation — pathology, follow-up imaging, confirmed diagnosis — anchors the label to something harder than an eyeball. A mature annotation plan is explicit about which layer of truth each label rests on, because a model told that a report-derived label and a biopsy-confirmed label are equally certain will learn the wrong lesson.

The QA clinical AI actually demands

Multi-pass QA is our standard practice in agritech. Clinical work raises the bar in specific, non-negotiable ways:

• Multi-reader labelling. Findings are not single-annotator calls. Widely-used chest X-ray benchmarks illustrate the pattern: validation studies annotated by a panel of board-certified radiologists, and test sets fixed by a consensus of several readers. Multiple independent reads expose the genuine disagreement inherent in image interpretation instead of hiding it.
• Adjudication of disagreement. When readers disagree, a defined process resolves it — a senior adjudicator, a tie-break reader, or a documented consensus meeting — and the disagreement itself is logged, because inter-reader variability is data about label difficulty, not noise to be discarded.
• Explicit uncertainty. A "positive / negative / uncertain" scheme respects that radiologists themselves hedge. Collapsing uncertainty into a false binary is one of the most common ways to poison a clinical training set.
• Agreement metrics that fit the geometry. Consensus IoU for masks and boxes (the same discipline we run on crop imagery), and reader-agreement statistics for image-level findings — measured, tracked, and used to retrain guidelines, not filed away.
• A quality and regulatory mindset. Clinical-AI ground truth may end up supporting a regulated medical device. That demands traceability: who labelled what, under which guideline version, adjudicated by whom, against which reference standard. Provenance is part of the deliverable.

Privacy is a first-class requirement

Medical images are protected health information, and a chest scan is not de-identified simply by hiding a name. DICOM files carry patient identifiers across dozens of header attributes, and identifying text can be burned directly into the pixels of ultrasound and scanned film; some modalities even reconstruct a recognisable face. Rigorous de-identification follows a published standard — the DICOM PS3.15 Attribute Confidentiality Profile, aligned with frameworks such as HIPAA Safe Harbor — to systematically strip or replace identifiers while preserving the clinical content a model needs. For any studio handling this data, secure, access-controlled workflows and a defensible de-identification step are preconditions for touching the imagery at all, not afterthoughts.

The transferable lesson from our agritech practice is that reliable labels come from disciplined guidelines and layered QA, not from more annotators working faster — and in medical imaging that discipline is mandatory rather than merely advisable. If your team is scoping a radiology annotation programme and wants to talk through label geometry, measurement standards, multi-reader QA design, or a de-identification workflow, get in touch — we would rather help you scope it rigorously than quickly.

Frequently asked

Should I use bounding boxes or segmentation for a medical imaging model?

Use boxes when the model only needs to detect and roughly localise a finding — screening and triage. Use segmentation when the model must reason about shape, margins, or volume, such as radiotherapy planning or volumetric tumour follow-up. Segmentation costs several times more per image, so match the geometry to the clinical question rather than defaulting to the most precise option.

Why do radiology labels need multiple readers when other annotation tasks do not?

Because a radiology image often does not fully determine the answer — expert readers genuinely disagree, and a single annotator's opinion is not automatically ground truth. Multiple independent reads plus a defined adjudication step expose that real disagreement and produce a defensible consensus label, and where possible the label is anchored to a harder reference standard like pathology or follow-up imaging.

Is removing the patient name enough to de-identify a medical scan?

No. DICOM files hold identifiers across many header attributes, text can be burned into the image pixels themselves, and some 3D scans can reconstruct a recognisable face. Proper de-identification follows a published standard such as the DICOM PS3.15 Attribute Confidentiality Profile, aligned with frameworks like HIPAA Safe Harbor, and is a precondition for handling the data at all.