Medical image segmentation, explained:
X-ray, CT, MRI, dermoscopy and endoscopy

Medical image segmentation is the task of deciding, pixel by pixel, which part of a scan is the thing you care about — a tumour, an organ, a vessel, a skin lesion, a fracture line — and which part is everything else. It is the quiet foundation under most clinical AI: before a model can measure a tumour's volume, flag a nodule, or plan a radiotherapy dose, someone has to teach it where the boundary lies. That teaching signal comes from human annotators, and the quality of the boundary they draw propagates directly into every downstream measurement.

What makes medical segmentation genuinely different from labelling cats and cars is that the images are physically diverse, the boundaries are often ambiguous even to experts, and the cost of a wrong edge is measured in millimetres of tissue rather than a misclassified photo. This article walks through the main imaging modalities, the annotation types they call for, how the data is actually stored and reconstructed in 3D, and why consensus and expert review are not optional niceties but structural requirements. H2L India labels agritech field imagery today — Dutch potato and weed detection for autonomous laser weeders — and the same discipline that makes crop segmentation reliable is what we bring to how we think about medical work.

The modalities, and what "a boundary" means in each

Segmentation is not one problem. It changes shape depending on how the image was formed, because each modality carries different physics, dimensionality, and noise.

Two structural axes fall out of this list. First, 2D versus 3D: X-ray, dermoscopy and endoscopy are planar, while CT and MRI are volumetric stacks that must be labelled with 3D consistency in mind. Second, still versus video: endoscopy and ultrasound add a temporal dimension where a boundary must stay coherent across frames.

The annotation types you actually use

"Segmentation" is shorthand for a family of label types, and choosing the right one is half the battle. Over-specifying wastes expensive clinician time; under-specifying starves the model.

• Pixel / semantic masks: every pixel gets a class (liver, tumour, background). This is true segmentation and the backbone of volumetric measurement.
• Instance masks: like semantic masks but each object is a separate entity — three polyps get three distinct labels, not one merged blob.
• Organ and lesion outlines (contours): a traced boundary rather than a filled region, common in radiotherapy planning where the contour is the deliverable.
• Detection boxes: a coarse bounding box saying "a nodule is somewhere here." Cheaper, faster, useful for triage or as a first pass before fine masking.
• Landmark keypoints: discrete anatomical points — vertebral centres, cephalometric landmarks, catheter tips — for measurement and alignment.

A mature pipeline mixes these. You might run detection boxes to localise candidates cheaply, then spend expert effort on pixel-accurate masks only where they matter. The architectures these labels feed — U-Net and its many variants remain the workhorse across CT, MRI and X-ray precisely because they perform well even on the small datasets typical of medicine — are only as good as the masks underneath them.

Why boundary precision has clinical consequences

In consumer vision, a mask that is a few pixels loose is invisible. In medicine it is a measurement error. Tumour volume, growth rate, ejection fraction, stenosis percentage and radiotherapy dose maps are all computed from the segmented boundary. A contour drawn two millimetres wide of the true edge does not just look slightly off — it changes the number a clinician acts on, and it can push healthy tissue into a radiation field or exclude disease from it.

This is why evaluation in medical segmentation leans on overlap metrics like the Dice coefficient and IoU rather than simple accuracy: a mask can be 99% "accurate" pixel-for-pixel and still miss the small, thin, high-stakes structure entirely. It is also why we treat boundary regions as the place to spend review effort, not the place to rush.

DICOM, NIfTI, and 3D reconstruction

Before anyone labels anything, the data has to be handled correctly — and medical formats are not ordinary images. DICOM (Digital Imaging and Communications in Medicine) is the ubiquitous clinical standard: a tag-based format where each file typically holds a single 2D slice plus extensive clinical and patient metadata. A single CT study can therefore be hundreds of separate DICOM files that must be kept in order and, critically, stripped of protected health information before annotation.

NIfTI, developed under the US National Institutes of Health for neuroimaging, takes the opposite approach: it consolidates a whole scan into one 3D (or 4D) volume, which is far more convenient for computational pipelines. A standard first step in many workflows is converting DICOM to NIfTI so that a CT or MRI can be treated as a single addressable volume rather than a pile of slices.

That volume is what enables 3D reconstruction: because CT and MRI slices are acquired at known spacing, individual slice contours stack into a coherent three-dimensional model of an organ or lesion. Good annotators do not label slices in isolation — they check that a structure's boundary is consistent from slice to slice, because an edge that jumps around between adjacent slices produces a jagged, unusable 3D surface. Volumetric consistency is a quality signal in its own right.

Inter-observer variability, consensus, and adjudication

Here is the uncomfortable truth that shapes the whole discipline: expert radiologists and pathologists frequently disagree with each other — and with their own earlier work — on where a boundary lies. This inter- and intra-observer variability is well documented; in some tumour-segmentation studies the variability between observers left only a minority of derived quantitative features stable enough to trust. If you train a model on one person's opinion, you bake that person's bias into the system.

The response is structural, not heroic. Rather than trusting a single pass, robust pipelines gather multiple independent annotations and fuse them into a consensus reference — the widely used STAPLE algorithm, for instance, estimates each annotator's reliability and produces a probabilistic consensus segmentation, and newer probabilistic and diffusion-based methods model the disagreement explicitly rather than averaging it away. Where annotators disagree beyond a threshold, an adjudication step routes the case to a senior reviewer who resolves the boundary.

Practically, that means a defensible medical-segmentation workflow tends to include: de-identification of DICOM data up front; trained annotators working to a written protocol per modality; independent multi-pass labelling on the cases that matter; agreement measured with overlap metrics; explicit adjudication of disagreements; and expert sign-off before anything is called ground truth. None of these steps is glamorous. All of them are the difference between a dataset a clinician can rely on and one they cannot.

Getting it right, in short

Medical image segmentation rewards teams that respect the specifics: match the annotation type to the clinical question, handle DICOM and NIfTI data properly, preserve 3D consistency across slices, treat the boundary as the crucial region, and build consensus and adjudication into the process rather than hoping one expert got it right. That is a workflow discipline as much as it is a labelling skill.

H2L India's roots are in high-overlap, QA-driven agritech annotation, and that same methodology is how we approach medical imaging as we expand into it. If you are building a segmentation dataset and want to talk through modality, annotation design, or a QA pipeline, get in touch — we are happy to think it through with you.

Frequently asked

What is the difference between DICOM and NIfTI, and why does it matter for annotation?

DICOM is the clinical standard that stores each 2D slice as a separate, metadata-rich file, so one CT scan can be hundreds of files that must be de-identified before labelling. NIfTI consolidates a whole scan into a single 3D or 4D volume, which is easier for computational and annotation pipelines. Converting DICOM to NIfTI is a common first step so a scan can be handled as one coherent volume with consistent 3D boundaries.

Why do you need multiple annotators for medical images instead of one expert?

Even experienced radiologists and pathologists disagree on where a boundary sits, and disagree with their own earlier work — this is inter- and intra-observer variability. Training on a single pass bakes that person's bias into the model. Gathering several independent annotations and fusing them into a consensus reference (for example with the STAPLE algorithm), then adjudicating hard cases with a senior reviewer, produces far more reliable ground truth.

Which annotation type should I use for a medical imaging project?

It depends on the clinical question. Use pixel or instance masks when you need volume or precise shape; contours for radiotherapy planning; detection boxes for cheap localisation or triage; and landmark keypoints for measurement and alignment. Many pipelines combine them — coarse boxes to find candidates, then expert pixel-accurate masks only where precision truly matters.