MedSAM and foundation models in medical
imaging: what they change for annotation

For most of the last decade, segmenting a medical image meant training a bespoke model for one organ, one modality, one disease. A network that could outline liver tumors on CT knew nothing about lesions on a brain MRI, and a team that wanted both trained both — from scratch, on hand-labeled data. That specialization is precisely what foundation models are now dissolving. A single promptable network, trained once on a vast and varied corpus, can be pointed at almost any structure in almost any modality and produce a usable mask.

The headline model here is SAM (the Segment Anything Model) from Meta AI, and its medical descendant, MedSAM. For anyone who runs an annotation pipeline, these are not just research curiosities — they change the economics and the shape of the work. But they change it in a specific and often misunderstood way. The right mental model is not "the model labels the data now." It is "the model does the tracing, and your experts do the judging." This article explains what promptable segmentation is, what MedSAM actually delivers, and why — counterintuitively — expert review becomes more load-bearing in a foundation-model workflow, not less.

What "promptable segmentation" actually means

Classic segmentation models are fully automatic: image in, mask out, no interaction. SAM introduced a different paradigm. Instead of predicting a fixed set of categories, it takes an image plus a prompt — a click, a bounding box, or a rough point — and returns the mask of the object you indicated. The model does not need to know the object is a "kidney" or a "tumor"; it only needs you to say where. This is interactive, promptable segmentation, and it maps almost perfectly onto how a human annotator already works: you look, you point, the boundary appears.

SAM was trained on over a billion masks from natural images — photos of everyday scenes — and it is remarkable at objects with crisp, high-contrast edges. Medical images are a harsher test. As the MedSAM authors document, SAM struggled badly on typical medical targets: structures with weak boundaries, low contrast, and ambiguous edges — exactly the properties of soft-tissue lesions, infiltrating tumors, and organs that blur into their neighbors on a grayscale scan.

What MedSAM changes

MedSAM is SAM adapted to the medical domain. It was fine-tuned on a large-scale curated dataset of more than 1.5 million image–mask pairs spanning 10 imaging modalities and over 30 cancer types, and it was validated across 86 internal and 60 external tasks. The result outperformed modality-specific specialist models on that benchmark while remaining a single, general-purpose network. Published in Nature Communications in 2024, it is one of the clearest demonstrations that the foundation-model recipe transfers to medicine when you feed it enough well-curated medical data.

The follow-up, MedSAM2, extends the idea to 3D volumes and video — CT and MRI stacks, ultrasound and endoscopy clips — using a memory-attention mechanism to keep a mask coherent as it propagates across slices or frames. In a large user study, the authors reported reductions in manual annotation effort of more than 85% across CT lesions, liver MRI lesions, and echocardiogram frames. That number is worth sitting with: it describes effort reduced, not effort eliminated. The residual 15% is where the real work — and the real risk — now lives.

The workflow it produces: pre-annotate, then correct

In practice, a foundation model slots into annotation as a pre-annotation engine. The pattern is straightforward and it is the one we build our own pipelines around: the model produces a first-pass mask from a light prompt, and a trained annotator then accepts, refines, or rejects it. This is human-in-the-loop labeling, and it inverts the cost structure. The expensive, slow part used to be tracing every boundary pixel by hand. Now the expensive part is judgment — and judgment is where you want your expensive people spending their time.

This is the same principle that governs our agritech annotation work, where we label field imagery to train autonomous laser weeders and the POTECTOR crop-disease robot on real Dutch farms. Model-assisted pre-annotation gets a crop or a weed 80% outlined; the annotator's job is the contested 20% — the overlapping leaves, the occluded stem, the ambiguous edge. The tooling differs across sectors, but the workflow logic is identical: let the model do the tracing, and reserve human attention for the decisions that carry consequences.

Why expert review becomes MORE important, not less

It is tempting to read "85% less manual effort" as "85% less need for experts." That inference is not just wrong — in medical imaging it is dangerous. Three reasons:

• Confident, plausible errors. A foundation model produces a clean, smooth, believable mask even when it is wrong. A hand-tracing that is off looks messy; a model mask that is off looks authoritative. Automation bias means reviewers under-scrutinize confident output, so a mask that is subtly clipping a tumor margin can pass unless a real expert is actively looking for it.
• The errors cluster on the cases that matter. Models fail hardest on exactly the low-contrast, weak-boundary, rare-presentation cases that are also the most clinically important. The easy 85% the model handles well is the part you least needed help judging. The hard residue is where diagnostic ground truth is made or broken.
• Ground truth is a claim, not a click. In medicine, a label asserts what is anatomically or pathologically true. That assertion has to come from qualified human judgment, ideally with multi-reader consensus. A model's suggestion is an efficiency tool for producing that claim faster — it can never be the source of the claim itself.

This is why our QA philosophy does not relax when a model is in the loop — it tightens. In our agritech work we run 3x-overlap review with consensus IoU agreement to catch exactly the disagreements that a single pass would miss. The same multi-pass, consensus-driven discipline is the approach we bring to medical-imaging annotation: model-assisted speed on the front end, and rigorous, redundant human adjudication on the back end where the ground truth is actually decided.

Domain adaptation and honest evaluation

Two practical caveats keep foundation-model annotation honest. The first is domain adaptation. MedSAM is broad, but "medical imaging" is not one distribution — a 3T MRI from one scanner, a portable ultrasound, and a pediatric CT differ enough that out-of-the-box performance varies. Getting reliable pre-annotations on a specific institution's data usually means light fine-tuning or careful prompting strategies on a representative sample first, and measuring whether it actually helps before trusting it at scale.

The second is evaluation. If the model helps generate labels and you evaluate against those same labels, you have measured nothing. A credible pipeline keeps a held-out, expert-verified ground-truth set that no model touched, and reports agreement (Dice, IoU, or reader consensus) against it. That gold set is the anchor that lets you claim your accelerated pipeline is faster without being quietly worse.

Foundation models are the most significant change to segmentation annotation in years, and the direction is clear: promptable, cross-modality, model-assisted, and much faster. But the value is realized only when the speed is paired with the review discipline the data demands. If you are exploring model-assisted medical-imaging annotation and want a partner who treats expert review and consensus QA as the point rather than an afterthought, get in touch with our team — we would be glad to talk through how this maps to your data.

Frequently asked

Does MedSAM replace human annotators in medical imaging?

No. MedSAM and MedSAM2 accelerate labeling by producing first-pass masks from a click or box, but they still require an expert to prompt them and to accept, refine, or reject the output. Reported effort reductions of over 85% describe faster tracing, not autonomous labeling — the human still owns the judgment and the final ground truth.

What is the difference between SAM and MedSAM?

SAM (Segment Anything Model) is Meta AI's general promptable segmentation model, trained on over a billion masks from natural photos; it excels at sharp-edged objects but struggles on low-contrast medical targets. MedSAM is SAM fine-tuned on more than 1.5 million medical image–mask pairs across 10 modalities, and MedSAM2 extends it to 3D volumes and video.

Why does expert review matter more with foundation models, not less?

Foundation models produce clean, confident masks even when wrong, which invites automation bias, and their errors cluster on the hardest, most clinically important cases. Because a medical label is a claim about what is anatomically true, that claim must come from qualified human judgment — ideally multi-reader consensus — so review becomes the load-bearing step, not an optional one.