In-house vs crowdsourced annotation:
what changes for quality

Ask two teams to label the same 10,000 images and you can get two very different datasets — not because the images changed, but because the people and process behind the labels did. The single biggest lever on a training set's quality is rarely the annotation tool or the guideline document. It's the operating model: who does the work, how well they understand the domain, and how fast a mistake gets caught and corrected.

The choice usually gets framed as in-house versus crowdsourced, but that hides a third option most teams actually want. On one end sits the anonymous crowd platform — thousands of on-demand workers, elastic capacity, low unit cost. On the other end sits building and training your own annotation team from scratch — maximum control, maximum overhead. In the middle sits the in-house managed studio: a small, trained, permanent team that behaves like your own staff without you having to hire, train and manage them. This article compares all three across the dimensions that actually move quality.

What "quality" really means for a label

Before comparing models, it helps to be precise about what we're optimising. A good label is not just "correct" — it's correct and consistent with every other label in the set. A model learns the pattern you show it repeatedly, so systematic inconsistency (one annotator draws tight boxes, another draws loose ones; one treats a shadow as part of the weed, another doesn't) is often more damaging than the occasional honest error.

Two families of metrics capture this. Inter-annotator agreement — Cohen's kappa for two raters, Fleiss' kappa for more — measures how often independent annotators make the same call, corrected for chance. For spatial tasks like bounding boxes and segmentation masks, Intersection over Union (IoU) measures geometric overlap between two annotations of the same object, scored 0 to 1. These aren't just evaluation metrics you run at the end; used well, they're a live signal that tells you whether your workforce is converging on one shared definition of the task or quietly drifting apart.

The three models at a glance

Where the crowd wins — and where it hurts

Crowdsourcing earns its reputation on two axes: scale and speed. If you need a million images bucketed into "cat / not cat," or a large corpus of short text snippets rated for sentiment, a crowd platform can turn that around in days at a unit price an in-house team can't match. For simple, self-evident, high-volume tasks with a low cost of a single wrong label, it is often the right tool.

The trade-offs show up as the task gets harder or more specialised. Anonymous workers rotate constantly, so the person labelling your data today may never see it again — institutional knowledge doesn't accumulate. Quality is typically bought back through redundancy: send each item to several workers and take the majority vote. That works for categorical tasks but is expensive and blunt for nuanced ones, and it can bake in systematic bias from task design rather than remove it — the way you word a question or lay out an interface measurably shifts what the crowd produces. And because motivation is per-task, the incentive is to finish fast, not to flag the ambiguous edge case that your model most needs handled correctly.

Where in-house changes the equation

A permanent team changes quality along four dimensions the crowd struggles with.

• Consistency. The same people apply the same guideline day after day, so labels converge on one shared standard. Agreement scores tend to rise over weeks rather than reset with every new batch of workers.
• Domain expertise. Some judgments can't be crowdsourced because the crowd doesn't know what it's looking at. Distinguishing an early-blight lesion from ordinary leaf mottling, or a volunteer potato plant from the weed you want a laser weeder to zap, is domain knowledge that compounds with exposure. H2L's annotators built this labelling field imagery for autonomous weeders and the POTECTOR crop-disease robot on real Dutch farms — the same depth we now bring to medical imaging, autonomous-vehicle and text/voice work, where the frame is capability and method rather than a decade-long track record.
• Feedback loops. With a fixed team you can review, correct and re-teach. A missed edge case becomes a guideline update and a five-minute conversation, and the whole team improves. That loop simply doesn't exist with anonymous, rotating labour.
• Data security. Medical scans, in-cabin footage and proprietary field data carry real privacy and IP obligations. A known, vetted, contractually-bound team working in a controlled environment is a fundamentally different risk posture than fanning sensitive data out to an anonymous pool.

How multi-pass QA closes the gap

The reason a small team can beat a large crowd on hard data is process, not headcount. The pattern that works is overlap plus review. At H2L, tricky batches are labelled with roughly 3x overlap — several annotators independently label the same items — and the results are reconciled by consensus IoU, so disagreement surfaces as a number instead of hiding in the dataset. Items where annotators diverge are exactly the items worth a senior reviewer's attention; everything else can move faster.

This is the same statistical idea the crowd uses for redundancy, but applied to a stable, trained team, it does more. Instead of just voting away noise, you diagnose why annotators disagreed, fix the guideline, and lift every future batch. Redundancy on a crowd platform buys you one clean answer for one item. Redundancy on a permanent team buys you a better process. Tooling matters here too — a shared workspace like Label Studio lets you route the ambiguous cases into a review queue rather than shipping them blind.

Choosing the right model

There's no universally correct answer, only a fit for the task. A practical way to decide:

• Reach for the crowd when the task is simple and self-evident, volume is enormous, the cost of a single wrong label is low, and the data isn't sensitive.
• Reach for a managed in-house studio when labels require domain judgment, consistency across a long project matters, the data carries privacy or IP obligations, or you need a feedback loop that makes the work better over time — without absorbing the cost of building a team yourself.
• Build your own internal team when annotation is a core, permanent, differentiating capability and you're prepared to own the tooling, QA design and management that come with it.

Many mature pipelines end up hybrid: crowd-scale the easy majority, route the hard, sensitive or safety-critical slice to a trained team. The mistake is treating "in-house vs crowdsourced" as a loyalty test rather than an engineering decision — and paying for one model's strengths on a task that needed the other's.

If you're weighing where your data sits on that spectrum — and which slices deserve a trained, QA-driven team rather than an anonymous crowd — talk to us. We're happy to be honest about which parts of your pipeline we'd take on and which are better served elsewhere.

Frequently asked

Is crowdsourced annotation always cheaper than in-house?

Per label, usually yes — anonymous crowd platforms have low unit costs and no fixed overhead. But the sticker price is misleading for hard tasks. Once you add the redundancy needed to hit quality (labelling each item several times and voting), plus the cost of rework and model failures from inconsistent or naive labels, an in-house team focused on the difficult slice is often cheaper on a cost-per-usable-label basis.

What agreement score should I expect from a good annotation team?

It depends on the task and metric. For categorical work, Cohen's or Fleiss' kappa above ~0.75 is generally considered excellent, while below ~0.40 is poor. For bounding boxes and masks, teams track Intersection over Union (IoU), where 1.0 is a perfect overlap. More important than any single threshold is the trend: a trained team's agreement should climb as the guideline stabilises, and low-agreement items should be flowing into review rather than into your training set.

Can I use both crowdsourcing and an in-house team?

Yes, and mature pipelines often do. A common pattern is to crowd-scale the simple, high-volume majority of the data and route the hard, ambiguous, sensitive or safety-critical slice to a trained in-house team with multi-pass QA. The key is deciding the split deliberately, based on task difficulty and data sensitivity, rather than defaulting everything to one model.