2D vs 3D annotation for autonomous-vehicle
perception

Ask two annotation teams to "label the car" and you can get two completely different deliverables. One draws a rectangle around the car in a camera image. The other places a 3D box in a LiDAR point cloud, gives it a heading angle, and carries the same object identity forward through fifty frames while it is simultaneously seen by two cameras and a radar. Both are legitimate. They cost very different amounts, they train different parts of the perception stack, and picking the wrong one is one of the more expensive mistakes an autonomous-vehicle data program can make.

H2L Robotics India's deepest annotation experience is in agritech — we label field imagery that trains autonomous laser weeders and the POTECTOR crop-disease robot on real Dutch farms, running 3x-overlap QA with consensus IoU on every pass. That work is 2D by nature: a camera looking down at crop rows. Autonomous driving is where the geometry gets genuinely three-dimensional, and the questions we get asked most as we extend into this sector are about that jump. This article lays out the honest trade-offs between 2D and 3D annotation for AV perception, and when each one earns its cost.

What 2D annotation actually covers

2D annotation lives in the image plane — pixels, not metres. For AV perception it usually means one of a few things, in ascending order of effort:

• Bounding boxes — axis-aligned rectangles around vehicles, pedestrians, cyclists, and signs. Fast, cheap, and the workhorse of camera-based detection.
• Semantic segmentation — every pixel gets a class label (road, sidewalk, vegetation, sky). No individual objects, just regions. Essential for drivable-area and free-space estimation.
• Instance and panoptic segmentation — pixel-precise masks that also separate one car from the next car. Much slower to draw than boxes, and where most of the labour cost hides.
• Lane and polyline geometry — lane lines, road edges, crosswalks marked as splines or polylines, often with attributes like colour and dashed-vs-solid.

The strength of 2D is density and dollar-efficiency. Cameras are cheap and high-resolution, so a single frame carries enormous semantic richness — texture, colour, text on signs, brake lights. What 2D fundamentally cannot give you is metric depth. A bounding box tells a planner where in the image a pedestrian is, not how many metres away, and monocular depth is an estimate, not a measurement.

What 3D annotation adds — and demands

3D annotation happens in the sensor's real-world coordinate frame, usually built on LiDAR. A LiDAR unit sweeps a scene and returns a point cloud — the 32-beam sensor in the widely-used nuScenes dataset produces roughly 300,000 points per 360° frame. Annotators place cuboids: 3D boxes defined by a centre position (x, y, z), dimensions (width, length, height), and a yaw (heading) angle. That heading is the whole point — it tells the planner not just where a car is, but which way it is pointed and therefore likely to go.

The public benchmarks make the shape of this work concrete. KITTI, collected with a 64-beam LiDAR and stereo cameras, stores its labels as 3D tracklets — temporally tracked cuboids — with the same x/y/z, dimensions, and yaw parameters. nuScenes carries the full sensor suite (6 cameras, 5 radars, 1 LiDAR, all 360°) and annotates objects in the LiDAR frame, then projects those boxes into every camera using calibrated extrinsics. That projection is the key idea behind sensor fusion annotation: you label once in 3D, and the label is geometrically consistent across every sensor that can see the object.

2D vs 3D at a glance

When to use which

The decision is rarely 2D or 3D — mature stacks use both — but the emphasis should follow the task:

• Reach for 2D when the answer is semantic rather than geometric: what colour is the light, is that sign a stop or yield, where does the drivable surface end, is that a dashed or solid lane line. Cameras are the right sensor and depth is not the question.
• Reach for 3D when the planner needs metric truth: distance to the lead vehicle, closing speed, whether an oncoming car's heading implies a turn. Motion planning and collision avoidance run on metres and headings, not pixels.
• Reach for fusion when you need robustness across conditions: radar sees through fog and gives direct velocity, LiDAR nails geometry, cameras carry classification. A fused label lets a model learn to lean on whichever sensor is trustworthy in the moment.

Cost tracks this ladder steeply. A rough industry rule of thumb is that a LiDAR cuboid takes several times longer to place well than a 2D box, and full fused sequence labeling with tracking is more expensive again. That is a budgeting reality, not a reason to over-buy 3D: labeling everything in 3D when half your questions are semantic is a common and costly error.

The hard part: identity consistency

The single most underestimated requirement in AV annotation is not drawing accuracy — it is identity consistency. The same physical object must keep one stable ID across two axes at once: across time (the car in frame 12 is the same car in frame 60, even after it is briefly occluded by a bus) and across sensors (the cuboid in the LiDAR frame is the same object as the box in the left camera and the return in the radar).

Get this wrong and the damage is silent. A tracker trained on data where IDs flicker will learn to drop and re-acquire objects — exactly the failure you least want near a cyclist. This is why datasets like KITTI store labels as tracklets rather than independent per-frame boxes, and why nuScenes triggers each camera exposure to the instant the LiDAR sweep crosses that camera's field of view: without tight temporal synchronization, a fast-moving object sits in two different places in two "simultaneous" sensors, and no annotator can reconcile them.

How we think about it at H2L

We are candid that autonomous driving is a sector we are expanding into, building on years of production agritech annotation rather than a long AV client list. What transfers is the part that is genuinely hard to buy: a trained in-house team of 15+ annotators, fluency in Label Studio, and a multi-pass QA culture that treats consistency as a first-class deliverable, not a nice-to-have. The geometry changes from 2D crop rows to 3D point clouds; the disciplines of clear labeling specs, adversarial review, and measurable agreement do not.

If you are scoping an AV perception dataset and weighing how much of it belongs in 2D versus 3D — or worrying about track continuity across a fused sensor suite — we are happy to talk through the trade-offs for your specific stack. Get in touch and we will give you a straight, practitioner's read.

Frequently asked

Is 3D annotation always better than 2D for autonomous vehicles?

No. 3D cuboids give metric distance, size, and heading that motion planning needs, but they cost several times more per object and add nothing when the question is purely semantic — traffic-light colour, sign type, lane-line style, or drivable-area extent. Mature stacks use both and match the annotation type to the task rather than defaulting to 3D everywhere.

What is sensor-fusion annotation?

It means labeling an object once in 3D (typically the LiDAR coordinate frame) and projecting that single label into every other synchronized sensor — cameras and radar — using calibrated extrinsics. The object keeps one identity across all sensors, so a model can combine camera semantics, LiDAR geometry, and radar velocity for the same target. It depends on tight temporal synchronization between sensors.

Why is identity consistency so important in AV data?

Perception has to track objects over time and across sensors, not just detect them frame by frame. If the same car gets different IDs across frames or sensors, a tracker learns to drop and re-acquire objects — a dangerous failure near pedestrians and cyclists. That is why benchmark datasets store labels as temporally tracked tracklets and enforce ID-switch and track-continuity checks in QA.