Lane, drivable-space and road-scene
annotation: an AV perception primer

Ask an autonomous-driving engineer what their perception model needs and you will rarely hear "more images." You will hear "cleaner, more consistent labels." A self-driving stack does not see the road the way a person does; it sees whatever the training data taught it to see. Every lane the car holds, every gap it treats as drivable, every sign it obeys traces back to a human decision made in an annotation tool months earlier. Road-scene annotation is where the driving policy is quietly written.

This primer walks through the main layers of road-scene labeling as they actually appear in production annotation work — lane geometry and its attributes, drivable-area and free-space segmentation, road markings, signs and signals, and the vulnerable road users the car must never miss. Then it covers the two problems that separate a demo from a deployable dataset: occlusion handling and temporal consistency. H2L India's deepest hands-on experience is in agritech field imagery — labeling for Dutch laser-weeding and the POTECTOR crop-disease robot — but the QA discipline that work demands is exactly what road-scene annotation rewards, and it is the lens we bring as we extend into automotive perception.

Lane lines are geometry plus attributes

The most common mistake in lane annotation is treating a lane as a single kind of thing. It is not. A usable lane label carries a geometry and a set of attributes, and the attributes are often what the planner actually cares about. Public benchmarks make this concrete: TuSimple annotates lanes as 2D sample points at fixed pixel-height intervals on stable highway footage, while CULane spans nine harder scenarios — night, shadow, crowded, curve, arrow, dazzle — precisely because attributes and context change how a line should be read.

Geometry is usually captured as a polyline (an ordered set of points along the marking) or, for pixel-accurate models, a thin polygon or mask. On top of that geometry sits the semantics the driving policy needs:

• Line type — solid, dashed, double, or Botts'-dots — which encodes whether a lane change is legal.
• Color — white versus yellow — which in many regions distinguishes same-direction from opposing traffic.
• Role — ego-left, ego-right, adjacent, or a bus/HOV boundary.
• Physical class — painted line, curb, road edge, or a construction-cone boundary that overrides the paint underneath.

Getting the attribute schema right before labeling starts matters more than raw point precision. A perfectly traced dashed line labeled as "solid" teaches the car not to change lanes where it legally can — a behavior bug that no amount of geometric accuracy repairs.

Drivable area and free space: two different questions

"Where can the car go?" is not one question but two, and good datasets keep them separate. Drivable-area segmentation is a semantic judgment about road structure: this is a lane the ego vehicle may occupy, that is an alternative lane, that is oncoming. The BDD100K driving dataset formalizes exactly this split, distinguishing directly drivable from alternatively drivable areas rather than painting all road surface a single color.

Free-space (or general-obstacle) segmentation asks a more immediate question: which pixels are unobstructed right now, regardless of lane semantics? A parked car, a pothole, or a puddle interrupts free space even where the road is nominally drivable. Cityscapes-style dense pixel labeling — layered polygons across 30-plus urban classes — is the workhorse here, and its "road" class is a standard basis for free-space validation.

The reason to keep them apart is that they fail differently. A drivable-area error is a planning error; a free-space error is a collision. Annotators need explicit rules for the boundary cases — shoulders, painted islands, faded lots — because a mask drawn "roughly right" pushes ambiguity straight into the model.

Road markings, signs and signals

Beyond lane lines, the road surface itself is covered in instructions: arrows, stop bars, crosswalks, speed legends, and turn boxes. These are typically labeled as polygons with a class and, where relevant, a direction. Arrows in particular are worth a dedicated class — CULane treats "arrow" as its own scenario — because a model that confuses a straight arrow for a turn arrow will misread intersection intent.

Traffic signs and lights add a temporal and state dimension that static object detection alone does not capture. A traffic light is not just a box; it is a box with a state (red, amber, green, arrow) and a relevance (does this signal govern the ego lane, or a lane the car is not in?). Signs carry a class and often a value — a speed number, a restriction. The hard part is rarely drawing the box; it is deciding which of five visible lights the ego vehicle must obey, and labeling that relevance consistently across a whole team.

Pedestrians, cyclists and the cost of a miss

Vulnerable road users — pedestrians, cyclists, scooter riders, wheelchair users — are the highest-stakes class in the entire scene, because the cost of a false negative is measured in lives rather than metrics. Large multimodal datasets reflect this: nuScenes annotates 23 object classes with attributes such as pedestrian.moving versus pedestrian.standing, because a person's motion state changes how the planner must treat them.

In practice, VRU annotation is where recall discipline matters most. The instruction we would carry over from our field-robotics QA is simple and uncompromising: when in doubt, label it. A partially visible leg behind a parked van, a cyclist merging from the blind spot, a child at the curb — each must be captured even when the box is uncertain, because the model can only learn caution from examples of caution.

Occlusion: labeling what you cannot fully see

Real roads are cluttered, and objects constantly hide behind one another. An annotation policy has to decide, explicitly, what happens when a car is 70% hidden behind a bus. Guessing individually guarantees an inconsistent dataset. The common approaches, each with trade-offs:

• Amodal boxes — annotate the object's full extent including the hidden part, plus a visibility/occlusion flag. Good for tracking and 3D reasoning; demands consistent estimation of the unseen.
• Modal boxes — annotate only visible pixels. Unambiguous to draw, but fragments a single object across frames as it emerges from occlusion.
• Visibility tiers — datasets like nuScenes bucket occlusion into visibility bands, letting models weight or ignore heavily hidden instances.

There is no universally correct choice; there is only a documented, uniformly applied one. The failure mode we watch for is silent drift — annotators quietly adopting different occlusion habits over a long project — which is exactly what multi-pass review with agreement scoring is built to catch.

Temporal consistency: the label must survive across frames

A single mislabeled frame is a rounding error. A pedestrian whose track ID flickers, or a lane that jumps a foot sideways between consecutive frames, is a systematic fault — and video is where AV data lives. BDD100K ships 100,000 clips of roughly 40 seconds; nuScenes provides 1,000 twenty-second scenes with keyframes annotated at 2 Hz and interpolation between them. Consistency across those frames is a first-class annotation requirement, not a nicety.

Temporal work adds obligations that single-image labeling never faces:

• Stable identity — the same object keeps one track ID for its entire visible life, including through brief full occlusions.
• Smooth geometry — box and lane positions interpolate believably; no teleporting between keyframes.
• State continuity — a traffic light's red→green transition happens on one labeled frame, not oscillating back and forth.

This is where an annotation operation's QA architecture earns its keep. The multi-pass, overlap-based review we run on agritech imagery — independent second passes scored by IoU and consensus agreement — transfers directly: track continuity and lane stability are measurable across annotators, and disagreement is a signal that finds real errors rather than a report you file and forget.

Road-scene annotation is deceptively deep. The tools look like drawing boxes and painting masks, but the actual product is a set of consistent decisions — about attributes, boundaries, occlusion and time — encoded thousands of times without drift. That consistency is a process problem, and it is the problem H2L India has spent years solving in the field. If you are building an AV perception dataset and want a partner who treats QA as infrastructure rather than an afterthought, get in touch — we would be glad to talk through your labeling schema and review pipeline.

Frequently asked

What is the difference between drivable-area and free-space segmentation?

Drivable-area segmentation is a semantic judgment about road structure — which surface is the ego lane, an adjacent lane, or oncoming traffic. Free-space segmentation is a geometric judgment about which pixels are physically unobstructed at that instant, regardless of lane rules. A parked car interrupts free space even on a nominally drivable lane. They fail differently: a drivable-area error is a planning mistake, a free-space error is a potential collision, so most serious datasets label them as separate layers.

How should occlusion be handled in road-scene annotation?

With an explicit, documented policy applied uniformly across the team. The main options are amodal boxes (annotate the full extent including hidden parts plus an occlusion flag), modal boxes (only visible pixels), or visibility tiers that bucket how much of an object is seen. No option is universally correct — what matters is that every annotator uses the same rule, which is enforced through multi-pass review and agreement scoring rather than left to individual judgment.

Why does temporal consistency matter for AV datasets?

Autonomous-driving data is video, so labels must hold up across frames, not just within one image. Objects need stable track IDs through brief occlusions, box and lane geometry must interpolate smoothly without jumping between frames, and signal states must transition cleanly. A flickering track ID or a lane that shifts sideways frame-to-frame is a systematic fault that corrupts training for tracking and motion models, which is why continuity is treated as a measurable QA requirement.