The Safety Case Lives in the Tail

This is a supporting blog post for our ideas paper titled “Auditing Quality Filters for Long-Tail Human Data Curation”. The paper is presented as part of the Long-term Perception Workshop (link) at IROS 2026

A worker crouched or kneeling beside a mounted solar panel is doing routine construction work. For a robot working in the same environment, that worker may be an unusual example, as they wear heavy PPE, are close to the ground, and are partly hidden by steelwork. In short, they look very different from a standing pedestrian. Whether the robot detects them accurately matters for site safety, regardless of how often such scenarios show up in the training set.

However, this difficulty begins before training, as automatic annotation pipelines process a lot of data and look at TBs of data per day. A pipeline to label crouched people would use a detection algorithm and then classify these awkward poses. Because these estimates are imperfect, annotation pipelines often discard examples with low detection confidence or uncertain joint positions. This improves the reliability of the retained pose labels, and the top-line precision and recall may look good, but it can also remove critical images that would make testing more robust.

We call this the selection-scarcity confound.

A worker's bounding box may be usable even when their knees and hips cannot be reconstructed reliably. If acceptance depends on the quality of the whole reconstruction, pose-estimation requirements end up filtering which people a detector learns to identify.

To illustrate this, we run a detection pipeline and then run a pose estimator to classify detections. Our pipeline, when run on 4 months of site data, produced a library of 3,762 person instances, grouped into 14 postures after quality filtering and removal of near-duplicates. Only 84 instances (about 2.2%) fell into the kneeling, squatting, or deeply bent-over groups, which we call low poses. That proportion alone tells us little about how common these poses are on a construction site. It reflects the images collected, the people detected, the reconstructions accepted, and the repeated observations removed. Different camera positions and image-selection practices could explain some of this variation.

We therefore turned to four alternative filters: how wide the person's box is compared to its height (box shape), how upright the body looks relative to its size (vertical span), pose grouping using the scene (not the body) to tell which way is up (scene-based alignment), and an image classifier that judges posture from the picture alone (image appearance). These checks support the concern that filtering contributes to the observed scarcity, but they do not establish its magnitude.

These checks aren't perfect. A wide box might mean someone is kneeling, or just that part of their body is hidden behind equipment. There's also a subtler problem. To figure out which way is "up" in each image, our pipeline assumes people stand upright and uses the line from their ankles to their hips. That works for standing workers, but a kneeling or bent-over worker makes a poor reference. It's like judging whether a photo is tilted by looking at someone doing a handstand. When we compared this body-based "up" with one estimated from the surrounding scene, the two disagreed far more in images with low poses, sometimes pointing in nearly opposite directions.

One immediate change is to judge person boxes and pose labels separately. If the body reconstruction fails, that should rule out the pose label, not throw away a perfectly good example of a person.

The analysis also had a useful side effect. Mining these poses gave us a library of realistic working postures, like kneeling at a motor or bending over a panel, that we could bring into simulation. When we added simulated workers in these poses to our training data, detection recall improved by 1.6 percentage points on real site footage for low poses. The study started as an audit of what we were losing, and it ended up showing us what to add.

For a robot working beside construction crews, the frequency of a pose in a curated dataset is a poor measure of how much attention it deserves. Kneeling and bending are ordinary work activities, even when the annotation pipeline makes them look rare. A credible evaluation must account for workers missing from the retained data, including those rejected during annotation.

If you're excited to solve challenging data problems like this one, come join us at Gritt! Solving real problems requires deep thinking and fundamental research into questions like these.