How Gritt works: Practical intelligence for building the real world

As we wrote, the world needs far more infrastructure than the construction industry can deliver. Construction productivity has fallen for sixty years and the workforce is declining at every level, from the crews in the field to the supervisors planning their work. However, a significant portion of construction involves repeatable, well-defined tasks. A typical energy project alone involves installing 1-2 million solar panels over 10,000+ acres. This combination of high-volume, repeatable tasks in difficult environments is exactly the kind of dull, dirty and dangerous work that robots should be doing.

Gritt is building the intelligence platform that generalizes across tasks, environments and spatial reasoning in construction. We’re tackling dexterous manipulation and navigation in these challenging, unstructured environments. This post outlines our pragmatic approach that has already scaled to commercial deployments in the real world, including 900kg robots manipulating 50kg payloads at high speed. Future posts will dive deeper into various pieces of our tech stack.

Challenges with automating construction

There are barely any robots helping build infrastructure today. Construction environments are unstructured or semi-structured at best, and present a host of challenges -

• Exposure to outdoor conditions
• Variable lighting that can lead to significant appearance changes (e.g. shading, reflection, glare)
• Navigating uneven and unsteady terrain
• Dynamic environments that keep changing as things move and get built
• Variations in temperature, weather and season

Construction demands performance and throughput, where tolerances are often measured in millimeters and delays are extremely costly. Mistakes can damage expensive components, and systems must operate robustly around other workers and equipment.

As for a dataset for construction, one simply doesn’t exist.

Unlocking the next frontier of physical AI

Gritt is solving challenging problems around assembly, pick-and-place and transportation in these dynamic, unstructured environments. There is often a tradeoff between generalization and performance/throughput in physical AI, but as we’ve outlined, construction tasks require both. Our approach prioritizes a deployment-first mindset, access to valuable real-world data, learning scalable representations and running on widely available hardware.

Deployment-first

When it comes to applications like construction, robots can’t learn (and earn) enough in the lab. However, deploying to live jobsites requires a minimum bar of performance and robustness that typically eludes AI models in the first pass. We’ve developed uncertainty-aware observation systems that validate actions before execution (e.g. “is this component being placed incorrectly?”) and prevent serious failures. We deploy new models in “shadow-mode” for last-mile online validation before switching over. We’ve also implemented takeover and recovery mechanisms to allow the process to continue while providing training examples from failures. Wrapping our models with these guardrails that loosen over time fuels our continuous deployment-improvement cycle. Gritt has installed tens of thousands of solar panels on construction sites across the country, clocking 3-4x productivity boost with zero breakages.

Access to real-world data

There are no existing datasets for construction. Unlike applications like home and industrial/warehouse robotics, it can be impossible to replicate these jobsites in the lab for training physical AI. Long-tail events can also be especially hard to simulate from scratch. The only way to access data, therefore, is to work on actual sites. Our deployment-first approach has led us to successfully operate across geographies and environments (deserts, plains, hills, marshes, rain, snow, shine, wind). This access has allowed us to assemble thousands of hours of a continually-expanding, one-of-a-kind dataset spanning manipulation and navigation tasks that feeds into training and eval.

Real-to-sim-to-real

We make extensive use of simulations for training and validation. Instead of simply developing unconstrained simulations from scratch, we’ve developed a generative pipeline that creates sims from real-world data. This pipeline ingests images/videos and generates rich descriptions of the scene that are semantically accurate and encode position and viewpoint information. Using a combination of components with known CAD models and reconstructed objects from the original data, these descriptions are used to generate simulation episodes with domain randomization. For instance, when a robot encounters a rare, anomalous situation and requires a takeover, we use this pipeline to generate training data and validate on the real episode.

Learning tasks from a scalable representation

We believe that learning good representations is a scalable and efficient way to generalize across tasks and environments. The Gritt Foundation Model (GFM) encodes the objects, environments, transitions and interactions on construction jobsites. We pre-train this backbone on both labelled and unlabelled image/video data.

We use GFM to train task-specific models. Depending on the application, we train either policies or intermediate tasks like segmentation and 6-dof pose estimation. GFM’s weights are also updated when training these models, enabling continual improvement across tasks. Building on a generalized representation lets us rapidly train models for deployment with fewer task-specific episodes. It also enables robust operation, where a model’s output can be validated by another’s. For instance, predicting poses for objects in the environment lets us independently validate a policy’s output and gracefully ask for help if an action might cause a collision. Such “training wheels” enable rapid deployment and data-collection during the initial march of 9s while providing a surrogate signal to improve the base representation model.

Learning a scalable representation model also enables robust generalization across construction environments. These environments have many similar elements, from workers wearing hard-hats to common tools and equipment. Learning to use a power-tool in one environment is a skill that is transferable to another. Successful deployment doesn’t just depend on the immediate task being performed correctly, but also on the environmental awareness that GFM encodes. We’ve leveraged GFM, for instance, to extend capabilities from initial applications in solar to other construction tasks like rebar-tying and building walls with cinderblocks.

Leveraging mature hardware

Gritt is embodiment-agnostic, allowing our models to run on mature, off-the-shelf equipment like industrial robot arms that simply attach to construction vehicles that are already present on site.

Looking ahead - the operational layer

Workforce shortages aren’t just restricted to frontline workers building infrastructure, but also to the project and construction managers overseeing these jobsites. Gritt’s intelligence is already developing a unique spatial understanding of how work is sequenced and executed on these projects. For instance, factors like incoming weather and material availability affect how our robots are deployed, and these operational parameters are captured by our platform. As we continue to scale across physical tasks, we will also complete the ecosystem by making Gritt a decision-making layer that helps plan, verify and manage work.

We’re always looking for people who’re excited about putting AI and robots to work building the world’s infrastructure.