Decoding Waymo: Google’s Autonomous Ride-Hailing Service

June 11, 2025.

Over 10 million autonomous rides so far, and counting, that's the story of Waymo.

 You’ve probably heard the name—it’s the self-driving arm spun out of Google, and lately, it’s been making waves. After launching its autonomous ride-hailing service in Phoenix, LA, and San Francisco last year, Waymo is now expanding into Atlanta, D.C., Austin, Las Vegas, and even Tokyo, with plans to roll out in 10 new cities this year.

You might ask, what’s fueling this surge? Well, it's the years of research that's finally reaching a tipping point and a fresh $5.6 billion in funding it secured last year. But, the real story isn’t the capital it has raised or the miles it has covered, it’s the journey behind it, and more importantly, the approach adopted by Waymo. 

We will first cover the evolution of Waymo and then the novel approach it adopted to get to this point. 

The Beginning: From Toyota Prius to Waymo One

The origins of Project Waymo can be traced back to 2009, the same year Steve Jobs unveiled the iPhone 3GS.  Around this time, the founders of Google set out to tackle a bold challenge: developing an autonomous system capable of driving a 100-mile route. To solve this challenge, they retrofitted a Toyota Prius with a 1.5-ton LIDAR mounted on top.  While it’s unclear if the LIDAR outweighed the car itself—on a lighter note, this was an era when LIDAR systems were anything but compact, often requiring a suitcase just to transport from one place to another. Still, this early prototype developed by Google demonstrated that fully autonomous driving was possible. 

While the Toyota Prius allowed the team to demonstrate early self-driving capabilities, retrofitting sensors onto an existing vehicle quickly proved cumbersome. They realized that building a truly autonomous vehicle would require designing one from the ground up. By 2015, Waymo made a pivotal shift towards a custom-built platform—leading to the development of its first fully self-driving prototype, known as Firefly.

This tiny car, equipped with a large LIDAR system and other advanced sensors was unique in a way because it was amongst the first generation of autonomous vehicles ever developed without a steering wheel or a dedicated driver seat. With its cutting-edge design, Firefly became the first vehicle to complete a fully autonomous ride on public roads in the U.S.

While Firefly helped Google experiment with its sensor suite, the capital and time required to produce a custom car and the sensor suite was humongous. Recognizing the need for scalability and resource efficiency, Waymo decided to shift back to using existing vehicles—this time with a much more compact and efficient sensor suite. Their approach? Strategic partnerships with automakers like Jaguar and Hyundai.

Waymo’s Transition to Jaguar & Hyundai Platform

Waymo now partners with leading automakers like Jaguar and Hyundai, integrating its cutting-edge technology into fully autonomous vehicles. These cars are equipped with an advanced array of sensors and software suite, enabling them to handle a wide range of unpredictable environments. We're talking 29 high-definition cameras, multiple LiDARs, and six radars strategically placed around the vehicle—powering a fully autonomous ride-hailing service.

At first glance, Waymo sounds like Uber without a driver. That’s technically true—the “driver” here is the Waymo Driver System, an advanced blend of hardware and software that pilots the car. But then you might ask, doesn’t Tesla do self-driving too? And so do others? What makes Waymo’s approach novel?

That’s where it gets interesting.

Waymo’s Novel Approach – Introduction

There are many ways to design a self-driving system. As we’ve explored in past essays, Tesla leans heavily on cameras, AI, and vision systems, stripping away sensors like LiDAR because of cost and the performance degradation that comes with varying light conditions. Waymo, on the other hand, takes a mission-critical approach—embracing an array of sensors, including LiDAR, cameras, and radar, all mounted in a multi-layer redundant configuration inside and outside the vehicle to maximize reliability and safety.

You might wonder: does simply packing in more sensors make Waymo’s approach novel? Not quite. You see hardware configuration is just one half of the equation, the other half is the software. The software piece developed by Waymo is a combination of a bunch of home-grown systems that have evolved over the years. What do I mean by that? 

Remember when Google launched Google Maps & Street View features?  Yes, we are talking about these features that allowed us to visualize 3D versions of our planet. That same system later evolved into the localization mechanism that allows Waymo’s cars to continuously track their position in the real world with centimeter-level accuracy. Voila!

And that’s not all.

If you’ve used modern AI models like ChatGPT or Gemini, you’ve seen how they rely on modern AI architectures like transformers. These architectures that form the core of modern AI models were developed by researchers at Google and released in a paper in 2017 titled "Attention Is All You Need". Waymo's perception stack—the system that allows its cars to “see” and understand the world—is a direct byproduct of this research. 

Now, before we do a deep-dive on Waymo's self-driving software system, it’s helpful to understand the standardized architecture that underpins most autonomous vehicles today. Leading players in the industry typically organize their autonomy stack into four key pillars: perception, localization, path planning, and motion control. We’ve explored this framework in detail in our blog Automotive Is Now Autonomous—so if you’re new to self-driving cars, that’s a great place to start. 

While this four-pillar structure forms the foundation of most autonomous vehicular systems today, Waymo takes it a step further with a distinct, mission-critical philosophy. Instead of merely optimizing each layer of the stack, Waymo builds for safety and reliability from the ground up.

Waymo's Mission-Critical Self-Driving System - 2 Pillars

Waymo’s approach to autonomy rests on two core pillars.

First, there’s the multi-redundant sensor array, also referred to as the mission-critical sensor configuration. Simply put, Waymo equips the car with a wide array of sensors: LiDAR, cameras, and radar—arranged in multiple redundant layers.  If one fails, others are ready to take over — and not just one backup, but often several. You might ask, why so many sensors? Because each type has its own strengths and blind spots. That’s why Waymo has embraced a mission-critical sensor configuration. That’s the first pillar.

Second, there’s the software—the brain behind it that fuses all this sensor data, interprets the environment and makes real-time driving decisions. But before we dive into the software, let’s first take a closer look at the sensor suite. 

Pillar I - Waymo’s Mission-Critical Sensor Suite

Waymo understands that sensors can fail at any time, so each sensor is placed in a redundant configuration around the vehicle. This means if one sensor fails, others can compensate for it, ensuring continued reliability. Beyond the configuration, the types of sensors used by Waymo are equally novel. Waymo recognizes that every sensor—whether it’s a camera, LiDAR, radar, or ultrasonic—has its limitations. Therefore, it doesn’t rely on just one or two types but employs an array of complementary sensors.

Each Waymo vehicle typically has around 29 cameras offering a 360-degree view of its environment, 5 LiDAR units delivering high-resolution 3D imaging, 6 radar sensors, and a ring of ultrasonic sensors to assist with close-range detection. You might ask why use so many sensors? 

Well,  each of those sensors has its own limitations. For example, while Waymo uses a set of 29 cameras, these same cameras are less effective at night, as they struggle in low-light conditions. To overcome this limitation, Waymo integrates LIDAR, which provides high-resolution imaging in both day and night conditions. 

Now, you might ask, if LiDAR and cameras can solve all our problems, why do we need radar? Well LiDAR has two limitations. First, it doesn’t perform well in bright sunlight. Hold a LiDAR near a window flooded with sunlight and try mapping a room—you’ll likely see a ton of blind spots on your map. Now imagine that same sensor mounted on a vehicle, exposed to direct sunlight on a city street. That's not a rarity for these vehicles that are designed to operate in the urban environments 24*7, that's a reality. 

In addition to that, LIDAR offers limited visibility in fog, rain, and snow. This is where radar comes into play. It offers reliable performance in such adverse weather conditions, though it lacks the range and the precision provided by LIDAR and cameras. Hence, Waymo combines data from all these sensors to mitigate blind spots and ensure comprehensive coverage.

You might also ask, what happens during rain, snow, or dust storms? Do the sensors get blocked? Not quite. To address this challenge, Waymo has equipped its sensors with an automated cleaning system—dedicated, custom-built sensor wipers. These ensure that sensors remain clean and operational at all times. Safety is at the core of Waymo’s system design philosophy.

And this brings us to the second pillar that makes the Waymo system truly novel: its self-driving software system. While pillar 1 i.e. the mission-critical hardware configuration provides the eyes and ears to Waymo vehicles, it’s the pillar 2 i.e. software that acts as the brain for these vehicles, fusing raw sensor data, interpreting the world, making decisions, and executing actions with precision. Without this software stack, the sensors alone would be little more than silent observers.

Let’s take a closer look at what makes Waymo’s software system novel.

Pillar II - Waymo’s Mission-Critical Self-Driving Software System

The software system developed by Waymo to power its perception, localization, path planning and motion control modules employs custom-built artificial intelligence (AI) models developed by in-house teams.  You might say—so what? Doesn’t everyone use AI models today? That’s true. But what sets Waymo apart is not just the use of AI models, it’s how these models are trained and deployed. It's the approach adopted by Waymo to train these models that differs significantly from others.

What do I mean by that? Let's take a look at the perception system. 

Waymo's perception system includes a sensor suite consisting of 29 cameras, 5 LiDAR units, radar, and ultrasonic sensors. The first thing this system does is simple yet critical—collect and calibrate the raw data reliably from all these sensors in real time. You might ask, why not use raw data as-is? Why calibrate this data?

Sensor Data Collection & Pre-processing

Well, raw data alone isn’t enough. Sensors aren’t perfect—they capture a lot of noise, meaning bad or irrelevant data. That’s why we need a system that can separate signal from noise. This is where the next sub-system comes into play: data preprocessing. Here, Waymo’s perception software runs that raw sensor data through custom-trained artificial neural networks i.e. convolutional neural networks (CNNs), which filter out noise and boost the signal. Now, by just removing the noise, we get clean and reliable data from each sensor. But even this good data stream presents a challenge. 

If you look at the sensors—29 cameras, multiple LiDARs, and radars, they are mounted at different positions, angles, and orientations across the vehicle. If you try to interpret all 29 camera feeds in their raw form, you’ll quickly get overwhelmed. It’s disorienting and chaotic—there’s no clear view.

In order for Waymo Vehicle to develop a coherent understanding of its surroundings it needs a calibrated and uniformly oriented visual of the scene. That's where the calibration and alignment module comes into play. It uses a series of in-house trained CNNs to orient this stream of data. The result? You get a clean, well-oriented, 3D point cloud from the LiDAR, sharper visual data from the cameras, and a refined radar map—all ready for the next stage.

Now, we have clean and uniformly oriented data streams coming from different sensors. But we still don’t have a unified view. We haven’t stitched all our insights together into a single, coherent view yet. That’s the job of Waymo’s custom sensor fusion algorithm.

Custom Sensor Fusion Algorithm

Here, Waymo uses custom-trained AI models to fuse the sensor data streams in real-time. This sensor fusion mechanism allows the Waymo vehicle to develop a coherent and detailed 3D view of its environment. This detailed 3D map provides the current state of all the agents present in the scene—pedestrians, cars, stop signs, and more.

But here’s the key challenge: knowing the current state of all agents in the scene—where they are right now—isn’t enough to safely make decisions in the real world. We also need to understand: how fast are they moving? where will they go next? and how does that impact the vehicle’s own decisions?

This is where the agent detection and trajectory tracking module comes in. 

Agent Detection & Tracking Module

The mission of the agent detection and tracking module is twofold: first, to classify the agents present in the scene—people, cars, cyclists, and more—and second, to predict how these agents are likely to move over time.

Waymo uses custom-trained algorithms here: convolutional neural networks (CNNs) to classify the different types of agents present in the scene and recurrent neural networks (RNNs) to predict their future states. 

At this stage, the system now knows who is in the environment and where they’re likely headed. Basically, the Waymo vehicle is operating at an agent level—analyzing one object at a time. But that’s still not enough. Why? Because understanding a complex, real-world environment requires more than just tracking individual agents. The system must also understand how these agents interact—how a pedestrian might hesitate at a crosswalk when a car approaches, or how one driver’s behavior might influence another’s.

The Waymo vehicle must understand inter-agent dynamics. That’s the job of the next module: scene interpretation & decision-making. In simple words, we are heading towards action from perception.

Scene Interpretation & Decision-Making Module 

Here, custom-developed AI models analyze the inter-agent movements across the scene, enabling Waymo to predict the future state of the scene as whole rather than just individual agents. Now the challenge is that there could be many future states for the scene as a whole because the combination of inter-agent dynamics is virtually infinite. What do I mean by that?

Well, imagine the Waymo vehicle standing at a busy intersection and waiting for a safe window to turn left. The possible moves the cars coming from the right and left can make are endless. The same goes for the pedestrians near the crosswalk. Now factor in variables like weather, lighting, and visibility conditions.  So how does Waymo simulate these combinations of future states of all the agents in the scene and make decisions in real-time?

That's where the decision-making module comes into play. 

This system, again powered by artificial neural networks, enables the Waymo vehicle to make multiple permutations and  combinations of scene data and inter-agent dynamics provided by the scene interpretation module and predict, with high accuracy, the most likely evolution of the scene in the next moment. Based on this predicted future state, the system decides its next move.

And this, truly, is just the tip of the iceberg when it comes to Waymo’s software system. There’s much more happening beneath the surface—in the perception stack, localization, path planning, and motion control modules. 

We cover this system in great detail in our Self-Driving Cars course. In this course, you'll learn how to build a self-driving system from the ground up. You'll gain a deep understanding of the autonomous stack developed by Tesla, Waymo, NVIDIA, and Zoox. Our courses have helped over 80,000 students from 150 countries kick-start their careers in these emerging fields.

If you're serious about entering the self-driving car industry, this course is for you. 

Cheers,

Prathamesh

Disclaimer: This blog is for educational purposes only and does not constitute financial, business, or legal advice. The experiences shared are based on past events. All opinions expressed are those of the author and do not represent the views of any mentioned companies. Readers are solely responsible for conducting their own due diligence and should seek professional legal or financial advice tailored to their specific circumstances. The author and publisher make no representations or warranties regarding the accuracy of the content and expressly disclaim any liability for decisions made or actions taken based on this blog.