“Some people say give the customers what they want, but that's not my approach. Our job is to figure out what they're going to want before they do. I think Henry Ford once said, 'If I'd asked customers what they wanted, they would've told me a faster horse.'”

This Steve Jobs quote speaks to a fundamental tension in self-driving. The technology has yet to arrive in its truest, fullest form. Self-driving represents the mother of all market pushes, since the average person is, at best, nervous about the idea of riding shotgun to software.

The story only gets stranger. There’s no full-feature version of this product, despite decades of development. You read that right: There aren’t any fully self-driving cars out there. Nada. Zilch.

So why are billions of dollars and the world’s brightest minds chasing something that doesn’t exist?

Well, two big reasons: 1) There could be a trillion-dollar payday. 2) An AI system with superhuman driving abilities could make our roads much safer.

In the industry’s defense, this is uncharted territory. Building a driverless car that can safely and capably go anywhere, at any time, in any weather, is staggeringly difficult.

Let’s take a road trip. This guide will start with the terms you need to know about autonomous vehicles (AVs). We’ll look at where AVs have been, how the tech stack works, and where they’re going. You may have come in asking, “Are we there yet?,” and with any luck, you’ll leave with a clearer sense of the ETA.

I. Zero to one, and then, five

“Self-driving” describes a vehicle that can sense its environment and move safely without any human input or guidance. Automated driving systems, by comparison, can “perform all driving tasks and monitor the driving environment—essentially, do all the driving—in certain circumstances,” according to the National Highway Traffic Safety Administration.

In 2014, the Society for Automotive Engineers published a taxonomy for automated and autonomous driving. SAE’s six levels have become the gold standard for industry benchmarking, but for the average person, they read like wonk-speak.

We’ll do our best to wade through the confusion and simplify things. And as you're reading this, keep in mind what we wrote in our AI Guide, “Autonomy is a machine’s ability to do a task on its own. But it’s not a binary—it’s a spectrum. A system becomes more autonomous as it tackles more complex tasks in less-structured environments.” Without further ado, SAE’s six levels:

Do you see how this gets messy? Even though standardization for AVs exists, the sliding scale is not straightforward. Marketers have seized on the ambiguity, pushing terms like Level 2+ or Autopilot for their advanced driver-assistance systems (ADAS).

Offerings vary carmaker by carmaker, but for L1–L2, you’re still in control and must always stay attentive when using ADAS systems.

• What pairs nicely with L2 ADAS? Driver-monitoring systems. Carmakers with advanced ADAS tend to use these systems, such as a driver-facing camera, to ensure you are remaining attentive.
• L2 is increasingly common. According to Canalys, passenger cars with L2 driving functionality represented 30% of all new US cars sold in Q4 2020. Globally, automakers sold 3.5 million cars with L2 features over the same stretch, a 91% year over year jump.

Finally, let’s make one thing clear: There are no driverless passenger cars that you can go buy, despite what most US adults think. In fact, no SAE L3 or above vehicles in the US, either. In 2020, National Transportation Safety Board Chairman Robert L. Sumwalt summed it up very clearly:

II. The AV annals, in fits and starts

As you scroll through, you may be surprised to learn that GM envisioned a one-off idea for automated driving 82 years ago. And also by the progress that a handful of technologists made in the late 20th century. What won’t surprise are the recent headlines, filled with ambitious deployment timetables, accelerating M&A activity, and equally aggressive fundraising levels.

III. The tech trappings

When it comes to the technology, AV builders all agree on one thing: It will be very valuable in a fully working form. But that’s where the consensus on the technology starts and stops.

From hardware component selection to algorithm fine-tuning, there’s a near-infinite amount of technological permutations for how to build the dang things. And there’s plenty of disagreement over what you do—and don’t—need to make self-driving work at scale.

So, what advanced technologies do you need? Here goes nothing...

Body: We’re easing into this. To make an autonomous vehicle, you will first need a car. You can start from the ground up with a purpose-built vehicle. You could retrofit an original equipment manufacturer (OEM)’s existing production vehicle. Or you could aim for some combination of the two. In any event, you’ll need special drivetrains, brakes, acceleration, steering, and electronics.

Eyes: An AV needs to “see.” Your typical perception stack involves cameras, radars, and light detection and ranging sensors. The latter, more commonly known as lidar, uses light in the form of a pulsed laser to measure ranges and map environments.

• There’s a new type of lidar that could be more accurate: Frequency modulated continuous wave (FMCW). It’s a mouthful, but essentially, FMCW lidar emits a continuous stream of light and—this is what’s new—regularly changes the frequency of that light.
• As of publication, there are at least five publicly traded lidar companies: Innoviz, Aeva, Ouster, Velodyne, and Luminar.

Do you need lidar to succeed? Hope you like beef. Most startups and OEMs would say yes. Elon Musk, on the other hand, has famously called lidar a “fool’s errand” and said anyone relying on it is “doomed.” His thinking: It’s too expensive.

Team lidar disagrees, saying the technology catches environmental details that cameras and radars (what Tesla uses) might miss. Plus, they say, the cost has dropped by orders of magnitude already:

• Velodyne, a SPAC’d lidar maker, used to sell its rooftop spinning lidar set for $75,000. It now hopes to make an auto-grade unit with a sub-$500 price tag.
• Peter Stern, CEO of lidar startup Voyant Photonics, gave us a more straightforward assessment. Fifteen years ago, lidar sensors cost ~$40,000 apiece, and now, it’s more like $4,000.

Brain: You need an astounding amount of software smarts to make a self-driving system that can perceive, predict, and plan. You must generate, annotate, clean, and continuously update data so that it stays machine-readable and relevant. Never anger the machine.

• To make sense of what the eyes see, you’ll need sophisticated computer vision. You’ll also need sensor fusion (the combination of sensory data from different sources) to meld all the inputs and create a high-fidelity model of the car’s surroundings.
• You’ll need a load of on-board compute, too. Future AVs could run on up to 500 million lines of code and produce terabytes of data daily.

Internal compass: AVs need to know their exact location for path planning and prediction. You could rely on the Global Positioning System, but since GPS can be off by a meter or more, you’ll probably want top opt for the Global Navigation Satellite System (GNSS). This is much more precise, on the order of centimeters.

• Companies also rely on HD maps, which store highly granular 3D data about roadways around the world. “Memory” of recent maps can be stored locally on the vehicle and used if a vehicle does not have LTE or 5G connection. The maps can be continuously updated, thanks to over-the-air software updates and cloud-based databases. A nice segue….

Out-of-body: AVs will be Chatty Cathys, with vehicle-to-vehicle, -infrastructure, and even -everything communications. Cloud and in-vehicle connectivity will be essential.

• Outside the vehicle, in server centers, you’ll need software systems to continuously train your deep learning models and update your data sets.

And, depending on whom you ask, you’ll want simulation software. Many AV developers run their driver software through millions of miles of virtual road testing. “Virtual testing has always been a pillar of our development process and a way to make rapid progress at scale,” Aurora CEO Chris Urmson has told the Brew.

And Waymo Distinguished Scientist Drago Anguelov told us that “simulation at scale” is the biggest development obstacle for AI over the next five years.

IV. Industries

We’re beating a dead robo-horse at this point, but productizing the prototypical AV is its own industry. There are many players and few actual commercialized products. But when it arrives, the technology has platform-level potential. It will disrupt plenty of traditional industries in a big way.

V. Politicos, the public, and other key players

There are a bunch of critical questions that we haven’t even touched on yet. In this section, we’ll consider them through the lenses of key constituencies involved with self-driving.

Human drivers: To back up for a sec—Do we even need AVs? The kneejerk response from industry insiders is that yes, we do, to save us from ourselves. Safety officials, who have a different agenda that Detroit or Silicon Valley, would also tell you self-driving is about safety.

• Human error causes 94% of all crashes.
• The four D’s—distraction, drowsiness, drunkenness, and driver error—can provoke dangerous situations on any road.
• Last year, there were 16,650 motor vehicle crash fatalities in the US alone.

In March 2018, TV host and rally race driver Alex Roy published the Human Driving Manifesto and formed the Human Driving Association. The tagline: “When human driving is outlawed, only outlaws will drive.”

As for you—let’s say you like driving. Really like it. Then, you can take a deep breath. Few AV proponents say they think human driving should be outlawed—and even fewer governments agree.

Employed drivers: “We are in the third inning of the greatest economic and technological transformation of this country,” Andrew Yang, then on the presidential campaign trail, told us in 2019. He used the potential automation of the trucking industry as an example. It’s a valid employment concern, especially since the “driver” occupation goes far beyond long-haul trucks.

• It’s more of a chronic than acute issue, though—one that will require policy solutions, industry gameplans, and (likely) large-scale retraining/upskilling programs. It’s also not quite as imminent as Yang suggested, as we wrote earlier.

Governments: In the US, Washington sets the rules of the road but regulation tends to fall to the states. We’ll talk more about states’ roles in a sec. The National Transportation Safety Board and National Highway Traffic Safety Administration will be highly involved in response to any crashes involving automated or autonomous vehicles.

Elsewhere, China recently passed a new law which would allow the testing of autonomous vehicles on highways for the first time. Local authorities around China are going even further, enacting AV-friendly policies in cities and in some cases, constructing autonomous-only lanes on new highways. Japan recently passed a law to allow Level 3 cars on highways (a world first).

Laboratories of democracy: Across the US, a smorgasbord of different AV laws and regulations have emerged state by state. Nevada, Florida, Texas, and California, to name a few, have passed AV legislation. California also requires companies to report autonomous miles logged and “disengagements,” which is when a system hands back control to a safety driver.

29 companies with AV testing permits in California reported test drives in the state from December 2019 to November 2020. They drove 1.96 million miles with 3,695 disengagements. The year before, companies drove 2.88 million miles on California roads with 9,338 disengagements.

Execs from Cruise, Waymo, Aurora, and other companies say disengagements are a bad metric for judging self-driving systems’ performance. It’s true, but since these companies guard internal data as proprietary trade secrets, there’s not much else for us to go off.

Arizona’s government is the most receptive to advanced AV testing. That tolerance has its limits—Arizona Governor Doug Ducey suspended Uber’s autonomous testing permits after one of its cars fatally struck a pedestrian on March 18, 2018. A few months later, The Arizona Republic reported that Chandler police had documented at least 21 incidents of Arizonans threatening Waymo vans with rocks, knives, aggressive driving maneuvers, and once, a gun.

Cities: One common criticism of AVs, especially among urban planners, is that they’re not necessary. Why unleash more cars on roads when you could just invest more in public transit? If ride-hail and car-sharing AV networks are commercialized first—and they replace vehicles with lower utilization rates—it could help with congestion.

• But if way more commuters are traveling to and from a city by themselves in personally owned AVs, it’s difficult to see how that would help with gridlock.
• One compromise we could eventually see: autonomous shuttles run by cities, or in public-private partnerships.

Consumer advocacy groups: Remember SAE’s six levels? When it comes to automated and autonomous driving, industry marketing materials can distort consumers’ expectations of a technology’s capability.

• A German court recently ruled that Tesla can’t use “Autopilot” or "full potential for autonomous driving” in advertising language for vehicle sales in Germany.
• AAA recently found that consumers can encounter as many as 20 names for a single ADAS feature.
• The Department of Transportation, Consumer Reports, and other groups have called for standardized names for new ADAS features.

Lawyers, lots of ‘em: Who will be legally responsible in the event of a crash with a self-driving vehicle? This remains a largely unsolved question, but not a hypothetical. Waymo’s vans get into fender benders too. Out of 6.1 million miles of self-driving in 2019 and Q1–Q3 2020, Waymo’s cars had 18 crashes and 29 near-miss collisions.

VI. Pulse check

From March 12–15, we partnered with The Harris Poll and surveyed a nationally representative group of 1,052 US adults about AVs. Most Americans (62%) indicated that they thought a fully self-driving vehicle is available for consumers to purchase somewhere in the world. As you know from reading this far in the guide, that is not true.

As we expected, younger American adults were more likely to say they’d be willing to try out the technology. Gen Z (71%) and Millennials (62%) were almost twice as likely to say they’d ride as a passenger in a self-driving vehicle as Baby Boomers (46%). Nearly half of Gen Z respondents said they’d use the time in an AV to eat or drink.

Parents also stood out as a demographic highly receptive to the technology. 59% of parents say they would feel at least somewhat safe riding in a passenger seat (vs. 48% for all adults).

Safety: 48% of Americans said they’d feel “at least somewhat safe” in the passenger seat of a self-driving vehicle, but that number jumps to 59% if they were in the driver’s seat. 16% say they’d feel very safe riding shotgun. And, again, younger respondents were much more likely to say they'd feel at least somewhat safe riding as passengers.

Willingness to ride: Trust and safety are the largest reason that Americans wouldn’t want to ride in a self-driving car. “I live driving myself” is the next most prominent reason (cc: Human Driving Association).

Other interesting data points:

• 75% said they’d like more clarification on who’s legally responsible in the event of an AV-involved accident. Remember the lawyers we were just mentioning?
• 41% of Americans said they’d prefer to buy a self-driving vehicle from Tesla, over double the share who’d prefer to buy from a legacy automaker. Of those who said they’d be somewhat likely to buy a self-driving car, 52% would prefer to buy from Tesla.
• Of those Americans willing to ride in a self-driving vehicle, most aren’t willing to pay more than they do now (for either personally owned vehicles or ride-hail services).

And here's what US adults who are somewhat or very likely to ride in a self-driving car said they'd be most excited to do with their newfound time:

VI. The Road Ahead (Sorry)

This nascent-yet-powerful autonomous vehicle industrial complex won’t let up. In 2019, AV startups captured more than 10% of all private AI investment. According to one early 2020 tally, 30 companies had spent $16 billion on self-driving R&D over a few years.

More than one exec has framed the challenge by comparing it to spaceflight and concluding that hey, actually, exploring the final frontier is easier? “In space, you have just got to avoid the other rocks,” Nissan CTO Maarten Sierhuis has said. “But on Earth, you have got to avoid a lot of other human beings.”

We won’t make bold self-driving predictions. An iron law is that when you predict self-driving’s imminent arrival, you blow past your deadline. But we’ll conservatively wager that the arrival of robotaxi- and goods delivery fleets is inevitable and bound to happen in our lifetimes.

When/if that happens, the R&D spend and private investment could seem less silly. Self-driving is unlikely to be winner-take-all, but it could very well be few-take-most. Bullish analysts peg the economic value generation of AVs on the order of trillions of dollars within the decade.

There’s also second-wave effects—if you can work rather than drive, it boosts productivity. You could also nap, boosting productivity later in the day...or just stare out the window, as many adults indicated they are excited to do. Cottage industries will be built on in-AV services, from advertising to entertainment.

These future unlocks are how today’s players rationalize the burning holes in their balance sheet. From Detroit to Silicon Valley to Bavaria to Beijing, all sorts of software and car-making companies are all-in.

Last year, the rumor mill in Cupertino went into overdrive. Apple was planning to release a self-driving EV. A former engineer on the project told us, “Apple will own the design, experience, and technology—the autonomy. That’s the killer combination.”

Remember the Steve Jobs quote from the beginning? The company he cofounded is converging with the industry Henry pioneered. That alone might be the best articulation of the potential significance of AVs.