It’s becoming something of a truism that AI is only as good as the data behind it. AI algorithms, after all, are products of the reams of data on which they’re trained.
This makes for a high-stakes situation in the world of autonomous vehicles, where it’s common for simulations based on data to play a crucial role in the testing process. That’s why AV trucking company Waabi sought to solve what founder and CEO Raquel Urtasun saw as a problem in the industry: Everyone’s using simulators, but there’s no universal framework for assessing how realistic these simulations actually are.
“It doesn’t matter how much you test unless that simulation really reflects reality,” Urtasun told Tech Brew. “If you’re testing something that is not realistic, even if you solve all the cases, who cares, if it’s not correlated with reality?”
So Waabi set out to establish a way to verify the accuracy of its simulator, and then publicly released its realism score (99.7%)—and is calling on industry peers to do the same.
What’s up with Waabi: Urtasun founded Waabi four years ago, and its autonomous trucks are now driving with human safety drivers on public roads in Texas. Waabi—which has partnerships with major players like Uber Freight and Volvo Autonomous Solutions—is slated to launch fully driverless operations later this year.
In 2024, the company completed a $200 million Series B round with participation from investors including Nvidia, Uber, and Volvo.
Generative AI is at the heart of Waabi’s tech, in both its “Waabi World” simulator and its Waabi Driver, an end-to-end AI system. The simulator leverages GenAI to devise new scenarios, including ones that may have never happened in the real world before, and to replicate real-world environments.
The company recently announced that it’s developed a way to gauge the accuracy of simulators used in AV testing. The approach involves re-creating real-world scenarios using the Waabi World simulator, and then comparing its AV system’s performance to how it behaves on public roads.
“Every single player uses some form of simulation tools,” Urtasun said. “But what is important is that if you want to prove the safety of a self-driving truck that drives mostly on highways, then it turns out that you need something more than just to say, ‘I use simulation.’ Why is this? Because just driving in the real world is never going to be enough. Because the rate of events of when potentially you get exposed to something—like bodily injury or a severe accident—only happens every 10 million miles.”
Keep up with the innovative tech transforming business
Tech Brew keeps business leaders up-to-date on the latest innovations, automation advances, policy shifts, and more, so they can make informed decisions about tech.
In the simulation: The idea behind the use of simulation is that it would be basically impossible to gather enough real-world data to account for every single possible driving scenario a self-driving vehicle might encounter.
But historically, testing AVs has relied heavily on gathering test drive data and then having engineers pore over the outcomes and make tweaks to the AV system accordingly.
“The process is resource-intensive, inefficient, and insufficient as a solution,” Urtasun wrote in a blog post.
Virtual simulation has emerged as a widely used and important method of testing the efficacy and safety of AV systems. Some of the benefits include cost savings, as simulators can run through an infinite number of scenarios based on troves of data; the ability to test out tricky or uncommon edge cases; and improvements to scalability.
This is where the idea of realism comes into play.
“What you want to show is that under the exactly same situations, when you’re driving your self-driving vehicle in the real world with the same software that you’re driving in the simulation, that self-driving vehicle should be able to do exactly the same maneuver, the same behavior, exactly the same trajectory, etc.,” Urtasun explained. “And if you’re able to prove this across many, many, many situations that reflect reality, then you basically have proven that there’s no difference between the real world and simulation.”
That enables an AV company to reduce the amount of real-world driving it has to do to prove out the safety of its technology, and allows it to more quickly and cost-effectively scale its business, according to Urtasun.
Waabi’s approach to measuring realism uses digital twin technology to apply a method known as pair-setting, which, according to the company, “involves meticulously recreating a set of real-world scenarios within the simulator…and then measuring the difference in the AV’s trajectories across both environments.”
This demonstration, according to Urtasun, is important in part because it helps bolster Waabi’s case that its autonomous trucks are safer than human drivers.
“You’re going to see simulation as a very big part of the evidence we provide in terms of, is our system safer than a human? And should we deploy this system without the human safety driver on public roads?”