The National Road Safety Administration has a person's egg after a small research and consulting firm called Quality Control Systems has triggered a devastating criticism of an agency report for 2017, according to which Tesla's autopilot has reduced collisions with 40 %. The new analysis is now coming out – almost two years after the original report – because QCS had to sue NHTSA under the Freedom of Information Act to get the data on which the Agency's conclusions are based. In her report, QCS highlights the shortcomings in the NHTSA methodology that are serious enough to completely discredit the 40 percent figure Tesla has repeatedly cite in the past two years.
NHTSA undertook an autopilot safety study following the fatal accident of Tesla owner Josh Brown in 2016. The autopilot, in particular the Tesla belt maintenance function called Autosteer, was active during the crash and Brown ignored several warnings, the hands they returned to the wheel. Critics question whether Autopilot actually makes Tesla owners less safe by encouraging them to pay less attention to the road.
In 2017, NHTSA found that Autosteer had reduced the crash rate by 40%, which seems to postpone that concern. When another Tesla customer, Walter Huang, died in a crash related to Autosteer last March, Tesla pointed to 40% of NHTSA in a blog post, defending the technology. A few weeks later, Tesla's CEO, Elon Musk, has shuffled reporters to focus on crash stories instead of highlighting the benefits of the autopilot for safety.
"They have to write the story about how autonomous cars are really safe," Musk said during the May 2018 revenue call. "But this is not a story people want to click on, they write inflammatory titles that fundamentally mislead readers."
But now NHTSA's complete set of data is available and, if anything, it seems to contradict Musk's claims. Most of the vehicles in the Tesla data set suffered from missing data or other problems that made it impossible to say whether activation of Autosteer has increased or reduced the crash rate. But when QCS focuses on 5714 vehicles whose data does not suffer from these problems, it has been found that the actual activation of the Autosteer increases Percentage of crash by 59%.
In a sense, these are old news. NHTSA has distanced itself from its own findings in May last year, describing them as a "light comparison" that "did not evaluate the effectiveness" of Autosteer technology. In addition, the NHTSA report focuses on version 1 of the Autopilot hardware Tesla has not sold since 2016.
However, these new findings are relevant to a broader debate about how the federal government is monitoring driver assistance systems such as Autopilot. By publishing this figure of 40%, NHTSA has provided undue legitimacy to Tesla's Autopilot technology. NHTSA then struggles to prevent public disclosure of data that could help the public independently evaluate these findings, allowing Tesla to continue quoting this figure for another year. In other words, NHTSA seemed more focused on protecting Tesla from shame than on protecting the public from potentially dangerous car technologies.
Autopilot's serious math errors seem better
In 2014, Tesla began delivering cars with autopilot hardware but the Autosteer tape support system was not activated until October 2015. This provided something like a natural experiment: by comparing the crash speeds for the same car before and after October 2015 NHTSA may try to assess how technology has affected safety.
So NHTSA asked Tesla for data for cars sold between 2014 and 2016 – how many miles have passed before and after receiving the Autopilot upgrade and whether they have suffered a crash (measured with airbags) before or after the Autopilot. Tesla has unusually complete data about this type of thing because its cars have wireless connections to Tesla's headquarters.
The mathematics of NHTSA's findings had to be clear. Here's how the agency summarized its findings in January 2017:
To calculate the crash rate, take the number of crashes and divide by the mileage. NHTSA did this calculation twice – once per mile before the Autosteer upgrade, and again for miles, then. NHTSA found that catastrophes were more common before Autosteer, and the percentage dropped 40% after the technology was activated.
In such a calculation, it is important that the numerator and denominator are derived from the same set of data. If miles of a particular car are not in the denominator, then the catastrophes for the same car can not be in the numerator – otherwise the results are meaningless.
Still, according to QCS, that's what NHTSA did. Tesla provided NHTSA with data on 43,781 vehicles, but 29,051 of these vehicles lacked the data fields needed to calculate how many miles these vehicles were driving before activating the Autosteer. NHTSA did this by counting these cars as driving zero before starting Autosteer miles. Yet NHTSA counted these same vehicles that had 18 interruptions before Autosteer – more than 20% of the 86 crashes before Autosteer in the data set. The result is a significant overestimation of Tesla's crash rate.
The other vehicles in the dataset had a different, more subtle problem. Tesla supplied two different data points for cars: the last odometer reading before the Autosteer installation and the first reading of the odometer afterwards. Ideally, these testimonies would be the same. However, 8 881 vehicles had a significant difference between these figures. These cars have collectively registered millions of miles, during which it is not known whether Autosteer was active or not.
NHTSA decided not to include these miles in the denominator for calculation before or after Autosteer. The result is to inflate the crash rate for the "before" and "after" periods. This almost certainly made the results less accurate, but it is impossible to know whether this makes autopilot better or worse on the network.
It is possible to calculate accurate crash values for vehicles that have full data and there is no gap between the odometer reading before and after Autosteer. The Tesla data set included only 5,714 such vehicles. When QCS director Randy Whitfield counted the number of these vehicles, he found that one-mile crash rates increased by 59% after Tesla activated Autosteer technology.
Does that mean that Autosteer actually makes the crashes 59% more likely? Probably not. These 5714 vehicles represent only a small part of Tesla's fleet and there is no way of knowing whether they are representative. And this is the question: it is reckless to draw conclusions from such shortcomings. The NHTSA should have either asked Tesla for more data or left the calculation entirely from its report.
NHTSA keeps its data from the public on Tesla's order
Disinformation in the NHTSA report could be corrected much faster if NHTSA chose to be transparent about its data and methodology. QCS filed a request for a Freedom of Information Act on data and methodology based on the NHTSA conclusions in February 2017, about a month after the report was published. If NHTSA provided the information in a timely manner, NHTSA computational problems would likely be detected quickly. Tesla could not continue quoting them more than a year after they were published.
Instead, NHTSA fights for QCS's Freedom of Information request after Tesla indicated that the data was confidential and would have led to Tesla's competitiveness if released. QCS sued the agency in July 2017. In September 2018, a federal judge dismissed most of NHTSA's arguments by clearing the way for NHTSA to release QCS information at the end of last year.
QCS says it has no financial interest in the Autopilot dispute and has not received external assistance for its litigation. "It seemed important," Whitfield said. "We had some experience with FOIA for NHTSA data and we knew a good lawyer."
We asked NHTSA to comment on the QCS analysis on Monday morning. At their request we postponed the publication of this story with 24 hours to give them time to prepare an answer. Here's what the organization organizes, sending us: "The Agency reviews the report published by the Quality Control Systems with interest and will provide a comment as appropriate."
We asked Tesla for a comment. "The QCS analysis rejected all of the data except for 5714 vehicles out of a total of 43,781 vehicles in the dataset provided to NHTSA as early as 2016," a spokesman for the company said. "Given the dramatic increase in the number of Tesla vehicles on the road, their analysis today accounts for about 0.5% of the total mileage Tesla's vehicles have so far and about 1% of the total mileage the vehicles Tesla's have traveled so far with the Autopilot being engaged.
Tesla also noted that the NHTSA report for 2017 also had other positive things, which can be said for the Autopilot over the 40% figure. The NHTSA writes that it "has not identified any defects in the design or operation of the AEB or Autopilot systems" nor has it found "any incidents in which the systems have failed to function properly".
Tesla also highlights a new quarterly safety report Tesla publishes on its website. The report shows that Tesla cars with autopilot face fewer incidents per kilometer of Tesla cars with autopilot off. They in turn experience fewer incidents per mile of the average car on the road.
But Whitfield also rejects this data.
"The statistics are not controlled for many of the effects we know are important," Whitfield told Ars in a telephone interview Monday. It is assumed that autopilot use is limited to motorways that typically have fewer incidents per mile than other streets. So the fact that there are fewer crashes per mile when the autopilot is engaged does not necessarily prove that the Autopilot makes trips safer. This may simply reflect the fact that catastrophes are simply less common on highways.
As for the comparison between Tesla and non-Tesla vehicles, new and high-end vehicles tend to have much lower incidents than cars in general. Tesla's lower crash levels may reflect the fact that most Tesla cars are newer than the average – which means they are less likely to have problems that inevitably develop as the age of the cars.
Moreover, the high price of Tesla vehicles means that Tesla drivers will probably be richer and older than the average driver. Middle-age drivers are safer than young drivers, and richer drivers can better afford regular maintenance, avoiding safety hazards such as bald tires. Thus, the relatively low speed of Tesla's collision may reflect the demographics of its customer base more than car safety features.
But to our knowledge, Tesla has not provided the latest data on the crash of independent experts who could control these factors to assess the safety of autopilot in a rigorous manner.