Where automated decisions went wrong

NHTSA investigation PE24016: Unexpected ADS behavior

Waymo's automated driving system exhibited unexpected behavior during driving.

Company involved

Waymo

1 source article · read the reporting →

Consumer Reports finds Tesla Autopilot operates without driver

Consumer Reports engineers tested a Tesla Model Y on a closed track and found that the Autopilot system could operate the vehicle without anyone in the driver's seat, failing to warn or detect the absence of a driver. The test highlighted a safety flaw as investigators probed a fatal crash in Texas where two men died in a Tesla Model S believed to be operating without a driver. Tesla did not respond to a request for comment.

Company involved

Tesla

AI system involved

Autopilot

10 source articles · read the reporting →

Apple Intelligence generates false BBC news alert about suicide

On 13 December 2024, Apple's new generative AI feature, Apple Intelligence, generated a false news alert attributing it to the BBC, claiming the suicide of Luigi Mangione. The BBC complained to Apple. Reporters Without Borders (RSF) urged Apple to remove the feature, citing risks to reliable journalism.

Company involved

Apple

AI system involved

Apple Intelligence

7 source articles · read the reporting →

Tesla on Autopilot Crashes into Overturned Truck on Taiwan Highway

On 1 June 2020, a Tesla sedan with Autopilot engaged failed to detect an overturned delivery truck on National Highway 1 in Taiwan and collided with it. The driver, a 53-year-old man, had set the system to 110 km/h and assumed it would avoid obstacles, but he braked too late. The truck driver, who was standing on the highway after his own accident, narrowly avoided being struck. No serious injuries were reported, and police are investigating the cause, with possible factors including the truck's unusual shape and the system's limitation to detecting only moving vehicles.

AI system involved

Autopilot

3 source articles · read the reporting →

San Jose is training AI to spot homeless encampments in first US pilot

San Jose has been running a pilot in which cameras on a municipal vehicle feed computer vision software being trained to detect lived-in vehicles and homeless encampments in District 10. The scheme, believed to be the first of its kind in the US, has prompted concern among housing advocates that it could be used to punish and displace unhoused residents. The city says the aim is to respond more efficiently to complaints about encampments and that the footage is not being actively monitored for law enforcement.

Company involved

City of San Jose

10 source articles · read the reporting →

Tesla Full Self-Driving system fails to recognise train, driver intervenes

A Tesla owner in Ohio reported that his car in Full Self-Driving mode attempted to drive into a crossing train on May 8, 2024. The driver took control and avoided a collision, but the car hit the crossing arm causing damage. The driver said it was the second such incident in six months. Tesla did not respond to requests for comment.

Company involved

Tesla

AI system involved

Full Self-Driving (FSD)

8 source articles · read the reporting →

Cruise Self-Driving Car Rear-Ended by Human-Driven Cruise Vehicle in San Francisco

On 11 June 2018, a self-driving Cruise Chevrolet Bolt was rear-ended by another Cruise vehicle driven by a human employee on Bryant Street in San Francisco. The collision caused only minor scuffs and no injuries. Cruise reported the incident to the California DMV as required. The incident highlights the challenges of autonomous vehicles interacting with human drivers.

Company involved

Cruise

2 source articles · read the reporting →

AI-powered tram crashes into pedestrians in St Petersburg test run

An AI-powered 'Smart Tram' on a test run in St Petersburg, Russia, crashed into pedestrians after its AI system turned off and brakes failed. The driver, who was present, attempted to use emergency brakes but they did not fully work. Several people were injured, including a 45-year-old woman trapped under the tram. The city authorities have launched an investigation and a criminal case has been opened.

Company involved

St Petersburg city authorities

AI system involved

Smart Tram

1 source article · read the reporting →

Google and Delphi Self-Driving Cars Involved in Minor Accidents in California

Four self-driving cars in California have been involved in accidents since September 2014, when the state began issuing testing permits. Two occurred while the cars were in autonomous mode, and two while a human was driving. Google and Delphi, the companies operating the vehicles, stated that their cars were not at fault and that the collisions were minor, with no injuries. The accidents have raised concerns about transparency, as details are kept confidential under state law.

Company involved

Google Inc. and Delphi Automotive

10 source articles · read the reporting →

Google Cloud Used in CBP AI Virtual Border Wall Contract

The Intercept reported that U.S. Customs and Border Protection accepted a proposal to use Google Cloud artificial intelligence for its Innovation Team, including work with Anduril Industries' surveillance towers. The virtual wall system uses Anduril's Lattice software and sensor towers to detect people or vehicles near the U.S.-Mexico border and relay their locations to agents. Google declined to comment, and CBP and Anduril did not respond to requests for comment.

Company involved

U.S. Customs and Border Protection (CBP)

AI system involved

Google Cloud AI Platform with Anduril Lattice and Sentry Towers

1 source article · read the reporting →

Tesla Model X on Autopilot hits crash attenuator in Mountain View, killing driver

On 23 March 2018, a Tesla Model X P100D operating with Autopilot hit a damaged crash attenuator on US Highway 101 in Mountain View, California, killing the driver. The NTSB found that Autopilot steered the vehicle into a highway gore area because of system limitations, and that the driver was distracted and over-reliant on the partial driving automation. Two other vehicles were struck; one driver sustained minor injuries.

Company involved

Tesla

AI system involved

Autopilot

9 source articles · read the reporting →

Beep automated bus rear-ended by Tesla in Washington D.C.

An automated bus operated by Beep was rear-ended by a Tesla driver making an illegal lane change in Washington D.C. on 18 January 2026. The bus, part of a pilot programme with the U.S. Department of Transportation, had a human backup driver and sustained minor cosmetic damage. Service was temporarily paused but resumed after a review found the bus operated appropriately. No police were called and the incident was not reported to NHTSA as the damage was under $1,000 and the other driver was at fault.

Company involved

Beep

AI system involved

Beep automated bus

2 source articles · read the reporting →

Waymo Self-Driving Taxis Crash into Backwards-Towed Pickup in Phoenix

On 11 December 2023, two Waymo autonomous vehicles collided with a pickup truck being towed backwards in Phoenix, Arizona. The vehicles were operating without passengers and sustained minor damage, with no injuries reported. Waymo attributed the crashes to a software error that mispredicted the towed vehicle's motion and subsequently deployed a fix. The company filed a voluntary recall report with the NHTSA.

Company involved

Waymo

AI system involved

Waymo Driver

3 source articles · read the reporting →

Xiaomi SU7 fatal crash in Chengdu raises EV safety concerns

A Xiaomi SU7 electric vehicle crashed in Chengdu, China, and caught fire. Bystanders were unable to open the electronic door handles to rescue an occupant, and a 31-year-old male driver died. The incident follows an earlier fatal SU7 crash that raised questions about the vehicle's smart driving features, and has intensified scrutiny on electronic door handle safety.

Company involved

Xiaomi

AI system involved

SU7

3 source articles · read the reporting →

NHTSA investigation PE25012: Traffic safety violations while Full Self Driving ("FSD") is engaged

The system made driving decisions that violated traffic safety laws, affecting other road users.

Company involved

Tesla, Inc.

AI system involved

Full Self Driving (FSD)

1 source article · read the reporting →

OpenClaw vulnerabilities enable data leakage and prompt injection

In January 2026, researchers at Giskard exploited a deployment of OpenClaw, an open-source agentic AI. They found that architectural weaknesses in the Control UI and session management allowed prompt injection and unauthorized tool use, leading to potential data leakage across user sessions. The article outlines hardening steps to prevent such vulnerabilities.

AI system involved

OpenClaw

6 source articles · read the reporting →

Waymo Self-Driving Taxis Proactively Report to Police After Discovering Passenger Hiding a Gun

قامت سيارات الأجرة ذاتية القيادة من شركة Waymo بإبلاغ الشرطة بشكل استباقي بعد اكتشافها أن أحد الركاب كان يخفي مسدساً.…

A Waymo self-driving taxi in San Francisco detected two teenage passengers carrying an unidentified AR-15 rifle loaded with ammunition. The vehicle automatically stopped and alerted police, leading to the arrest of the two teenagers. Waymo stated that its system can identify violations of its terms of service, including carrying weapons, and will proactively notify police.

Company involved

Waymo

1 source article · read the reporting →

Tesla driver dies after vehicle stops on highway while using FSD/Autopilot

一名特斯拉司機在高速公路上因車輛突然停下而身亡，當時他正在使用FSD/自動駕駛系統 - Moomoo

A 43-year-old Tesla Model 3 driver was killed when his vehicle stopped in a travel lane on Interstate 4 and was rear-ended by a semi-truck. Florida Highway Patrol could not explain why the car stopped, but Tesla’s own filing confirms that the driver-assistance system was engaged and the car was stopped at 0 mph. Tesla has redacted key data, including the software version and event details, in its report to NHTSA.

Company involved

Tesla, Inc.

AI system involved

Autopilot / Full Self-Driving (FSD)

1 source article · read the reporting →

Woman Blindly Trusts Driver Assistance, Rear-Ends Lane-Changing Bus on Highway

女子轻信辅助驾驶放任不管 高速上直接追尾变道大巴车 - 驱动之家

On August 28, on the Shanghai-Kunming Expressway, a woman driving an SUV with driver assistance engaged rear-ended a bus that suddenly changed lanes. She assumed the system would brake automatically and took no action. The bus driver was found at fault for the lane change, but police warned against over-reliance on driver assistance.

1 source article · read the reporting →

Investigation probes autopilot failure after fatal Tesla accident in the United States

Investigação apura falha no piloto automático após acidente fatal com Tesla nos Estados Unidos - dfmobilidade.com.br

A fatal crash involving a Tesla Model 3 occurred at an intersection in Buena Vista Township, New Jersey. Authorities are investigating a possible failure of the Autopilot system and whether it was active at the time, adding to scrutiny of Tesla's driver-assistance technology.

Company involved

Tesla

AI system involved

Autopilot

1 source article · read the reporting →

Sibuxiang Genuine + Genuine Sibuxiang: Tesla Autopilot in Another Accident, Owner Says System Failed to Recognize Construction Barriers, Causing Rear-End…

四不像正版+正版四不像特斯拉自动驾驶再出事故，车主称系统未识别施工路障致追尾 - Pchome电脑之家

A Tesla owner reported that the vehicle's Autopilot system did not detect construction barriers, leading to a rear-end collision. The incident was reported by Pchome.

Company involved

Tesla

AI system involved

Autopilot

1 source article · read the reporting →

NHTSA investigation PE24015: Rear-end Collisions Involving Zoox Vehicles

Automated driving system controlled vehicle braking and acceleration, leading to rear-end collisions with other road users

Company involved

Zoox

AI system involved

Zoox vehicle

1 source article · read the reporting →

Ford BlueCruise crash in Philadelphia kills two, injures one

On March 3, 2024, a Ford Mustang Mach-E operating in hands-free partial automation mode (BlueCruise) collided with stationary vehicles in a work zone on I-95 in Philadelphia. The system recorded a speed of 72.4 mph with no braking or steering before impact. The drivers of the two stationary vehicles died, and the Ford driver was injured. The NTSB determined the probable cause to be driver impairment and distraction due to overreliance on automation, as well as Ford's inadequate integration of speed management with the automation system.

Company involved

Ford Motor Company

AI system involved

BlueCruise

9 source articles · read the reporting →

First Accident Before Test Phase: Robotaxis Launch in Furttal

Erster Unfall, bevor die Testphase beginnt: Robotaxis starten im Furttal - nzz.ch

Switzerland's first self-driving taxi service launched in the Furttal region near Zurich, but three days before the start a test vehicle had a minor accident in Dänikon, damaging a ground plate and two tires. The pilot project, called Iamo, aims to complement public transport, with a safety driver on board during the test phase.

Company involved

Swiss Transit Lab

1 source article · read the reporting →