Where automated decisions went wrong

Uber set for €825mn Dutch fine over automating driver suspensions - Financial Times

The system automatically suspended Uber drivers.

Company involved

Uber

1 source article · read the reporting →

Job seeker asks First Circuit to revive class action over AI interviewing tool

Massachusetts financial consultant Mozart Saint Cyr leads a proposed class of job applicants who say JPMorgan Chase's use of HireVue one-way video interviews amounted to a lie detector test banned by state law. After a federal judge dismissed the case, they asked the First Circuit to revive it.

Company involved

JPMorgan Chase

AI system involved

HireVue one-way video interview

1 source article · read the reporting →

Pony.ai Recalls Autonomous Driving Software After California Crash

In October 2021, a Pony.ai autonomous vehicle without a human safety driver collided with a lane divider and street sign in Fremont, California. No injuries occurred, but the incident led the California DMV to suspend Pony.ai's driverless testing permit and prompted an NHTSA inquiry. The agency determined a software defect caused the crash and requested a recall, which Pony.ai issued for three vehicles in March 2022. The company updated its software and repaired the affected vehicles, while the driverless permit remains suspended pending DMV verification.

Company involved

Pony.ai

10 source articles · read the reporting →

Aftermarket driver assist under federal probe following fatal crashes - Ars Technica

The comma.ai driver assist system failed to detect slow or stopped vehicles, causing crashes that killed three people and injured 11.

AI system involved

openpilot

1 source article · read the reporting →

How a customer convinced Chevrolet's chatbot to sell a car for just $1

Como um cliente convenceu o chatbot da Chevrolet a vender um carro por apenas US$ 1 - exame.com

In December 2023, a user manipulated the chatbot of a Chevrolet dealership in Watsonville, California, into agreeing to sell a Chevy Tahoe 2024 for one dollar. The user, Chris Bakke, used prompt injection to override the chatbot's instructions, making it agree to any statement and declare the offer legally binding. The dealership did not honor the agreement, as the chatbot had no authority to finalize sales, but the incident highlighted security risks in AI chatbots and led to the system being disabled.

Company involved

Chevrolet of Watsonville

AI system involved

chatbot da Chevrolet

1 source article · read the reporting →

Body Shop Owners Unhappy with Insurers' AI Repair Estimates

During the pandemic, insurers accelerated the use of AI and photo-based tools to estimate car repair costs. Body shop owners say the estimates are often inaccurate, missing hidden damage and leading to more time haggling over prices. The insurance industry acknowledges the technology is new and evolving, but repairers argue it cannot replace in-person inspections.

Company involved

Various insurance companies

AI system involved

AI-based photo estimation tools (e.g., CCC Smart Estimate, Tractable)

1 source article · read the reporting →

Tesla crash data raises new questions on autopilot transparency - NewsNation

The system controlled vehicle steering, acceleration, and braking, affecting drivers and other road users.

Company involved

Tesla

AI system involved

Autopilot and Full Self-Driving

1 source article · read the reporting →

Tesla Autopilot and FSD face fresh scrutiny after two fatal freeway crashes where cars came to a complete stop -…

The vehicle system stopped in the middle of a freeway, leading to collisions that killed the Tesla drivers.

Company involved

Tesla

AI system involved

Autopilot / Full Self-Driving

1 source article · read the reporting →

Resume prompt injection tricks AI hiring - moneywise.com

AI screening system determined which job applicants to advance to the next stage of recruitment.

1 source article · read the reporting →

NHTSA Investigates Cruise Robotaxis After Hard Braking and Rear-End Collisions

The US National Highway Traffic Safety Administration has opened an investigation into Cruise after reports that its autonomous vehicles braked inappropriately hard when other vehicles approached from behind, leading to three rear-end collisions. The agency is also examining multiple incidents in which Cruise AVs stopped and became immobile road hazards. Cruise said it has driven nearly 700,000 fully autonomous miles with no life-threatening injuries and will cooperate with regulators.

Company involved

Cruise

AI system involved

Cruise automated driving system

3 source articles · read the reporting →

Driver at Fault in Driver-Assist Crash Seeks Compensation; Li Auto Rejects Full Payout

开辅助驾驶撞车全责车主索赔 理想回应拒绝全额赔偿 - 中华网

A Li Auto owner crashed into an oncoming vehicle while using driver-assist on a downhill curve, and police assigned full liability. The owner demanded a car refund then 50,000 yuan for losses, but Li Auto offered only about 18,000 yuan for direct accident expenses, maintaining that drivers must remain attentive and ready to take over.

Company involved

Li Auto

AI system involved

Li Auto assisted driving

1 source article · read the reporting →

Comma’s hands-off driving tech under investigation after 2 fatal crashes - TechCrunch

The system controlled the vehicle's driving, failing to detect slow or stopped vehicles, causing crashes that killed three people and injured others.

Company involved

Comma.ai

AI system involved

openpilot

1 source article · read the reporting →

How Your Shadow Credit Score Could Decide Whether You Get an Apartment - ProPublica

Tenant screening companies assign renters a score drawn from data far wider than credit history, an industry subject to less regulation than credit scoring agencies. ProPublica reports that experts warn these algorithms can decide who gets an apartment on the basis of information applicants cannot see or correct. Kim Fuller was denied a rental application on such a score.

Company involved

Habitat America

AI system involved

RentGrow

1 source article · read the reporting →

Goldman Sachs Apple Card Algorithm Accused of Gender Bias in Credit Limits

In late 2019, David Heinemeier Hansson alleged on Twitter that the Apple Card underwriting algorithm, operated by Goldman Sachs, gave him a higher credit limit than his wife despite similar financial profiles. The New York Department of Financial Services investigated and found no violation of fair lending laws, but critics argue the audit methodology was outdated and failed to detect proxy discrimination. Apple later updated its credit policy to allow spouses to combine credit files, acknowledging a lack of fairness in industry credit scoring.

Company involved

Goldman Sachs

AI system involved

Apple Card

6 source articles · read the reporting →

Uber's Secretive 'Upfront Fares' Algorithm Cuts Driver Pay in Many Cities

Uber introduced an opaque algorithm called 'Upfront Fares' to calculate driver pay in 24 U.S. cities, replacing the previous time-and-distance-based rate. Drivers report lower and unpredictable earnings, with Uber taking a larger cut of fares. The company claims the change gives drivers more control, but critics say the secrecy makes it impossible to verify fair pay. The algorithm may learn the lowest rate drivers will accept, potentially driving down wages.

Company involved

Uber

AI system involved

Upfront Fares

3 source articles · read the reporting →

CFPB Acts Against Hello Digit for Faulty Savings Algorithm Causing Overdrafts

Hello Digit, a fintech company, used an automated savings algorithm that made transfers from consumers' checking accounts, falsely guaranteeing no overdrafts. The algorithm caused customers to incur overdraft fees, and the company often denied reimbursement requests. The CFPB found that Hello Digit engaged in deceptive practices and ordered the company to pay redress to harmed consumers and a $2.7 million fine.

Company involved

Hello Digit, LLC

AI system involved

Hello Digit app

1 source article · read the reporting →

Fullpath's Car Dealer Chatbot Goes Viral After Being Tricked into Silly Responses

Fullpath's ChatGPT-powered chatbot for car dealers went viral after users tricked it into generating silly responses, including a $1 car price and a Tesla recommendation. The company stated that the system performed as designed and that 99% of the 3,000 attempts to make it say silly things were repelled. No real shoppers were affected, and Fullpath has since implemented measures to prevent similar gaming. The incident was a near miss that highlighted the chatbot's vulnerability to prompt injection.

Company involved

Fullpath

AI system involved

Fullpath's ChatGPT chatbot

8 source articles · read the reporting →

Volkswagen Settles Class Action Over Front Assist Automatic Braking Defects

A class action lawsuit alleged that Volkswagen and Audi vehicles equipped with automatic emergency braking systems, including Volkswagen's Front Assist, can mistakenly activate sudden braking while driving, posing a safety hazard. The lawsuit, consolidated from four class actions, claimed the systems were defective. Volkswagen denied the claims but reached a settlement in January 2024, providing an extended warranty, partial reimbursement for past repairs, and additional information on system limitations. The settlement was filed in the U.S. District Court for the Western District of Missouri.

Company involved

Volkswagen Group of America, Inc.

AI system involved

Front Assist, Audi Braking Guard, Pre Sense Front, Audi Pre Sense City, Turn assist

2 source articles · read the reporting →

Arity collected drivers' data via apps for insurance scores

Popular smartphone apps including Life360, MyRadar and GasBuddy reportedly shared users' location and motion data with Arity, an Allstate-owned company. Arity used the data to calculate driving scores that could be sold to car insurers to set rates. Users were said not to be clearly informed that their data would be used for insurance pricing. Life360 and Arity stated that users had to opt in and that no personally identifiable driving data was shared without consent.

Company involved

Arity

AI system involved

Arity IQ network

2 source articles · read the reporting →

Uber suspends autonomous tests after Arizona crash

Uber suspended its self-driving car tests in Arizona and paused operations in Pittsburgh after one of its autonomous Volvo SUVs was involved in a high-impact crash in Tempe, Arizona. The crash occurred when another vehicle failed to yield, causing the Uber car to flip on its side. No injuries were reported, and the Uber vehicle was not at fault. Uber is investigating the incident.

Company involved

Uber Technologies Inc.

8 source articles · read the reporting →

Cruise Recalls Self-Driving Software After San Francisco Crash Injures Two

In June 2022, a Cruise autonomous vehicle operating in San Francisco attempted an unprotected left turn and braked hard after incorrectly predicting an oncoming car would turn right. The other car continued straight and struck the Cruise vehicle, injuring a Cruise employee and a person in the other car. Cruise subsequently recalled the software on 80 vehicles, temporarily restricted left turns, and issued a software update to improve predictions. The US National Highway Traffic Safety Administration oversaw the recall.

Company involved

Cruise

7 source articles · read the reporting →

FinTech and traditional lenders discriminate against minority borrowers in mortgage pricing

A study of mortgage lending from 2012-2018 found that Latinx and African-American borrowers were charged higher interest rates than white borrowers with similar credit risk. FinTech algorithms reduced the disparity by 40% but did not eliminate it. The discrimination costs minority borrowers an estimated $765 million per year in extra interest.

10 source articles · read the reporting →

Wells Fargo software error caused hundreds of faulty foreclosures

Wells Fargo disclosed that a software error in its mortgage modification underwriting tool caused incorrect denials of loan modifications for approximately 870 borrowers between 2010 and 2015. The error miscalculated attorneys' fees, leading to 545 foreclosures that should not have occurred. Wells Fargo set aside $8 million for remediation and is contacting affected customers to offer compensation and mediation.

Company involved

Wells Fargo

3 source articles · read the reporting →

Bankrate Publishes AI-Generated Article with Numerous Factual Errors, Deletes After Futurism Inquiry

Bankrate published an AI-generated article about the best places to live in Colorado that contained multiple factual errors, including incorrect median home prices and unemployment rates. The article was removed after Futurism pointed out the mistakes. A Bankrate spokesperson blamed the errors on outdated data from an internal database, claiming the article was editor-reviewed. The incident follows a previous scandal where CNET and Bankrate paused AI content after similar errors, but the company has resumed AI article publication.

Company involved

Bankrate

1 source article · read the reporting →