Where automated decisions went wrong

California AG declares out-of-state ALPR data sharing unlawful

California Attorney General Rob Bonta issued guidance confirming that sharing automated license plate reader (ALPR) data with out-of-state or federal agencies violates state law SB 34. The decision follows years of advocacy by EFF and the ACLU, who demonstrated through public records that many California law enforcement agencies were sharing data with hundreds of external agencies, including ICE and CBP. The guidance mandates that all California agencies review their data sharing and cut off access to out-of-state and federal entities.

Company involved

California law enforcement agencies

AI system involved

Automated license plate readers (ALPRs)

1 source article · read the reporting →

Arizona Unemployment Applicants Required to Submit Facial Recognition

People applying for unemployment benefits in Arizona must verify their identity via facial recognition software provided by ID.me. If the system fails to match a selfie to a photo ID, applicants can be denied benefits unless they successfully complete a video chat with a referee. A class-action lawsuit filed in June 2021 alleges that the practice violates due process rights and discriminates against people of colour.

Company involved

Arizona Department of Economic Security

AI system involved

ID.me

1 source article · read the reporting →

Clearview AI settles with ACLU over facial recognition database sales

Clearview AI has agreed to stop selling its facial recognition database to most private US companies as part of a proposed settlement with the ACLU. The company scraped billions of images from social media without consent to build its database, violating Illinois' Biometric Information Privacy Act. The settlement requires Clearview to delete old facial vectors and allow Illinois residents to opt out. The company can still sell its technology to law enforcement and government agencies.

Company involved

Clearview AI

AI system involved

Clearview AI facial recognition database

8 source articles · read the reporting →

NHTSA investigation PE25012: Traffic safety violations while Full Self Driving ("FSD") is engaged

The system made driving decisions that violated traffic safety laws, affecting other road users.

Company involved

Tesla, Inc.

AI system involved

Full Self Driving (FSD)

1 source article · read the reporting →

CommNV vs Uprise (Nevada DC): AI-hallucinated content in court filing, Monetary Penalty OR Order to volunteer and teach about AI…

The AI system generated fabricated legal citations that were included in a court filing, misleading the court and opposing counsel.

Company involved

Christopher Day

1 source article · read the reporting →

OpenAI sued over alleged undisclosed human review of ChatGPT conversations - Top Class Actions

ChatGPT conversations were reviewed by third-party contractors without adequate disclosure to users.

Company involved

OpenAI OpCo LLC

AI system involved

ChatGPT

1 source article · read the reporting →

DOJ, Pinnacle reach settlement in RealPage case - Yahoo Finance

The system recommended rent prices to landlords, affecting renters' housing costs.

Company involved

Pinnacle Property Management Services

AI system involved

RealPage

1 source article · read the reporting →

Pearl Gardner v. Nationstar Mortgage LLC (D. Arizona): AI-hallucinated content in court filing, Bar Referral

Pro Se Litigant | D. Arizona | State(s): USA. AI tool: Clearpoint. Problem: Fabricated: Case Law. Consequence: Bar Referral.

1 source article · read the reporting →

Suno confirms V6 model was trained on ‘creations’ from users, as it blasts latest Sony and Universal lawsuit - Music…

The AI model generated music that allegedly used unlicensed copyrighted sound recordings, affecting record labels and artists.

Company involved

Suno

AI system involved

Suno V6

1 source article · read the reporting →

Airbnb bans users in Australia using trustworthiness algorithm

Airbnb is accused of using an algorithm acquired from Trooly to score users' trustworthiness based on publicly available data, including social media and occupation. Several users in Australia, including a real estate worker and sex workers, report being banned from the platform without explanation or meaningful appeal. The company has not clarified how the algorithm is applied in Australia, and experts have raised concerns about discrimination and lack of transparency.

Company involved

Airbnb

AI system involved

Trooly

4 source articles · read the reporting →

Spinvox accused of using human transcribers for voice messages

Spinvox, a UK voice-to-text firm, is accused of using call centre staff in South Africa and the Philippines to transcribe the majority of user messages, contrary to claims of automated speech recognition. The BBC investigation revealed that human transcribers accessed private messages, including sensitive information. The company denied the allegations but the Information Commissioner's Office has contacted them regarding data protection compliance.

Company involved

Spinvox

AI system involved

D2 (the Brain)

4 source articles · read the reporting →

Capital Standard, LLC v. U.S. Bank National Association (CA Florida (2d)): AI-hallucinated content in court filing, Monetary Sanction; Adverse Costs…

The AI generated false legal citations that were included in a court filing, misleading the court.

1 source article · read the reporting →

Jackie L. Miller v. Regions Bank (N.D. Alabama): AI-hallucinated content in court filing, DQ from case and court (6 months);…

Attorney H. Gregory Harp was sanctioned with a public reprimand, disqualification from the case, a six-month court suspension, and a bar referral for filing AI-hallucinated legal citations and deleting evidence.

Company involved

Regions Bank

AI system involved

ChatGPT

1 source article · read the reporting →

DFFH breaches privacy by using ChatGPT in child protection report

A child protection worker at the Department of Families, Fairness and Housing (DFFH) used ChatGPT to draft a Protection Application Report for the Children’s Court, entering sensitive personal information about a child. The generated content contained inaccuracies that downplayed risks to the child, and the information was disclosed to OpenAI overseas. An investigation by the Office of the Victorian Information Commissioner found DFFH failed to ensure accuracy and protect personal information, contravening IPPs 3.1 and 4.1. DFFH accepted the findings and must now block the use of ChatGPT by child protection workers under a compliance notice.

Company involved

Department of Families, Fairness and Housing

AI system involved

ChatGPT

9 source articles · read the reporting →

Consumer Reports calls for action against Meta over scam ad proliferation

Consumer Reports has called on the FTC and state attorneys general to take enforcement action against Meta for allowing billions of scam advertisements on its platforms. According to a Reuters investigation, Meta's own documents showed that the company delivered an estimated 15 billion scam ads per day in 2024, generating billions of dollars in revenue. Meta is accused of failing to identify and remove most scam ads, exposing users to fraudulent schemes and illegal products. Consumer Reports argues that Meta's practices constitute unfair business practices under the FTC Act and state laws.

Company involved

Meta

6 source articles · read the reporting →

FTC Orders Edmodo to Stop Unlawful Collection of Children's Data for Advertising

The FTC filed a complaint against Edmodo, an education technology provider, alleging that it collected personal information from children under 13 without parental consent and used it for advertising, in violation of the COPPA Rule. Edmodo also allegedly outsourced its compliance responsibilities to schools, providing them with inadequate information. Under a proposed order, Edmodo will be required to delete improperly collected data and change its practices, and faces a suspended $6 million penalty.

Company involved

Edmodo, LLC

AI system involved

Edmodo

1 source article · read the reporting →

Target's pregnancy prediction algorithm reveals teen's pregnancy to father

Target used a data mining algorithm to predict when customers were pregnant based on their purchases. The algorithm sent baby product coupons to a high school student, leading her father to complain to the store. The father later learned his daughter was indeed pregnant. Target subsequently altered the coupons to disguise the data mining.

Company involved

Target

AI system involved

Target pregnancy prediction algorithm

9 source articles · read the reporting →

Geico and other insurers charge higher premiums in minority neighborhoods than in white areas with similar risk

ProPublica's analysis of auto insurance premiums in California, Illinois, Texas and Missouri found that major insurers such as Geico, Allstate and Liberty Mutual charge minority neighborhoods up to 30% more than white areas with similar accident costs. The disparity is alleged to result from pricing algorithms that may inadvertently use zip code as a proxy for race. Otis Nash, a 26-year-old Chicagoan, pays $190.69 per month for Geico insurance while a white driver in a safer neighborhood pays $54.67. The insurers deny discrimination, stating they do not collect race data.

Company involved

Geico

5 source articles · read the reporting →

FTC charges Ring with illegal surveillance and security failures

The Federal Trade Commission charged Ring, a home security camera company, with compromising customer privacy by allowing employees to access private videos and failing to prevent hackers from taking control of cameras. Hackers accessed approximately 55,000 U.S. customers' accounts, harassing individuals including children and the elderly. The proposed order requires Ring to pay $5.8 million in refunds and implement security measures.

Company involved

Ring LLC

AI system involved

Ring cameras

10 source articles · read the reporting →

FTC orders WW International and Kurbo to delete children's data and pay $1.5m penalty

The US Federal Trade Commission alleged that WW International, formerly Weight Watchers, and its subsidiary Kurbo illegally collected personal and health data from children under 13 through the Kurbo by WW weight-loss app without parental consent. The complaint said the signup process encouraged children to lie about their age and that the companies retained the data indefinitely. The companies settled, agreeing to delete the data, destroy any algorithms derived from it, and pay a $1.5 million penalty.

Company involved

WW International, Inc. (formerly Weight Watchers) and Kurbo, Inc.

AI system involved

Kurbo by WW

10 source articles · read the reporting →

Tinder accused of age discrimination in Tinder Plus pricing

A CHOICE mystery shop found Tinder charging users over 30 up to five times more for Tinder Plus than younger users, with prices ranging from $6.99 to $34.37 per month. An anti-discrimination law expert says this likely breaches Australia's Age Discrimination Act 2004. Tinder did not respond to CHOICE's requests for comment, but settled a similar US lawsuit by stopping age-based pricing in California and compensating affected users.

Company involved

Tinder

AI system involved

Tinder Plus

4 source articles · read the reporting →

Whitebridge AI faces complaint over false reputation reports

Whitebridge AI, a Lithuanian company, is accused of generating false reputation reports using unlawfully scraped social media data. The reports contained false warnings for 'sexual nudity' and 'dangerous political content'. Privacy group Noyb filed a complaint with the Lithuanian data protection authority, alleging violations of the GDPR. The complainants sought access to their data but received no response, and were later required to provide a qualified electronic signature to correct errors.

Company involved

Whitebridge AI

6 source articles · read the reporting →

Upstart Holdings sued for securities fraud over AI model claims

Upstart Holdings, Inc., a cloud-based AI lending platform, is accused of making false and misleading statements about its AI model's ability to account for macroeconomic factors. The lawsuit, filed on May 13, 2022, alleges that the company's positive statements about its business were materially false. Investors suffered losses when Upstart's stock price fell 56% on May 9, 2022, after the company reduced its fiscal 2022 guidance.

Company involved

Upstart Holdings, Inc.

AI system involved

Upstart AI lending platform

10 source articles · read the reporting →

Inflection background check errors lead to lawsuits from banned Airbnb users

Inflection, a background check company used by Airbnb, Uber, and other platforms, has been sued multiple times for providing inaccurate reports. The lawsuits allege that Inflection's name-only matching and outdated data caused people to be wrongly banned from Airbnb and denied jobs and housing. Some cases have been settled, while others remain ongoing. The company is accused of violating the Fair Credit Reporting Act.

Company involved

Inflection

AI system involved

Inflection background check

8 source articles · read the reporting →