Where automated decisions went wrong

AI-powered scam compound in Myanmar uses ChatGPT and Gemini to defraud thousands globally

Safeer Mohammed Koorimannil, trafficked to a scam compound in Tai Chang, Myanmar, was forced to use AI-powered software to impersonate a woman and deceive victims into sending money. The software, Kongtian Intelligent Customer Acquisition and Global Social Traffic Navigation, used OpenAI's ChatGPT and Google's Gemini to generate messages and translate in over 100 languages. Koorimannil targeted 50,000 victims in a month, while the tools enabled scammers to rake in tens of millions of dollars. U.S. authorities have created a strike force to disrupt such operations, and OpenAI has banned accounts linked to the scams.

Company involved

Tai Chang

AI system involved

Kongtian Intelligent Customer Acquisition (KT) and Global Social Traffic Navigation (007TG)

2 source articles · read the reporting →

Kmart's facial recognition system for refund fraud found unlawful by Privacy Commissioner

Kmart Australia deployed facial recognition technology in 28 stores from June 2020 to July 2022, capturing biometric data of every customer entering the stores and those at returns counters to detect refund fraud. The system collected sensitive information without notifying customers or obtaining their consent. The Australian Privacy Commissioner found that Kmart breached the Privacy Act, as the exemption for unlawful activity did not justify the indiscriminate and disproportionate collection of biometric data from thousands of individuals. Kmart has ceased using the system and cooperated with the investigation.

Company involved

Kmart Australia Limited

7 source articles · read the reporting →

US Secret Service bought access to cellphone location data

The US Secret Service signed a contract to access Locate X, a service that aggregates location data from phone apps and allows law enforcement to track devices without a warrant. The contract with Babel Street was worth about $36,000 and ran from 2017 to 2018. A former employee said the Secret Service used Locate X in 2018 to seize illegal credit card skimmers at petrol stations. Lawmakers and civil liberties advocates criticised the practice, and Senator Ron Wyden introduced a bill to ban such purchases.

Company involved

United States Secret Service

AI system involved

Locate X

1 source article · read the reporting →

Labor unions sue Trump administration over AI social media surveillance

Three labor unions (UAW, CWA, AFT) filed a lawsuit against the U.S. Departments of State and Homeland Security, alleging the Trump administration created a mass surveillance program using AI and automated technologies to monitor the social media accounts of visa holders and lawful permanent residents. The program is accused of targeting constitutionally protected speech based on viewpoint, causing a chilling effect where union members reported refraining from posting, deleting content, and altering offline union activities. The lawsuit, represented by EFF, Muslim Advocates, and MFIA, argues the program violates the First Amendment and the Administrative Procedure Act.

Company involved

U.S. Department of State

AI system involved

Catch and Revoke

9 source articles · read the reporting →

California AG declares out-of-state ALPR data sharing unlawful

California Attorney General Rob Bonta issued guidance confirming that sharing automated license plate reader (ALPR) data with out-of-state or federal agencies violates state law SB 34. The decision follows years of advocacy by EFF and the ACLU, who demonstrated through public records that many California law enforcement agencies were sharing data with hundreds of external agencies, including ICE and CBP. The guidance mandates that all California agencies review their data sharing and cut off access to out-of-state and federal entities.

Company involved

California law enforcement agencies

AI system involved

Automated license plate readers (ALPRs)

1 source article · read the reporting →

Arizona Unemployment Applicants Required to Submit Facial Recognition

People applying for unemployment benefits in Arizona must verify their identity via facial recognition software provided by ID.me. If the system fails to match a selfie to a photo ID, applicants can be denied benefits unless they successfully complete a video chat with a referee. A class-action lawsuit filed in June 2021 alleges that the practice violates due process rights and discriminates against people of colour.

Company involved

Arizona Department of Economic Security

AI system involved

ID.me

1 source article · read the reporting →

Clearview AI settles with ACLU over facial recognition database sales

Clearview AI has agreed to stop selling its facial recognition database to most private US companies as part of a proposed settlement with the ACLU. The company scraped billions of images from social media without consent to build its database, violating Illinois' Biometric Information Privacy Act. The settlement requires Clearview to delete old facial vectors and allow Illinois residents to opt out. The company can still sell its technology to law enforcement and government agencies.

Company involved

Clearview AI

AI system involved

Clearview AI facial recognition database

8 source articles · read the reporting →

NHTSA investigation PE25012: Traffic safety violations while Full Self Driving ("FSD") is engaged

The system made driving decisions that violated traffic safety laws, affecting other road users.

Company involved

Tesla, Inc.

AI system involved

Full Self Driving (FSD)

1 source article · read the reporting →

CommNV vs Uprise (Nevada DC): AI-hallucinated content in court filing, Monetary Penalty OR Order to volunteer and teach about AI…

The AI system generated fabricated legal citations that were included in a court filing, misleading the court and opposing counsel.

Company involved

Christopher Day

1 source article · read the reporting →

OpenAI sued over alleged undisclosed human review of ChatGPT conversations - Top Class Actions

ChatGPT conversations were reviewed by third-party contractors without adequate disclosure to users.

Company involved

OpenAI OpCo LLC

AI system involved

ChatGPT

1 source article · read the reporting →

DOJ, Pinnacle reach settlement in RealPage case - Yahoo Finance

The system recommended rent prices to landlords, affecting renters' housing costs.

Company involved

Pinnacle Property Management Services

AI system involved

RealPage

1 source article · read the reporting →

Pearl Gardner v. Nationstar Mortgage LLC (D. Arizona): AI-hallucinated content in court filing, Bar Referral

Pro Se Litigant | D. Arizona | State(s): USA. AI tool: Clearpoint. Problem: Fabricated: Case Law. Consequence: Bar Referral.

1 source article · read the reporting →

Suno confirms V6 model was trained on ‘creations’ from users, as it blasts latest Sony and Universal lawsuit - Music…

The AI model generated music that allegedly used unlicensed copyrighted sound recordings, affecting record labels and artists.

Company involved

Suno

AI system involved

Suno V6

1 source article · read the reporting →

Airbnb bans users in Australia using trustworthiness algorithm

Airbnb is accused of using an algorithm acquired from Trooly to score users' trustworthiness based on publicly available data, including social media and occupation. Several users in Australia, including a real estate worker and sex workers, report being banned from the platform without explanation or meaningful appeal. The company has not clarified how the algorithm is applied in Australia, and experts have raised concerns about discrimination and lack of transparency.

Company involved

Airbnb

AI system involved

Trooly

4 source articles · read the reporting →

Spinvox accused of using human transcribers for voice messages

Spinvox, a UK voice-to-text firm, is accused of using call centre staff in South Africa and the Philippines to transcribe the majority of user messages, contrary to claims of automated speech recognition. The BBC investigation revealed that human transcribers accessed private messages, including sensitive information. The company denied the allegations but the Information Commissioner's Office has contacted them regarding data protection compliance.

Company involved

Spinvox

AI system involved

D2 (the Brain)

4 source articles · read the reporting →

Capital Standard, LLC v. U.S. Bank National Association (CA Florida (2d)): AI-hallucinated content in court filing, Monetary Sanction; Adverse Costs…

The AI generated false legal citations that were included in a court filing, misleading the court.

1 source article · read the reporting →

Consumer Reports calls for action against Meta over scam ad proliferation

Consumer Reports has called on the FTC and state attorneys general to take enforcement action against Meta for allowing billions of scam advertisements on its platforms. According to a Reuters investigation, Meta's own documents showed that the company delivered an estimated 15 billion scam ads per day in 2024, generating billions of dollars in revenue. Meta is accused of failing to identify and remove most scam ads, exposing users to fraudulent schemes and illegal products. Consumer Reports argues that Meta's practices constitute unfair business practices under the FTC Act and state laws.

Company involved

Meta

6 source articles · read the reporting →

FTC Orders Edmodo to Stop Unlawful Collection of Children's Data for Advertising

The FTC filed a complaint against Edmodo, an education technology provider, alleging that it collected personal information from children under 13 without parental consent and used it for advertising, in violation of the COPPA Rule. Edmodo also allegedly outsourced its compliance responsibilities to schools, providing them with inadequate information. Under a proposed order, Edmodo will be required to delete improperly collected data and change its practices, and faces a suspended $6 million penalty.

Company involved

Edmodo, LLC

AI system involved

Edmodo

1 source article · read the reporting →

Target's pregnancy prediction algorithm reveals teen's pregnancy to father

Target used a data mining algorithm to predict when customers were pregnant based on their purchases. The algorithm sent baby product coupons to a high school student, leading her father to complain to the store. The father later learned his daughter was indeed pregnant. Target subsequently altered the coupons to disguise the data mining.

Company involved

Target

AI system involved

Target pregnancy prediction algorithm

9 source articles · read the reporting →

Geico and other insurers charge higher premiums in minority neighborhoods than in white areas with similar risk

ProPublica's analysis of auto insurance premiums in California, Illinois, Texas and Missouri found that major insurers such as Geico, Allstate and Liberty Mutual charge minority neighborhoods up to 30% more than white areas with similar accident costs. The disparity is alleged to result from pricing algorithms that may inadvertently use zip code as a proxy for race. Otis Nash, a 26-year-old Chicagoan, pays $190.69 per month for Geico insurance while a white driver in a safer neighborhood pays $54.67. The insurers deny discrimination, stating they do not collect race data.

Company involved

Geico

5 source articles · read the reporting →

FTC charges Ring with illegal surveillance and security failures

The Federal Trade Commission charged Ring, a home security camera company, with compromising customer privacy by allowing employees to access private videos and failing to prevent hackers from taking control of cameras. Hackers accessed approximately 55,000 U.S. customers' accounts, harassing individuals including children and the elderly. The proposed order requires Ring to pay $5.8 million in refunds and implement security measures.

Company involved

Ring LLC

AI system involved

Ring cameras

10 source articles · read the reporting →

FTC orders WW International and Kurbo to delete children's data and pay $1.5m penalty

The US Federal Trade Commission alleged that WW International, formerly Weight Watchers, and its subsidiary Kurbo illegally collected personal and health data from children under 13 through the Kurbo by WW weight-loss app without parental consent. The complaint said the signup process encouraged children to lie about their age and that the companies retained the data indefinitely. The companies settled, agreeing to delete the data, destroy any algorithms derived from it, and pay a $1.5 million penalty.

Company involved

WW International, Inc. (formerly Weight Watchers) and Kurbo, Inc.

AI system involved

Kurbo by WW

10 source articles · read the reporting →

Tinder accused of age discrimination in Tinder Plus pricing

A CHOICE mystery shop found Tinder charging users over 30 up to five times more for Tinder Plus than younger users, with prices ranging from $6.99 to $34.37 per month. An anti-discrimination law expert says this likely breaches Australia's Age Discrimination Act 2004. Tinder did not respond to CHOICE's requests for comment, but settled a similar US lawsuit by stopping age-based pricing in California and compensating affected users.

Company involved

Tinder

AI system involved

Tinder Plus

4 source articles · read the reporting →

Whitebridge AI faces complaint over false reputation reports

Whitebridge AI, a Lithuanian company, is accused of generating false reputation reports using unlawfully scraped social media data. The reports contained false warnings for 'sexual nudity' and 'dangerous political content'. Privacy group Noyb filed a complaint with the Lithuanian data protection authority, alleging violations of the GDPR. The complainants sought access to their data but received no response, and were later required to provide a qualified electronic signature to correct errors.

Company involved

Whitebridge AI

6 source articles · read the reporting →