Where automated decisions went wrong

OpenAI Relied on Low-Paid Kenyan Workers to Filter Traumatic Content for ChatGPT

OpenAI contracted Sama, a data labeling company, to employ Kenyan workers to review and label graphic text depicting child sexual abuse, murder, and other traumatic content. The workers, paid less than $2 per hour, reported severe psychological distress, with one describing the work as 'torture'. Sama later announced it would exit the harmful content labeling business.

Company involved

Sama

8 source articles · read the reporting →

Lawsuit Says UnitedHealth Used AI With a 90% Error Rate to Deny Care - Yahoo

Denied post-acute care coverage to elderly Medicare Advantage patients

Company involved

UnitedHealth Group

AI system involved

nH Predict

1 source article · read the reporting →

ChatGPT medical advice brought man 'to brink of death', lawsuit alleges - BBC

ChatGPT repeatedly misdiagnosed Scott Winters and advised him not to seek medical help, leading to a near-fatal pulmonary embolism.

Company involved

OpenAI

AI system involved

ChatGPT

1 source article · read the reporting →

Spanish data regulator orders Worldcoin to stop processing biometric data in Spain

The Spanish Data Protection Agency (AEPD) has ordered a precautionary measure against Tools for Humanity Corporation, the company behind Worldcoin, to cease collection and processing of personal data in Spain. The AEPD received complaints alleging insufficient information, collection of data from minors, and inability to withdraw consent. The regulator acted under GDPR Article 66.1 to prevent potentially irreparable harm to individuals' data protection rights.

Company involved

Tools for Humanity Corporation

AI system involved

Worldcoin

8 source articles · read the reporting →

Meta accused of exploiting content moderators in Ghana

Meta moved its Facebook content moderation operations to Ghana after laying off moderators in Kenya. Workers in Accra report poor working conditions, low pay, and psychological harm from reviewing traumatic content. Foxglove is investigating and planning a legal challenge.

Company involved

Meta

AI system involved

Facebook content moderation system

3 source articles · read the reporting →

Portuguese regulator suspends Worldcoin's biometric data collection

Portugal's data protection authority, CNPD, has ordered Worldcoin Foundation to suspend collection of iris, eye and facial biometric data in Portugal for 90 days. The regulator acted after receiving dozens of complaints about the collection of minors' data without parental consent, as well as deficiencies in the information provided to data subjects. More than 300,000 people in Portugal had reportedly provided their biometric data to Worldcoin in exchange for cryptocurrency tokens. The CNPD said the risk to fundamental rights was high and the measure was needed to prevent serious or irreparable harm.

Company involved

Worldcoin Foundation

AI system involved

Orb

5 source articles · read the reporting →

Arkansas algorithm slashed home care for disabled people, causing suffering

In Arkansas and other states, algorithms were used to allocate home care hours for disabled and elderly people, replacing nurse judgments. The system assigned hours based on a scoring tool that failed to account for specific disabilities, leading to drastic cuts. Many suffered severe harm, including bed sores, missed meals, and one death. After lawsuits, Arkansas's system was thrown out in 2018, and Idaho's was declared unconstitutional.

Company involved

Arkansas state government

1 source article · read the reporting →

DHS plans to collect migrant children's face biometrics for facial recognition training

MIT Technology Review has reported that the U.S. Department of Homeland Security (DHS) plans to collect and analyse facial images of migrant children at the U.S.-Mexico border to train facial recognition technology. The scheme, run through the Office of Biometric Identity Management's craniofacial structural progression program, would use children's face prints to improve recognition of people as they age. Civil rights groups have raised concerns about consent and privacy, and DHS has responded that its technologies are operated within the law.

Company involved

U.S. Department of Homeland Security (DHS)

AI system involved

OBIM craniofacial structural progression program

5 source articles · read the reporting →

Spinvox accused of using human transcribers for voice messages

Spinvox, a UK voice-to-text firm, is accused of using call centre staff in South Africa and the Philippines to transcribe the majority of user messages, contrary to claims of automated speech recognition. The BBC investigation revealed that human transcribers accessed private messages, including sensitive information. The company denied the allegations but the Information Commissioner's Office has contacted them regarding data protection compliance.

Company involved

Spinvox

AI system involved

D2 (the Brain)

4 source articles · read the reporting →

Capital Standard, LLC v. U.S. Bank National Association (CA Florida (2d)): AI-hallucinated content in court filing, Monetary Sanction; Adverse Costs…

The AI generated false legal citations that were included in a court filing, misleading the court.

1 source article · read the reporting →

Heathfield man sentenced in county's first AI-generated abusive image case

James Castell, 40, of Heathfield, was sentenced at Lewes Crown Court on 19 December after pleading guilty to possessing and distributing indecent images of children, including AI-generated pseudo-photographs. Sussex Police began investigating in October 2024 and found more than 3,800 images, along with AI image-generation software. He received an 18-month suspended sentence, a Sexual Harm Prevention Order, and was ordered to register as a sex offender.

3 source articles · read the reporting →

Jackie L. Miller v. Regions Bank (N.D. Alabama): AI-hallucinated content in court filing, DQ from case and court (6 months);…

Attorney H. Gregory Harp was sanctioned with a public reprimand, disqualification from the case, a six-month court suspension, and a bar referral for filing AI-hallucinated legal citations and deleting evidence.

Company involved

Regions Bank

AI system involved

ChatGPT

1 source article · read the reporting →

Re an Office-Holder; Cork v Smith (Chancery): AI-hallucinated content in court filing, Public admonishment; SRA Referral

The AI system generated a non-existent insolvency rule that was submitted to the court, misleading the court.

Company involved

Pinsent Masons LLP

1 source article · read the reporting →

DFFH breaches privacy by using ChatGPT in child protection report

A child protection worker at the Department of Families, Fairness and Housing (DFFH) used ChatGPT to draft a Protection Application Report for the Children’s Court, entering sensitive personal information about a child. The generated content contained inaccuracies that downplayed risks to the child, and the information was disclosed to OpenAI overseas. An investigation by the Office of the Victorian Information Commissioner found DFFH failed to ensure accuracy and protect personal information, contravening IPPs 3.1 and 4.1. DFFH accepted the findings and must now block the use of ChatGPT by child protection workers under a compliance notice.

Company involved

Department of Families, Fairness and Housing

AI system involved

ChatGPT

9 source articles · read the reporting →

Registered Trustees of the Assemblies of God Gospel Church International (AGGCI) v Registrar of Societies and Others (High Court): AI-hallucinated…

The AI generated non-existent court case citations that were presented to the High Court, misleading the judge and opposing counsel.

1 source article · read the reporting →

Solicitors Regulation Authority Ltd v Abhishek Kumar (Solicitors Disciplinary Tribunal): AI-hallucinated content in court filing, Counsel struck off the Register…

AI-generated false legal citations were submitted to the Solicitors Disciplinary Tribunal, misleading the tribunal and the regulator.

1 source article · read the reporting →

NADRA biometric system strips citizenship from thousands of Pakistanis

Pakistan's National Database and Registration Authority (NADRA) operates a biometric identification system that underpins citizenship. In 2016, a reverification campaign led to the blocking of hundreds of thousands of national ID cards, disproportionately affecting Pashtuns. Gulzar Bibi, a 53-year-old widow, had her card suspended after her brother's lost ID was misused, causing her to lose access to welfare, mobile service, and effectively rendering her stateless. Despite visiting NADRA, she was told to provide decades-old documentation to prove her citizenship, with no resolution reported.

Company involved

National Database and Registration Authority (NADRA)

AI system involved

NADRA biometric identification system

5 source articles · read the reporting →

Mother of robodebt suicide victim tells inquiry of government stonewalling

Jennifer Miller, whose son Rhys Cauzzo took his own life in 2017 while facing Centrelink debts of about $17,000, told the robodebt royal commission that the government stonewalled her for years. The commission heard that Cauzzo's debt was unlawfully calculated using income averaging and that no vulnerability indicator was on his file despite his mental health conditions. Miller said she felt vindicated by the commission's findings, while the former human services minister Alan Tudge had claimed the department acted appropriately. The robodebt scheme, which ran from 2015 to 2019, ended in a $1.8bn settlement after a class action.

Company involved

Department of Human Services (Services Australia)

AI system involved

Robodebt

10 source articles · read the reporting →

CNAF algorithm flags single mother Juliette for welfare fraud investigation

In 2022, Juliette, a single mother on welfare in France, was flagged by CNAF's secretive fraud detection algorithm. A fraud investigator later determined she owed thousands of euros, which were deducted from her monthly payments. The algorithm, which scores half of France's population, is accused of discriminating against vulnerable people by using factors like single parenthood and low income.

Company involved

CNAF

10 source articles · read the reporting →

UK ICO issues preliminary enforcement notice to Snap over My AI privacy risks

The UK Information Commissioner's Office (ICO) has issued a preliminary enforcement notice to Snap, the owner of Snapchat, over its AI chatbot My AI. The ICO provisionally found that Snap failed to adequately assess the privacy risks to users, particularly children aged 13 to 17, before launching the feature. Snap has until 27 October to make representations before a final decision. If enforced, Snap could face a fine of up to 4% of global turnover or £17.5 million, and may be required to stop processing data for My AI in the UK.

Company involved

Snap

AI system involved

My AI

8 source articles · read the reporting →

Whitebridge AI faces complaint over false reputation reports

Whitebridge AI, a Lithuanian company, is accused of generating false reputation reports using unlawfully scraped social media data. The reports contained false warnings for 'sexual nudity' and 'dangerous political content'. Privacy group Noyb filed a complaint with the Lithuanian data protection authority, alleging violations of the GDPR. The complainants sought access to their data but received no response, and were later required to provide a qualified electronic signature to correct errors.

Company involved

Whitebridge AI

6 source articles · read the reporting →

Poland's Ministry of Labor profiles unemployed with discriminatory algorithm

In May 2015, Poland's Ministry of Labor and Social Policy introduced an algorithmic system to profile unemployed people into three categories based on their readiness to work, location, and disabilities. The system is accused of being discriminatory and lacking transparency, as it uses default criteria that may lead to social exclusion. A report by the Panoptykon Foundation in October 2015 highlighted these issues and recommended reforms, including granting the right to appeal.

Company involved

Ministry of Labor and Social Policy

10 source articles · read the reporting →

Commonwealth Bank worker made redundant after unknowingly training AI chatbot to replace her

Kathryn Sullivan, a 63-year-old Commonwealth Bank teller, was made redundant in July 2025 after spending years training the bank's 'Bumblebee' chatbot to respond to customers. She was unaware that she was helping the AI replace her own role. Sullivan confronted CEO Matt Comyn at the bank's AGM in Brisbane, where chairman Paul O'Malley acknowledged the bank had made a mistake. The Finance Sector Union had brought a case before the Fair Work Commission, resulting in some jobs being offered back, but not all.

Company involved

Commonwealth Bank

AI system involved

Bumblebee

4 source articles · read the reporting →

Post Office Horizon scandal falsely accused subpostmasters of theft

The Post Office's Horizon IT system, supplied by Fujitsu, falsely reported financial shortfalls at many branches across England and Wales. Subpostmasters like Pete Murray were wrongly accused of theft and prosecuted, leading to loss of livelihood, reputational damage, and in some cases suicide. A group litigation, Bates v Post Office, settled, but many victims are still seeking compensation and justice. The scandal remains under investigation by the Post Office Horizon IT Inquiry.

Company involved

Post Office

AI system involved

Horizon

10 source articles · read the reporting →