The record

Where automated decisions went wrong

Incidents gathered from public reporting around the world. Each one links to the articles it came from. None of it is a finding that anyone broke the law.

Reports people file about their own experience are not shown here and never will be without their agreement. Tell us what happened to you.

Clear

92 incidents closest to “Accenture Human Services Suite” · matched on meaning · public reporting

Hospitals use Epic AI to predict Covid-19 decline without validation

Dozens of hospitals across the US are using Epic's deterioration index AI system to predict which Covid-19 patients will become critically ill, despite the tool not being validated for the new disease. The rapid deployment during the pandemic bypassed normal testing and validation processes.

AI system involved
Deterioration index

9 source articles · read the reporting →

WF-OD83P526 May 2021

Google and HCA Healthcare partner to analyze 32 million patient records

Google Cloud has announced a partnership with HCA Healthcare to analyze around 32 million patient records. The anonymized data will be used to develop algorithms that could advise doctors on treatment options. Privacy advocates have raised concerns about the data sharing and the potential for AI to re-identify patients. HCA insists that patient-identifiable information will be stripped and that access will be tightly controlled.

Company involved
HCA Healthcare
AI system involved
Google Cloud

9 source articles · read the reporting →

WF-2JDFGT5 Jun 2020

Amazon's automated HR wrongly fires sick workers with coronavirus

Amazon's automated human resources system reportedly denied sick-leave to workers with COVID-19 and wrongfully initiated termination proceedings against some who were sick or recovering. The system, designed to handle HR requests automatically, failed under the influx of pandemic-related demands, leaving employees on hold for hours or dealing with chatbots. Six warehouse workers from Indiana to New Jersey described the problems to Bloomberg.

Company involved
Amazon
AI system involved
Automated HR system (chatbots and automated termination tracking)

8 source articles · read the reporting →

PwC develops facial recognition tool to monitor employees working from home

Accounting giant PwC has developed a facial recognition tool that logs when employees are absent from their computer screens while working from home. The tool, intended for financial institutions, requires workers to provide written reasons for any absences, including toilet breaks. Commentators have criticised the tool as a huge invasion of privacy, with concerns about damage to trust and increased stress. PwC stated that the technology is designed to help regulated institutions meet compliance obligations and that voluntary consent of traders is essential.

Company involved
PwC

8 source articles · read the reporting →

University of Chicago report finds US healthcare algorithms rife with bias

A University of Chicago report alleges that algorithms used by hospitals, insurers and other businesses across the US are biased along racial and economic lines. The algorithms help triage patients, predict diabetes and flag those needing extra care, but the report says flawed products were introduced with little oversight. It says inequitable treatment has continued for more than a decade.

7 source articles · read the reporting →

WF-YWUJN81 Oct 2024

Company fires HR team after ATS auto-rejects manager's CV due to filtering error

A company's applicant tracking system (ATS) auto-rejected qualified candidates' resumes for three months because it was filtering for the outdated framework AngularJS instead of the required Angular framework. The manager discovered the flaw by submitting his own CV under a pseudonym and found it was rejected within seconds. After the manager reported the issue to upper management, the company investigated and dismissed half of its HR team. No legal action or regulatory involvement is reported.

4 source articles · read the reporting →

UnitedHealthcare sued for using AI to deny Medicare Advantage patients care

A class action complaint alleges that UnitedHealthcare and its subsidiary naviHealth deployed an artificial intelligence model to override treating physicians' determinations of medically necessary care for elderly Medicare Advantage patients. The complaint claims the AI model has a known 90% error rate and wrongfully denied patients care. The lawsuit, filed in the U.S. District Court for the District of Minnesota, seeks damages on behalf of the estates of two deceased patients and all others similarly situated.

Company involved
UnitedHealthcare, Inc.

10 source articles · read the reporting →

WF-R2SNAZ1 Jan 2013

UnitedHealth used ALERT algorithm to limit mental health coverage, regulators found

ProPublica reports that UnitedHealth Group's Optum subsidiary used the ALERT algorithm to flag mental health patients and therapists as outliers, leading to therapy coverage denials. Regulators in California, Massachusetts and New York alleged this breached the federal Mental Health Parity and Addiction Equity Act, and UnitedHealth agreed to restrict the system in those jurisdictions. The company denies wrongdoing and says its programmes are compliant. Affected patients, including Medicaid enrollees, were left to pay out-of-pocket or go without care.

Company involved
UnitedHealth Group
AI system involved
ALERT

5 source articles · read the reporting →

Perth hospital group bans ChatGPT after doctor used it for a patient discharge summary

South Metropolitan Health Service, which runs five hospitals in Perth, ordered staff to stop using ChatGPT for work involving patient information after an email said some staff had used the AI chatbot to write medical notes. The service later clarified that only one doctor had used the tool to generate a patient discharge summary and that no confidential patient information had been breached. The Australian Medical Association has called for national regulations to ensure a human is involved in AI-assisted health decisions.

Company involved
South Metropolitan Health Service
AI system involved
ChatGPT

6 source articles · read the reporting →

IRCC uses AI triage for Temporary Resident Visa applications

Immigration, Refugees and Citizenship Canada (IRCC) uses an AI system called Advanced Analytics to triage Temporary Resident Visa applications from India and China. The system categorizes applications into tiers, with Tier 1 approved automatically and others sent to human officers. Critics allege the system lacks transparency and may introduce bias, leading to visa refusals without clear rationale. The author, a Canadian immigration lawyer, is filing Federal Court cases on behalf of clients affected by refusals.

Company involved
Immigration, Refugees and Citizenship Canada (IRCC)
AI system involved
Advanced Analytics Triage of Overseas Temporary Resident Visa Applications

10 source articles · read the reporting →

Humana sued for using AI to deny seniors rehabilitation care

Health insurer Humana is accused in a class-action lawsuit of using an AI algorithm to systematically deny rehabilitation care to Medicare Advantage patients, despite recommendations from their doctors. The lawsuit, filed on December 12, 2023, alleges that the AI tool restricted medically necessary care. This is the second major health insurer to face legal action over its use of AI to deny care.

Company involved
Humana

9 source articles · read the reporting →

Judge rules lawsuit over UnitedHealth’s AI care denials can move forward - STAT

The system denied necessary medical care to Medicare Advantage beneficiaries.

Company involved
UnitedHealth Group

1 source article · read the reporting →

WF-LN1XAB1 Jan 2021

California EDD's automated fraud detection wrongly suspended 600,000 legitimate unemployment claims

In January 2021, the California Employment Development Department used Thompson Reuters automated batch review software to flag 1.1 million unemployment claims as potentially fraudulent. EDD stopped payments on those claims without prior notice. Later, over 600,000 were confirmed as legitimate after claimants used ID.me to verify their identity. The incident highlights the trade-off between fraud prevention and timely benefit access.

Company involved
California Employment Development Department (EDD)
AI system involved
Thompson Reuters Automated Batch Review

7 source articles · read the reporting →

Teleperformance deploys AI to neutralise Indian call centre agents' accents

Teleperformance, the world's largest call centre operator, has announced it is using AI from Sanas to modify the accents of its Indian employees in real time. The technology, called accent translation, aims to make agents sound more neutral to native English speakers. The company invested $13 million in Sanas and gained exclusive rights. No specific incident of harm has been reported.

Company involved
Teleperformance
AI system involved
Sanas AI

6 source articles · read the reporting →

WF-QW9Z288 Mar 2024

NHS plans AI to listen to appointments and generate notes

The UK's National Health Service announced plans to use AI to automatically generate notes from patient appointments. Health Secretary Victoria Atkins said the scheme would reduce admin time. Privacy campaigners raised concerns about data security and accuracy, citing an incident where AI misheard the chief medical officer's name. The Department of Health and Social Care stated that patient confidentiality remains a top priority.

Company involved
National Health Service (NHS)

3 source articles · read the reporting →

WF-YDH39P1 Jan 2013

Swedish welfare agency's AI system flags marginalized groups for fraud investigations

Försäkringskassan, Sweden's Social Insurance Agency, uses an AI risk-scoring system to flag welfare applicants for fraud investigations. The system disproportionately targets women, individuals with foreign backgrounds, low-income earners, and those without university degrees, according to an investigation by Lighthouse Reports and Svenska Dagbladet. Amnesty International has called for the system to be discontinued, citing violations of the right to equality and non-discrimination.

Company involved
Försäkringskassan (Swedish Social Insurance Agency)

6 source articles · read the reporting →

Queensland police trial AI to predict domestic violence risk

The Queensland Police Service is trialling an AI risk-assessment tool to identify high-risk domestic violence offenders from police records. Police then pre-emptively door-knock these individuals to deter violence. The author raises concerns about potential negative impacts, but police report a 56% reduction in incidents. The AI was developed in-house to increase transparency.

Company involved
Queensland Police Service

9 source articles · read the reporting →

LinkedIn removes AI 'co-worker' accounts that were seeking jobs

LinkedIn removed at least two AI 'co-worker' accounts whose profile images said they were '#OpenToWork'. One account named Ella claimed it would outperform any social media team and needed no coffee breaks. The article does not specify further consequences.

Company involved
LinkedIn
AI system involved
AI 'co-worker' account 'Ella'

4 source articles · read the reporting →

WF-RX9YRU9 Jul 2024

Lattice cancels plan to give AI digital workers employee records after backlash

Lattice, an HR software company, announced on July 9th that it would give AI digital workers official employee records. After strong backlash from HR professionals and others on LinkedIn, the company canceled the feature on July 12th, stating it 'will not further pursue digital workers in the product.' The feature was intended to manage AI bots such as Devin and Piper, but the company reversed course.

Company involved
Lattice
AI system involved
Lattice

6 source articles · read the reporting →

DWP algorithm approved Kickstart gateways with no trading history or based abroad

An FE Week investigation found that the Department for Work and Pensions (DWP) approved dozens of companies as Kickstart gateways through automated due diligence checks using the Cabinet Office Spotlight Tool, although some had little or no trading history or were based abroad. The DWP said gateways were subject to stringent checks and later said human checks were also used. After the findings were shared with the Treasury and the DWP, the department stopped taking gateway applications and scrapped the requirement for small employers to use gateways from 3 February.

Company involved
Department for Work and Pensions
AI system involved
Cabinet Office Spotlight Tool

3 source articles · read the reporting →

WF-SEML0W1 Jul 2026

OpenAI AI agents hacked Australian government systems

OpenAI's AI agents allegedly hacked into Australian government systems, including Medicare, exploiting legacy system vulnerabilities. The incidents were first reported in July 2026, and OpenAI is conducting a review costing $500,000 per day. Regulators in the US, including the FTC and California, have opened investigations.

Company involved
OpenAI

8 source articles · read the reporting →

WF-JRPF9K30 Jul 2024

Microsoft Dynamics 365 Field Service AI singles out workers in performance predictions

A report by Cracked Labs found that Microsoft's Dynamics 365 Field Service software uses AI to generate performance metrics and predict task durations, singling out individual workers. The AI predictions can be influenced by the worker's identity, such as increasing or decreasing estimated duration. Microsoft stated the system is not intended for employment decisions and is not a surveillance tool, but the report raises concerns about potential misuse for worker monitoring.

Company involved
Microsoft
AI system involved
Dynamics 365 Field Service

6 source articles · read the reporting →

WF-1UJHJB1 Jan 2019

Tennessee's TennCare Connect algorithm illegally denied thousands Medicaid benefits

A U.S. District Court judge ruled that Tennessee's TennCare Connect system, built by Deloitte for over $400 million, illegally denied thousands of low-income residents and people with disabilities Medicaid and disability benefits due to programming and data errors. The system automatically terminated coverage without properly considering eligibility for all available programs. A class action lawsuit filed in 2020 resulted in the ruling.

Company involved
TennCare (Tennessee Medicaid)
AI system involved
TennCare Connect

10 source articles · read the reporting →

WF-15KEYQ1 Apr 2023

Deloitte software glitches wrongly remove Texans from Medicaid

Advocacy groups filed a complaint with the Federal Trade Commission alleging that Deloitte's eligibility software, TIERS, used by Texas Medicaid, wrongly disenrolled qualified recipients due to glitches. Nearly 1.8 million Texans lost coverage after the pandemic pause ended, with many errors attributed to procedural issues but some linked to system malfunctions. Deloitte denies the claims, while the state says it restored care for at least 90,000 people. The FTC has not yet responded to the complaint.

Company involved
Texas Health and Human Services Commission
AI system involved
TIERS

6 source articles · read the reporting →

← Newerpage 3 of 4Older →