Where automated decisions went wrong

Maryland residents to pay $1.6 billion more in power bills due to out-of-state data centers: complaint

Maryland's Office of People's Counsel filed a complaint with the Federal Energy Regulatory Commission against grid operator PJM, saying its cost allocation would make Maryland households pay about $1.6 billion over a decade for transmission built to serve data centres in other states, from which they do not benefit.

Company involved

PJM Interconnection

AI system involved

Transmission cost allocation for data centres

1 source article · read the reporting →

Cigna uses algorithm to deny medical claims without doctor review

Cigna, one of the largest health insurers, used an algorithm called PXDX to automatically deny medical claims. The system flagged mismatches between diagnoses and procedures, and company doctors signed off on denials in batches without reviewing patient files. One patient, Nick van Terheyden, was denied coverage for a $350 vitamin D test that his doctor had ordered. Former employees said the system saved Cigna billions of dollars but left patients with unexpected bills.

Company involved

Cigna

AI system involved

PXDX

10 source articles · read the reporting →

SafeRent Settles Class Action Over Tenant Screening for Voucher Holders

Mary Louis and Monica Douglas filed a class action lawsuit in 2022 alleging that SafeRent Solutions' SafeRent Score product discriminated against rental applicants in Massachusetts who held public housing vouchers, violating fair housing and consumer protection laws. SafeRent denied wrongdoing. The court approved a settlement in November 2024, and payments were distributed to eligible class members in 2025 and 2026.

Company involved

SafeRent Solutions, LLC

AI system involved

SafeRent Score

5 source articles · read the reporting →

South Africa's Supreme Court of Appeal Considers Sassa Algorithm Case

Верховный апелляционный суд ЮАР рассмотрел дело об алгоритмах Sassa - UA.NEWS

The Supreme Court of Appeal heard an appeal on 25 August about Sassa's digital application system for the SRD grant. The system only accepts online applications and uses automated bank account checks that may deny grants to people whose accounts receive deposits that are not regular income. The Global Center on AI Governance submitted that automated decisions must uphold constitutional rights and be fair, non-discriminatory, and suited to South Africa's informal economy.

Company involved

South African Social Security Agency (Sassa)

AI system involved

Sassa digital application system

1 source article · read the reporting →

Deloitte report for N.L. healthcare contains likely AI-generated false citations

Newfoundland and Labrador's $1.6 million Health Human Resources Plan, authored by Deloitte, contains at least four citations to research papers that do not exist, likely generated by artificial intelligence. The errors were discovered after the report was published by the Department of Health and Community Services in May. Researchers named in the false citations confirmed their work was misattributed and the papers are nonexistent. The provincial government and Deloitte have not responded to questions about the report.

Company involved

Deloitte

AI system involved

Azure OpenAI

3 source articles · read the reporting →

Montefiore Allegedly Plans to Replace Nurses with AI for Insurance Reviews

The New York State Nurses Association alleges that Montefiore Health System plans to lay off 12 utilization review nurses and replace them with AI-powered software from Datavant. The union warns that the move could compromise patient care by having AI review insurance denials without clinical judgement. Montefiore denies the claims, stating they are investing in technology for better outcomes. The plan has sparked a town hall and calls from elected officials to halt the layoffs.

Company involved

Montefiore Health System

3 source articles · read the reporting →

Services Australia voiceprint system fooled by AI voice clone

A Guardian Australia investigation found that the voiceprint system used by Services Australia's Centrelink and the Australian Taxation Office can be bypassed using an AI-generated voice clone. A journalist created a clone of their own voice from four minutes of audio and used it with a customer reference number to access their Centrelink self-service account. The system is used by millions of Australians for identity verification over the phone. Services Australia stated that it continually assesses risks and applies additional tests if unusual circumstances are detected, but did not commit to changing the technology.

Company involved

Services Australia

AI system involved

Voiceprint

1 source article · read the reporting →

AI-generated Centrelink phishing emails target 270,000 Australians

More than 270,000 fake emails impersonating Services Australia and Centrelink were detected over four months in a broad phishing campaign. Cybersecurity firm Mimecast reports that cybercriminals are using artificial intelligence to create highly convincing clones of legitimate government communications about benefits. The attack targets vulnerable people and can lead to identity theft, data theft, malware, or ransomware.

Company involved

Services Australia

4 source articles · read the reporting →

CMS warns insurers against using AI to deny Medicare Advantage care

The Centers for Medicare & Medicaid Services (CMS) clarified that health insurers cannot use algorithms or AI to deny care to Medicare Advantage members. This follows lawsuits alleging UnitedHealth and Humana used an AI tool, nH Predict, to wrongfully deny post-acute care to elderly patients. The tool reportedly produced rigid estimates ignoring individual patient needs, leading to premature denials. CMS warned that non-compliance could result in penalties and increased audits.

Company involved

UnitedHealth, Humana

AI system involved

nH Predict

10 source articles · read the reporting →

California vaccine equity algorithm may exclude 2 million vulnerable residents

An ACLU of Northern California analysis alleges that California's Department of Public Health used an algorithm, built by Blue Shield, that allocates additional COVID-19 vaccine supply based on ZIP codes rather than census tracts. The analysis found that this approach could leave over 2 million people living in disadvantaged neighbourhoods, disproportionately communities of colour, without prioritised access to the vaccine. The ACLU called on the state to review the affected areas and explain how it will ensure these residents are not left behind.

Company involved

California Department of Public Health

AI system involved

Healthy Places Index

1 source article · read the reporting →

Estonia used AI to remove benefits from elderly without human involvement

The Council of Europe Human Rights Commissioner reported that Estonia's social security system used AI to establish incapacity levels and remove benefits from elderly people without human involvement. The system allegedly made decisions automatically, leading to complaints from NGOs.

8 source articles · read the reporting →

Idaho Medicaid used secret formula to cut assistance for disabled adults

The Idaho Department of Health and Welfare used a secret formula to drastically cut Medicaid assistance for adults with developmental disabilities, claiming the formula was a trade secret. The ACLU of Idaho filed a class action lawsuit, K.W. v. Armstrong, in 2012, alleging the cuts violated due process. A federal court struck down the formula in 2016 and ordered a new system, but as of 2024, a fair system has not yet been implemented.

Company involved

Idaho Department of Health and Welfare

10 source articles · read the reporting →

Australian government settles Robodebt class action for $1.7 billion

The Robodebt system, an automated debt recovery system used by Centrelink, calculated welfare debts using averaged income data from the ATO, leading to invalid debts. A class action was filed in November 2019 and settled in November 2020. The settlement provided refunds of over $751 million in invalid debts collected and $112 million in compensation to approximately 400,000 group members. The Australian government also dropped claims for over $1 billion in invalid debts.

Company involved

Australian Government (Centrelink)

AI system involved

Robodebt

10 source articles · read the reporting →

France: CNAF's discriminatory risk-scoring algorithm must be stopped

Amnesty International and coalition partners filed a complaint with the Council of State against CNAF's risk-scoring algorithm used to detect benefit overpayments. The algorithm assigns risk scores based on criteria that discriminate against vulnerable groups, including those with disabilities, single parents, and low-income households. The complaint alleges the system violates human rights to equality and privacy. The EU AI Act's social scoring ban may apply, but its definition remains unclear.

Company involved

CNAF

2 source articles · read the reporting →

Deepfake bot submitted 1,001 comments to Idaho Medicaid waiver website

A researcher created a bot that generated and submitted 1,001 deepfake comments to the federal public comment website for the Idaho Medicaid reform waiver in October 2019. The comments were indistinguishable from human submissions, and survey respondents could not tell them apart from real comments. The researcher later withdrew the comments. The study highlights the vulnerability of federal comment processes to automated manipulation.

8 source articles · read the reporting →

Allstate's proposed retention model charged big spenders more in Maryland

Allstate proposed an auto insurance rate adjustment plan in Maryland using a retention model algorithm that varied premium changes based on existing premiums. The algorithm would have increased rates up to 20% for customers already paying the highest premiums while capping increases at 5% for others, and would have denied meaningful discounts to thousands owed reductions. The Maryland Insurance Administration rejected the plan as discriminatory, but Allstate claims it was withdrawn and has continued proposing similar models elsewhere. The proposal never took effect in Maryland.

Company involved

Allstate Corporation

AI system involved

retention model

9 source articles · read the reporting →

Google and HCA Healthcare partner to analyze 32 million patient records

Google Cloud has announced a partnership with HCA Healthcare to analyze around 32 million patient records. The anonymized data will be used to develop algorithms that could advise doctors on treatment options. Privacy advocates have raised concerns about the data sharing and the potential for AI to re-identify patients. HCA insists that patient-identifiable information will be stripped and that access will be tightly controlled.

Company involved

HCA Healthcare

AI system involved

Google Cloud

9 source articles · read the reporting →

University of Chicago report finds US healthcare algorithms rife with bias

A University of Chicago report alleges that algorithms used by hospitals, insurers and other businesses across the US are biased along racial and economic lines. The algorithms help triage patients, predict diabetes and flag those needing extra care, but the report says flawed products were introduced with little oversight. It says inequitable treatment has continued for more than a decade.

7 source articles · read the reporting →

Denmark's Udbetaling Danmark welfare algorithm illegally surveilled millions

Udbetaling Danmark (UDK), Denmark's centralized welfare payment agency, used automated algorithms to check eligibility and detect fraud. The system collected personal data on millions of beneficiaries and their relatives, including non-beneficiaries, in what a think-tank called 'systematic surveillance'. A complaint to the Danish Data Protection Authority led to a ruling that the data collection violated GDPR, and UDK eventually deleted the illegally obtained data. The system also caused erroneous payments, including underpaying 300,000 pensioners by 94 million euros annually.

Company involved

Udbetaling Danmark

10 source articles · read the reporting →

UnitedHealthcare sued for using AI to deny Medicare Advantage patients care

A class action complaint alleges that UnitedHealthcare and its subsidiary naviHealth deployed an artificial intelligence model to override treating physicians' determinations of medically necessary care for elderly Medicare Advantage patients. The complaint claims the AI model has a known 90% error rate and wrongfully denied patients care. The lawsuit, filed in the U.S. District Court for the District of Minnesota, seeks damages on behalf of the estates of two deceased patients and all others similarly situated.

Company involved

UnitedHealthcare, Inc.

10 source articles · read the reporting →

UnitedHealth used ALERT algorithm to limit mental health coverage, regulators found

ProPublica reports that UnitedHealth Group's Optum subsidiary used the ALERT algorithm to flag mental health patients and therapists as outliers, leading to therapy coverage denials. Regulators in California, Massachusetts and New York alleged this breached the federal Mental Health Parity and Addiction Equity Act, and UnitedHealth agreed to restrict the system in those jurisdictions. The company denies wrongdoing and says its programmes are compliant. Affected patients, including Medicaid enrollees, were left to pay out-of-pocket or go without care.

Company involved

UnitedHealth Group

AI system involved

ALERT

5 source articles · read the reporting →

NHS liver transplant algorithm causes longer waits for younger patients

An NHS computer algorithm that prioritises liver transplant patients most likely to die soon has led to younger patients waiting 156 days longer on average than those over 60. The algorithm, launched in 2018, gives a score based on 21 recipient and seven donor parameters. Patients like Sarah Meredith, 31, have been waiting two years for a transplant. NHS Blood and Transplant defends the system as saving lives but acknowledges the stress on younger patients.

Company involved

NHS Blood and Transplant

10 source articles · read the reporting →

NHS QCovid algorithm incorrectly labels young patients as high risk, causing vaccine call-up

The article reports that scientists are questioning the reliability of the QCovid risk prediction algorithm used by the NHS to prioritise COVID-19 vaccinations. Due to missing data on weight or ethnicity, the algorithm automatically assigns a high BMI and high-risk ethnicity, leading to young, healthy patients being incorrectly labelled as high risk and called for early vaccination. GPs have identified hundreds of such cases, causing anxiety and potentially delaying vaccinations for more vulnerable individuals. NHS Digital acknowledged the issue and said clinicians can remove patients from the shielding list.

Company involved

NHS

AI system involved

QCovid

10 source articles · read the reporting →

EviCore denied heart catheterization for patient using algorithm

In fall 2021, Little John Cupp's doctor requested a left heart catheterization exam. EviCore, a company hired by UnitedHealthcare, denied the request twice using an algorithm called 'the dial' that adjusts thresholds for review. The algorithm flagged the request for review, and EviCore's doctors determined it was not medically necessary. Cupp did not receive the procedure and his symptoms continued.

Company involved

EviCore (a Cigna company)

AI system involved

the dial

6 source articles · read the reporting →