Where automated decisions went wrong

Ibrahim Diallo fired by automated HR system, locked out for three weeks

Ibrahim Diallo, a software engineer in California, was automatically terminated by an HR system after a script triggered by a missed contract renewal revoked his access and disabled his key card. Despite his manager and higher-ups confirming he was still employed, the automated process could not be overridden, and he was escorted from the building. It took three weeks for the company to resolve the issue by rehiring him as a new employee, after which he quit. The incident highlights the risks of fully automated termination systems without human override.

10 source articles · read the reporting →

Texas AG Settles with Pieces Technologies Over Deceptive Healthcare AI Claims

The Texas Attorney General investigated Pieces Technologies, a Dallas-based healthcare AI company, for making false and misleading statements about the accuracy of its generative AI product used in hospitals. The company claimed an error rate of less than 1 per 100,000, but the investigation found these metrics were likely inaccurate. As part of the settlement, Pieces agreed to accurately disclose its product's accuracy and ensure hospital staff understand the appropriate reliance on its AI. The case marks the first-of-its-kind healthcare generative AI investigation by the AG.

Company involved

Pieces Technologies

4 source articles · read the reporting →

EEOC sues iTutorGroup for automated rejection of older tutor applicants

The US Equal Employment Opportunity Commission (EEOC) alleges that tutoring provider iTutorGroup programmed its recruitment software automatically to reject female applicants aged 55 or older and male applicants aged 60 or older. More than 200 qualified US-based applicants were denied work in 2020. The EEOC has filed a lawsuit in the Eastern District of New York seeking back pay and damages.

Company involved

iTutorGroup

10 source articles · read the reporting →

Barclays pilot of Sapience monitoring software causes employee stress

Barclays introduced a pilot of employee monitoring software from Sapience in its product control department at Canary Wharf. The system monitors computer activity and admonishes staff if they are not deemed active enough, recording breaks as "unaccounted activity". Employees reported significant stress and worry about taking breaks. Barclays acknowledged the pilot and said it would listen to feedback.

Company involved

Barclays

AI system involved

Sapience employee monitoring software

10 source articles · read the reporting →

India mandates Aarogya Setu app for workers despite privacy concerns

The Indian government has made its Aarogya Setu COVID-19 contact tracing app mandatory for gig workers and government employees, despite widespread privacy and algorithmic accountability concerns. The app lacks transparency and data protection safeguards, and could misidentify individuals as at risk, leading to forced isolation and loss of income. A coalition of 37 organisations and 75 individuals sent a joint representation to the Prime Minister's Office urging that the app not be made mandatory and calling for financial relief and healthcare for workers.

Company involved

Government of India

AI system involved

Aarogya Setu

10 source articles · read the reporting →

DWP's Universal Credit algorithm causes underpayment and hardship for claimants

The Department for Work and Pensions (DWP) uses an automated means-testing algorithm in its Universal Credit system that miscalculates earnings due to a design flaw, leading to underpayments and fluctuating benefit amounts. Claimants such as Janet R. received payments hundreds of pounds short, forcing them into debt and foodbank use. A Court of Appeal ruling in June 2020 ordered the DWP to fix the flaw, but the department had not yet fully implemented the remedy at the time of writing.

Company involved

Department for Work and Pensions

AI system involved

Universal Credit

10 source articles · read the reporting →

Dutch government continued SyRI welfare fraud profiling after ban

The Dutch government deployed the SyRI algorithm to profile welfare recipients in low-income neighborhoods, despite a 2020 court ruling that the system violated EU human rights. The algorithm flagged thousands of parents for fraud based on illegal risk factors such as dual nationality, leading to false accusations, bankruptcy, and trauma. The government has announced an external review of the practices.

Company involved

Ministry of Social Affairs of the Netherlands

AI system involved

SyRI

10 source articles · read the reporting →

Hospitals use Epic AI to predict Covid-19 decline without validation

Dozens of hospitals across the US are using Epic's deterioration index AI system to predict which Covid-19 patients will become critically ill, despite the tool not being validated for the new disease. The rapid deployment during the pandemic bypassed normal testing and validation processes.

AI system involved

Deterioration index

9 source articles · read the reporting →

Google and HCA Healthcare partner to analyze 32 million patient records

Google Cloud has announced a partnership with HCA Healthcare to analyze around 32 million patient records. The anonymized data will be used to develop algorithms that could advise doctors on treatment options. Privacy advocates have raised concerns about the data sharing and the potential for AI to re-identify patients. HCA insists that patient-identifiable information will be stripped and that access will be tightly controlled.

Company involved

HCA Healthcare

AI system involved

Google Cloud

9 source articles · read the reporting →

University of Chicago report finds US healthcare algorithms rife with bias

A University of Chicago report alleges that algorithms used by hospitals, insurers and other businesses across the US are biased along racial and economic lines. The algorithms help triage patients, predict diabetes and flag those needing extra care, but the report says flawed products were introduced with little oversight. It says inequitable treatment has continued for more than a decade.

7 source articles · read the reporting →

Denmark's Udbetaling Danmark welfare algorithm illegally surveilled millions

Udbetaling Danmark (UDK), Denmark's centralized welfare payment agency, used automated algorithms to check eligibility and detect fraud. The system collected personal data on millions of beneficiaries and their relatives, including non-beneficiaries, in what a think-tank called 'systematic surveillance'. A complaint to the Danish Data Protection Authority led to a ruling that the data collection violated GDPR, and UDK eventually deleted the illegally obtained data. The system also caused erroneous payments, including underpaying 300,000 pensioners by 94 million euros annually.

Company involved

Udbetaling Danmark

10 source articles · read the reporting →

Campaigners call for end to Home Office's IPIC 'robo-caseworker' AI tool

Migrants' rights campaigners have called for the UK government to stop using the IPIC (Identify & Prioritise Immigration Cases) AI system, which automatically identifies and recommends migrants for immigration decisions or enforcement action. Privacy International and the Migrants' Rights Network have raised concerns that the system could lead to racial bias and the 'rubberstamping' of algorithmic recommendations by caseworkers. The campaigners argue that people going through the immigration system may not know their information has been processed by an algorithm. The Home Office has not directly responded to the specific concerns, though Science Secretary Peter Kyle has defended the safe development of AI in public services.

Company involved

Home Office

AI system involved

IPIC (Identify & Prioritise Immigration Cases)

9 source articles · read the reporting →

EPIC lawsuit challenges USPS secret surveillance program using facial recognition

The United States Postal Service's law enforcement wing, the Postal Inspection Service, ran a secret surveillance program called the Internet Covert Operations Program (iCOP) that used facial recognition from Clearview AI and social media monitoring tools to infiltrate online communities and monitor protests. The Electronic Privacy Information Center (EPIC) sued the USPS for failing to conduct and publish a privacy impact assessment as required by the E-Government Act. A federal district court dismissed the lawsuit on standing grounds, ruling that EPIC did not suffer a cognizable injury from the agency's refusal to disclose information about the program.

Company involved

United States Postal Service

AI system involved

Internet Covert Operations Program (iCOP)

10 source articles · read the reporting →

UnitedHealthcare sued for using AI to deny Medicare Advantage patients care

A class action complaint alleges that UnitedHealthcare and its subsidiary naviHealth deployed an artificial intelligence model to override treating physicians' determinations of medically necessary care for elderly Medicare Advantage patients. The complaint claims the AI model has a known 90% error rate and wrongfully denied patients care. The lawsuit, filed in the U.S. District Court for the District of Minnesota, seeks damages on behalf of the estates of two deceased patients and all others similarly situated.

Company involved

UnitedHealthcare, Inc.

10 source articles · read the reporting →

Spanish Supreme Court orders release of BOSCO algorithm code for social electricity bonus

The Spanish NGO Civio won a Supreme Court case forcing the government to release the source code of BOSCO, the algorithm that decides eligibility for the social electricity bonus (bono social eléctrico). Civio had demonstrated in 2019 that BOSCO contained serious errors that denied the benefit to vulnerable people who met the requirements. The government had refused to disclose the code, citing intellectual property. The Supreme Court ruled that transparency must prevail, setting a precedent for public access to automated decision-making systems.

Company involved

Ministerio para la Transición Ecológica (Gobierno de España)

AI system involved

BOSCO

10 source articles · read the reporting →

UnitedHealth used ALERT algorithm to limit mental health coverage, regulators found

ProPublica reports that UnitedHealth Group's Optum subsidiary used the ALERT algorithm to flag mental health patients and therapists as outliers, leading to therapy coverage denials. Regulators in California, Massachusetts and New York alleged this breached the federal Mental Health Parity and Addiction Equity Act, and UnitedHealth agreed to restrict the system in those jurisdictions. The company denies wrongdoing and says its programmes are compliant. Affected patients, including Medicaid enrollees, were left to pay out-of-pocket or go without care.

Company involved

UnitedHealth Group

AI system involved

ALERT

5 source articles · read the reporting →

US immigration system's machine translation errors deny asylum to vulnerable people

The US immigration system's reliance on machine translation tools has caused errors in asylum applications, leading to denials for vulnerable individuals. An Afro-Indigenous man from Brazil spent six months in ICE detention unable to communicate, as the tools did not understand his regional accent. Another applicant's use of "I" was mistranslated as "we", and a domestic abuse survivor's reference to her abuser as "mi jefe" (colloquially meaning father) was translated as "my boss". These errors are alleged to have resulted in asylum denials and potential deportations.

Company involved

US Immigration and Customs Enforcement (ICE)

AI system involved

Machine translation tools

6 source articles · read the reporting →

EviCore denied heart catheterization for patient using algorithm

In fall 2021, Little John Cupp's doctor requested a left heart catheterization exam. EviCore, a company hired by UnitedHealthcare, denied the request twice using an algorithm called 'the dial' that adjusts thresholds for review. The algorithm flagged the request for review, and EviCore's doctors determined it was not medically necessary. Cupp did not receive the procedure and his symptoms continued.

Company involved

EviCore (a Cigna company)

AI system involved

the dial

6 source articles · read the reporting →

IRCC uses AI triage for Temporary Resident Visa applications

Immigration, Refugees and Citizenship Canada (IRCC) uses an AI system called Advanced Analytics to triage Temporary Resident Visa applications from India and China. The system categorizes applications into tiers, with Tier 1 approved automatically and others sent to human officers. Critics allege the system lacks transparency and may introduce bias, leading to visa refusals without clear rationale. The author, a Canadian immigration lawyer, is filing Federal Court cases on behalf of clients affected by refusals.

Company involved

Immigration, Refugees and Citizenship Canada (IRCC)

AI system involved

Advanced Analytics Triage of Overseas Temporary Resident Visa Applications

10 source articles · read the reporting →

Humana sued for using AI to deny seniors rehabilitation care

Health insurer Humana is accused in a class-action lawsuit of using an AI algorithm to systematically deny rehabilitation care to Medicare Advantage patients, despite recommendations from their doctors. The lawsuit, filed on December 12, 2023, alleges that the AI tool restricted medically necessary care. This is the second major health insurer to face legal action over its use of AI to deny care.

Company involved

Humana

9 source articles · read the reporting →

Five Illinois residents sue facial recognition company Pimeyes under BIPA

Five Illinois residents are suing Pimeyes, a facial recognition search engine, alleging that the company violated the Illinois Biometric Information Privacy Act by collecting and using their facial data without consent. The lawsuit claims that Pimeyes made the residents' photos and information available in search results, enabling unauthorized access. The company has not yet been served with the lawsuit, and its CEO stated he had no prior knowledge of the case.

Company involved

Pimeyes

AI system involved

Pimeyes

7 source articles · read the reporting →

Judge rules lawsuit over UnitedHealth’s AI care denials can move forward - STAT

The system denied necessary medical care to Medicare Advantage beneficiaries.

Company involved

UnitedHealth Group

1 source article · read the reporting →

California EDD's automated fraud detection wrongly suspended 600,000 legitimate unemployment claims

In January 2021, the California Employment Development Department used Thompson Reuters automated batch review software to flag 1.1 million unemployment claims as potentially fraudulent. EDD stopped payments on those claims without prior notice. Later, over 600,000 were confirmed as legitimate after claimants used ID.me to verify their identity. The incident highlights the trade-off between fraud prevention and timely benefit access.

Company involved

California Employment Development Department (EDD)

AI system involved

Thompson Reuters Automated Batch Review

7 source articles · read the reporting →

Haryana government's algorithm wrongly declares thousands dead, stops pensions

The Haryana state government's Family Identity Data Repository (PPP) algorithm wrongly declared thousands of elderly and widowed citizens as dead, stopping their pensions. Dhuli Chand, a 102-year-old man, had to stage a mock wedding procession to prove he was alive after his pension was cut for six months. The government acknowledged the errors and corrected some records, but many still face difficulties.

Company involved

Haryana state government

AI system involved

Family Identity Data Repository (Parivar Pehchan Patra or PPP)

6 source articles · read the reporting →