Where automated decisions went wrong

EEOC sues iTutorGroup for automated rejection of older tutor applicants

The US Equal Employment Opportunity Commission (EEOC) alleges that tutoring provider iTutorGroup programmed its recruitment software automatically to reject female applicants aged 55 or older and male applicants aged 60 or older. More than 200 qualified US-based applicants were denied work in 2020. The EEOC has filed a lawsuit in the Eastern District of New York seeking back pay and damages.

Company involved

iTutorGroup

10 source articles · read the reporting →

Newham Council fined £145,000 over leaked gang matrix data

Newham Council was fined £145,000 by the Information Commissioner's Office after a leaked unredacted gangs matrix, containing details of 203 suspected gang members, ended up in the hands of rival gang members via Snapchat. The leak occurred in January 2017 when a council employee emailed both redacted and unredacted versions to 44 recipients. The ICO found the breach was unnecessary and that the council failed to report it promptly. The council apologised and accepted the breach was not deliberate.

Company involved

Newham Council

AI system involved

Gangs matrix

8 source articles · read the reporting →

Titus Henderson files civil rights lawsuit against COMPAS

Titus Henderson, a prisoner, filed a civil rights lawsuit against COMPAS and other defendants in the U.S. District Court for the Western District of Wisconsin on July 16, 2018. The lawsuit alleges discrimination related to the use of the COMPAS risk assessment tool. The case is ongoing.

AI system involved

COMPAS

6 source articles · read the reporting →

Allstate's proposed retention model charged big spenders more in Maryland

Allstate proposed an auto insurance rate adjustment plan in Maryland using a retention model algorithm that varied premium changes based on existing premiums. The algorithm would have increased rates up to 20% for customers already paying the highest premiums while capping increases at 5% for others, and would have denied meaningful discounts to thousands owed reductions. The Maryland Insurance Administration rejected the plan as discriminatory, but Allstate claims it was withdrawn and has continued proposing similar models elsewhere. The proposal never took effect in Maryland.

Company involved

Allstate Corporation

AI system involved

retention model

9 source articles · read the reporting →

Study finds Italian car insurers charge more based on birthplace

A study by the Universities of Padua, Udine and Carnegie Mellon found that Italian car insurers, including Genertel, Mps, Quixa and Con.Te, use birthplace and citizenship in pricing algorithms, charging some drivers over €1,000 more. The practice was ruled against in a 2018 decree involving Linear, but the study says it continues. The companies contacted denied or explained the findings.

Company involved

Genertel, Mps, Quixa, Con.Te

8 source articles · read the reporting →

LA's VI-SPDAT housing scoring system gives lower priority to Black and Latino people

The Los Angeles Homeless Services Authority uses the VI-SPDAT scoring system to prioritise unhoused people for subsidised permanent housing. An investigation by The Markup found that Black and Latino people consistently receive lower vulnerability scores than White people, leading to lower priority for housing. The agency has acknowledged the racial disparities and is working on a new tool, but continues to use the current system.

Company involved

Los Angeles Homeless Services Authority

AI system involved

VI-SPDAT

10 source articles · read the reporting →

Ramirez v. Humala (E.D. New York): AI-hallucinated content in court filing, Monetary sanction jointly imposed on counsel and firm; order…

The AI generated nonexistent case citations that were filed in court, misleading the court and opposing counsel.

1 source article · read the reporting →

Google and HCA Healthcare partner to analyze 32 million patient records

Google Cloud has announced a partnership with HCA Healthcare to analyze around 32 million patient records. The anonymized data will be used to develop algorithms that could advise doctors on treatment options. Privacy advocates have raised concerns about the data sharing and the potential for AI to re-identify patients. HCA insists that patient-identifiable information will be stripped and that access will be tightly controlled.

Company involved

HCA Healthcare

AI system involved

Google Cloud

9 source articles · read the reporting →

Amazon's automated HR wrongly fires sick workers with coronavirus

Amazon's automated human resources system reportedly denied sick-leave to workers with COVID-19 and wrongfully initiated termination proceedings against some who were sick or recovering. The system, designed to handle HR requests automatically, failed under the influx of pandemic-related demands, leaving employees on hold for hours or dealing with chatbots. Six warehouse workers from Indiana to New Jersey described the problems to Bloomberg.

Company involved

Amazon

AI system involved

Automated HR system (chatbots and automated termination tracking)

8 source articles · read the reporting →

Barteca Holdings v. Tacobarn (D. Connecticut): AI-hallucinated content in court filing, Monetary Sanction; Bar Referral

The attorney filed a court document containing AI-hallucinated legal authorities, misleading the court and opposing party.

Company involved

Hilary Miller

1 source article · read the reporting →

TOV Realty, LLC v. Suarez; Kosel Equity, LLC v. MacGregor (SC Connecticut): AI-hallucinated content in court filing, 6 hours CLE;…

The AI generated legal briefs containing fabricated citations, which were submitted to the Connecticut Supreme Court.

Company involved

GLG Law LLC

AI system involved

ChatGPT

1 source article · read the reporting →

UnitedHealthcare sued for using AI to deny Medicare Advantage patients care

A class action complaint alleges that UnitedHealthcare and its subsidiary naviHealth deployed an artificial intelligence model to override treating physicians' determinations of medically necessary care for elderly Medicare Advantage patients. The complaint claims the AI model has a known 90% error rate and wrongfully denied patients care. The lawsuit, filed in the U.S. District Court for the District of Minnesota, seeks damages on behalf of the estates of two deceased patients and all others similarly situated.

Company involved

UnitedHealthcare, Inc.

10 source articles · read the reporting →

Mazaheri v Law Society of Ontario (Law Society Tribunal (ON)): AI-hallucinated content in court filing, Adverse Costs Order

AI generated fabricated legal citations in a court filing by Shahryar Mazaheri.

1 source article · read the reporting →

UnitedHealth used ALERT algorithm to limit mental health coverage, regulators found

ProPublica reports that UnitedHealth Group's Optum subsidiary used the ALERT algorithm to flag mental health patients and therapists as outliers, leading to therapy coverage denials. Regulators in California, Massachusetts and New York alleged this breached the federal Mental Health Parity and Addiction Equity Act, and UnitedHealth agreed to restrict the system in those jurisdictions. The company denies wrongdoing and says its programmes are compliant. Affected patients, including Medicaid enrollees, were left to pay out-of-pocket or go without care.

Company involved

UnitedHealth Group

AI system involved

ALERT

5 source articles · read the reporting →

Henry v. Long Island University (E.D. New York): AI-hallucinated content in court filing, Two-year filing-disclosure sanction

The court granted a motion to dismiss and imposed sanctions on plaintiff's counsel for submitting a brief with AI-hallucinated fake cases.

1 source article · read the reporting →

NHS liver transplant algorithm causes longer waits for younger patients

An NHS computer algorithm that prioritises liver transplant patients most likely to die soon has led to younger patients waiting 156 days longer on average than those over 60. The algorithm, launched in 2018, gives a score based on 21 recipient and seven donor parameters. Patients like Sarah Meredith, 31, have been waiting two years for a transplant. NHS Blood and Transplant defends the system as saving lives but acknowledges the stress on younger patients.

Company involved

NHS Blood and Transplant

10 source articles · read the reporting →

Jessica Fuller v. Hyde School, et al. (D. Maine): AI-hallucinated content in court filing, CLE; Firm procedures and certification; Serve…

The AI generated fictitious legal citations that were submitted to the court in a legal filing.

AI system involved

ChatGPT or Claude

1 source article · read the reporting →

Reaves Law Firm, PLLC v. Baker, Donelson, Bearman, Caldwell & Berkowitz, PC, et al. (W.D. Tennessee): AI-hallucinated content in court…

The AI generated fictitious legal authorities and arguments that were filed in a court motion, misleading the court and opposing counsel.

Company involved

Reaves Law Firm, PLLC

1 source article · read the reporting →

Perth hospital group bans ChatGPT after doctor used it for a patient discharge summary

South Metropolitan Health Service, which runs five hospitals in Perth, ordered staff to stop using ChatGPT for work involving patient information after an email said some staff had used the AI chatbot to write medical notes. The service later clarified that only one doctor had used the tool to generate a patient discharge summary and that no confidential patient information had been breached. The Australian Medical Association has called for national regulations to ensure a human is involved in AI-assisted health decisions.

Company involved

South Metropolitan Health Service

AI system involved

ChatGPT

6 source articles · read the reporting →

EviCore denied heart catheterization for patient using algorithm

In fall 2021, Little John Cupp's doctor requested a left heart catheterization exam. EviCore, a company hired by UnitedHealthcare, denied the request twice using an algorithm called 'the dial' that adjusts thresholds for review. The algorithm flagged the request for review, and EviCore's doctors determined it was not medically necessary. Cupp did not receive the procedure and his symptoms continued.

Company involved

EviCore (a Cigna company)

AI system involved

the dial

6 source articles · read the reporting →

IRCC uses AI triage for Temporary Resident Visa applications

Immigration, Refugees and Citizenship Canada (IRCC) uses an AI system called Advanced Analytics to triage Temporary Resident Visa applications from India and China. The system categorizes applications into tiers, with Tier 1 approved automatically and others sent to human officers. Critics allege the system lacks transparency and may introduce bias, leading to visa refusals without clear rationale. The author, a Canadian immigration lawyer, is filing Federal Court cases on behalf of clients affected by refusals.

Company involved

Immigration, Refugees and Citizenship Canada (IRCC)

AI system involved

Advanced Analytics Triage of Overseas Temporary Resident Visa Applications

10 source articles · read the reporting →

Humana sued for using AI to deny seniors rehabilitation care

Health insurer Humana is accused in a class-action lawsuit of using an AI algorithm to systematically deny rehabilitation care to Medicare Advantage patients, despite recommendations from their doctors. The lawsuit, filed on December 12, 2023, alleges that the AI tool restricted medically necessary care. This is the second major health insurer to face legal action over its use of AI to deny care.

Company involved

Humana

9 source articles · read the reporting →

Judge rules lawsuit over UnitedHealth’s AI care denials can move forward - STAT

The system denied necessary medical care to Medicare Advantage beneficiaries.

Company involved

UnitedHealth Group

1 source article · read the reporting →

Air Canada liable for chatbot's misleading bereavement advice

Air Canada was found liable by the B.C. Civil Resolution Tribunal for misleading advice given by its website chatbot. The chatbot told a passenger they could retroactively claim a bereavement rate, but the airline later denied the claim. The tribunal ordered Air Canada to pay $812 in compensation, noting the airline's argument that the chatbot was a separate legal entity was 'remarkable'.

Company involved

Air Canada

AI system involved

Air Canada chatbot

10 source articles · read the reporting →