Where automated decisions went wrong

These workers say an AI hiring platform discriminated against them. But who should pay?

The system screened out Black and other applicants, denying them job opportunities.

1 source article · read the reporting →

Black Job Seekers Seek Class Action Against Workday - BIN: Black Information Network

Workday's AI hiring tools screened and rejected job applications from Derek Mobley and other Black applicants, women, people over 40, and people with disabilities.

Company involved

Workday

AI system involved

Workday Recruiting

1 source article · read the reporting →

Cigna uses algorithm to deny medical claims without doctor review

Cigna, one of the largest health insurers, used an algorithm called PXDX to automatically deny medical claims. The system flagged mismatches between diagnoses and procedures, and company doctors signed off on denials in batches without reviewing patient files. One patient, Nick van Terheyden, was denied coverage for a $350 vitamin D test that his doctor had ordered. Former employees said the system saved Cigna billions of dollars but left patients with unexpected bills.

Company involved

Cigna

AI system involved

PXDX

10 source articles · read the reporting →

Workers Look To Lock In Sweeping Workday AI Bias Class - Law360

The system screened job applicants and rejected them, allegedly discriminating based on age and sex.

AI system involved

Workday AI tools

1 source article · read the reporting →

Stockton police disclose Flock misuse, outline oversight changes after employee’s departure - Stocktonia News

Stockton police disclose Flock misuse, outline oversight changes after employee’s departure Stocktonia News

1 source article · read the reporting →

US Federal Court Rejects Workday's Arguments, Allows AI Hiring Lawsuit to Proceed

米連邦地裁がWorkdayの主張を退け、AI採用訴訟の継続を認める - Межа. Новини України.

A federal court in San Francisco declined Workday's request to dismiss a lawsuit alleging its AI-powered hiring software discriminates against applicants with disabilities and other protected groups. Judge Rita Lin ruled that California anti-discrimination law may apply and allowed claims that the software screens candidates using proxy indicators such as employment gaps to move forward. Workday maintains its tools evaluate only qualifications and do not make hiring decisions.

Company involved

Workday

AI system involved

Workday

1 source article · read the reporting →

South Africa's Supreme Court of Appeal Considers Sassa Algorithm Case

Верховный апелляционный суд ЮАР рассмотрел дело об алгоритмах Sassa - UA.NEWS

The Supreme Court of Appeal heard an appeal on 25 August about Sassa's digital application system for the SRD grant. The system only accepts online applications and uses automated bank account checks that may deny grants to people whose accounts receive deposits that are not regular income. The Global Center on AI Governance submitted that automated decisions must uphold constitutional rights and be fair, non-discriminatory, and suited to South Africa's informal economy.

Company involved

South African Social Security Agency (Sassa)

AI system involved

Sassa digital application system

1 source article · read the reporting →

UberEats Sued for Fraud Over Stacked Orders and Halved Hours Allegedly Shorting Pay

疊單、工時砍半涉短付報酬！UberEats挨告詐欺 - TVBS新聞網

A new law effective July 21 sets minimum pay per delivery order. Unions accuse UberEats of underpaying workers by misinterpreting these rules, and they filed a fraud complaint with prosecutors. They say the platform's actions caused financial losses for delivery workers.

Company involved

Uber Eats

1 source article · read the reporting →

Uber's Secretive 'Upfront Fares' Algorithm Cuts Driver Pay in Many Cities

Uber introduced an opaque algorithm called 'Upfront Fares' to calculate driver pay in 24 U.S. cities, replacing the previous time-and-distance-based rate. Drivers report lower and unpredictable earnings, with Uber taking a larger cut of fares. The company claims the change gives drivers more control, but critics say the secrecy makes it impossible to verify fair pay. The algorithm may learn the lowest rate drivers will accept, potentially driving down wages.

Company involved

Uber

AI system involved

Upfront Fares

3 source articles · read the reporting →

Rotterdam's Welfare Fraud Algorithm Discriminated by Gender and Ethnicity

The city of Rotterdam deployed a machine learning algorithm built by Accenture to flag welfare recipients for fraud investigation. The system used personal data including gender, language, and subjective caseworker notes to generate risk scores, leading to investigations that disproportionately targeted women and migrants. An external review found the algorithm discriminatory and inaccurate, prompting the city to suspend its use in 2021. The system's opacity made it nearly impossible for those flagged to challenge the decisions.

Company involved

City of Rotterdam

6 source articles · read the reporting →

Michigan's MiDAS system falsely accused 34,000 of unemployment fraud

Michigan's Unemployment Insurance Agency (UIA) deployed the Michigan Integrated Data Automated System (MiDAS) to automate fraud detection. The system algorithmically adjudicated over 40,000 fraud cases between October 2013 and September 2015, with 85 percent of those resulting in incorrect fraud determinations. The false accusations led to severe financial penalties, ruined credit, and homelessness for many victims. A class-action lawsuit was filed, and while the state has apologised and returned fines, it has fought against compensation claims, though legislators have promised to seek at least $30 million for those affected.

Company involved

Michigan Unemployment Insurance Agency

AI system involved

Michigan Integrated Data Automated System (MiDAS)

10 source articles · read the reporting →

CFPB Acts Against Hello Digit for Faulty Savings Algorithm Causing Overdrafts

Hello Digit, a fintech company, used an automated savings algorithm that made transfers from consumers' checking accounts, falsely guaranteeing no overdrafts. The algorithm caused customers to incur overdraft fees, and the company often denied reimbursement requests. The CFPB found that Hello Digit engaged in deceptive practices and ordered the company to pay redress to harmed consumers and a $2.7 million fine.

Company involved

Hello Digit, LLC

AI system involved

Hello Digit app

1 source article · read the reporting →

Amazon tracked every minute of JFK8 warehouse workers' shifts, documents show

Internal documents show Amazon's Time Off Task tool tracked every minute of JFK8 warehouse workers' shifts using handheld scanners and identified a 'top offender' per shift. Managers questioned those workers about each period of inactivity, including bathroom breaks, and issued warnings or fired them for 30 to 120 minutes of time off task. The article reports that 18 workers were terminated between January and February 2020, and workers said they skipped water and bathroom breaks for fear of discipline. Amazon did not respond to a request for comment.

Company involved

Amazon

AI system involved

Time Off Task Tool

3 source articles · read the reporting →

Kronos Algorithmic Scheduling Blamed for Erratic Work Hours for Retail Workers

Kronos workforce management software is used by major retail chains to algorithmically generate employee schedules. The software has been criticised for causing erratic and unpredictable work hours, leading to financial instability and emotional distress for low-wage workers. Following an investigation by the New York attorney general and negative press, Kronos announced new features aimed at improving schedule stability and worker control, though critics argue the changes are optional and lack enforcement mechanisms.

Company involved

Various retailers (e.g., Gap, Target, J.C. Penney, Starbucks)

AI system involved

Kronos scheduling software

9 source articles · read the reporting →

Jordan's National Aid Fund Algorithm Denies Cash Assistance to Vulnerable Households

Human Rights Watch reported that Jordan’s Takaful cash transfer programme, run by the National Aid Fund and financed by the World Bank, uses an algorithm to rank households by estimated economic vulnerability and select recipients. The report found that the algorithm relies on inaccurate self-reported income and questionable indicators such as car ownership and utility use, causing eligible poor families to be denied benefits or receive reduced payments. The World Bank and NAF defended the programme and its targeting approach.

Company involved

National Aid Fund (NAF)

AI system involved

Takaful (Unified Cash Transfer Program)

3 source articles · read the reporting →

Idaho Medicaid used secret formula to cut assistance for disabled adults

The Idaho Department of Health and Welfare used a secret formula to drastically cut Medicaid assistance for adults with developmental disabilities, claiming the formula was a trade secret. The ACLU of Idaho filed a class action lawsuit, K.W. v. Armstrong, in 2012, alleging the cuts violated due process. A federal court struck down the formula in 2016 and ordered a new system, but as of 2024, a fair system has not yet been implemented.

Company involved

Idaho Department of Health and Welfare

10 source articles · read the reporting →

Ibrahim Diallo fired by automated HR system, locked out for three weeks

Ibrahim Diallo, a software engineer in California, was automatically terminated by an HR system after a script triggered by a missed contract renewal revoked his access and disabled his key card. Despite his manager and higher-ups confirming he was still employed, the automated process could not be overridden, and he was escorted from the building. It took three weeks for the company to resolve the issue by rehiring him as a new employee, after which he quit. The incident highlights the risks of fully automated termination systems without human override.

10 source articles · read the reporting →

France: CNAF's discriminatory risk-scoring algorithm must be stopped

Amnesty International and coalition partners filed a complaint with the Council of State against CNAF's risk-scoring algorithm used to detect benefit overpayments. The algorithm assigns risk scores based on criteria that discriminate against vulnerable groups, including those with disabilities, single parents, and low-income households. The complaint alleges the system violates human rights to equality and privacy. The EU AI Act's social scoring ban may apply, but its definition remains unclear.

Company involved

CNAF

2 source articles · read the reporting →

UberEats algorithm change causes pay cuts and financial hardship for car drivers

A survey by the Transport Workers' Union found that UberEats car drivers experienced a 50% pay cut after the company changed its delivery allocation algorithm to preference cyclists and scooter riders. Drivers reported bankruptcy, homelessness, and inability to support their families. The TWU has called for regulation of the gig economy and delivered a letter to the NSW shadow minister.

Company involved

UberEats

AI system involved

UberEats platform

10 source articles · read the reporting →

Shipt's new V2 algorithm reduces pay for 41% of workers, study finds

A study by MIT Media Lab and Coworker.org found that Shipt's new V2 payment algorithm reduced pay for 41% of workers by an average of 11% per shop. The algorithm, rolled out in September 2020, replaced a transparent payment system with a black-box model. Workers pooled data to analyze the impact, revealing uneven distribution of earnings.

Company involved

Shipt

AI system involved

V2

10 source articles · read the reporting →

DWP's Universal Credit algorithm causes underpayment and hardship for claimants

The Department for Work and Pensions (DWP) uses an automated means-testing algorithm in its Universal Credit system that miscalculates earnings due to a design flaw, leading to underpayments and fluctuating benefit amounts. Claimants such as Janet R. received payments hundreds of pounds short, forcing them into debt and foodbank use. A Court of Appeal ruling in June 2020 ordered the DWP to fix the flaw, but the department had not yet fully implemented the remedy at the time of writing.

Company involved

Department for Work and Pensions

AI system involved

Universal Credit

10 source articles · read the reporting →

Allstate's proposed retention model charged big spenders more in Maryland

Allstate proposed an auto insurance rate adjustment plan in Maryland using a retention model algorithm that varied premium changes based on existing premiums. The algorithm would have increased rates up to 20% for customers already paying the highest premiums while capping increases at 5% for others, and would have denied meaningful discounts to thousands owed reductions. The Maryland Insurance Administration rejected the plan as discriminatory, but Allstate claims it was withdrawn and has continued proposing similar models elsewhere. The proposal never took effect in Maryland.

Company involved

Allstate Corporation

AI system involved

retention model

9 source articles · read the reporting →

DoorDash changes tipping policy after backlash over worker tips

DoorDash, a food delivery company, changed its tipping policy after backlash. The company had been using customer tips to offset its base pay to delivery workers, meaning workers did not receive the full tips. After public outcry and promises from CEO Tony Xu, DoorDash announced a new model where workers will keep all tips, effective next month.

Company involved

DoorDash

10 source articles · read the reporting →

Uber algorithm reinforced gender pay gap in compensation offers to new hires

Uber used an algorithm to determine compensation offers for new hires, allegedly paying women less than men for similar roles. The algorithm was designed to protect existing shareholders and save money, but it reinforced existing gender pay gaps. The system is now being altered, according to The Information.

Company involved

Uber

7 source articles · read the reporting →