The record

Where automated decisions went wrong

Incidents gathered from public reporting around the world. Each one links to the articles it came from. None of it is a finding that anyone broke the law.

Reports people file about their own experience are not shown here and never will be without their agreement. Tell us what happened to you.

Clear

116 incidents closest to “Shift Claims Automation” · matched on meaning · public reporting

WF-KU95HS5 Jun 2023

Stack Overflow moderators strike over AI content policy

Volunteer moderators and contributors of Stack Overflow and the Stack Exchange network went on strike on June 5, 2023, protesting a policy by Stack Overflow, Inc. that prohibits removing AI-generated content except in narrow circumstances. The policy is alleged to allow incorrect information and plagiarism to proliferate on the platform. The strike paused moderation activities such as flagging, voting to close or delete posts, and running anti-spam bots. Negotiations concluded on August 2, 2023, with an agreement reached.

Company involved
Stack Overflow, Inc.
AI system involved
Stack Exchange network

10 source articles · read the reporting →

WF-YWUJN81 Oct 2024

Company fires HR team after ATS auto-rejects manager's CV due to filtering error

A company's applicant tracking system (ATS) auto-rejected qualified candidates' resumes for three months because it was filtering for the outdated framework AngularJS instead of the required Angular framework. The manager discovered the flaw by submitting his own CV under a pseudonym and found it was rejected within seconds. After the manager reported the issue to upper management, the company investigated and dismissed half of its HR team. No legal action or regulatory involvement is reported.

4 source articles · read the reporting →

Amazon's routing algorithm forces drivers to cross highways on foot

Amazon's routing algorithm for its Flex app uses 'stop consolidation' to combine deliveries on both sides of the street, requiring drivers to cross busy roads and highways on foot to save time. Drivers in multiple states reported safety risks, including sprinting across four-lane highways. Amazon denied that drivers frequently jaywalk and said it fixes map defects.

Company involved
Amazon
AI system involved
Flex app

9 source articles · read the reporting →

WF-U4WH351 Jun 2020

ScaleFactor reportedly failed to deliver promised automated bookkeeping software

ScaleFactor, an Austin-based startup, is reported to have failed to deliver the automated, real-time bookkeeping tools it promised customers, instead relying on human bookkeepers and a Filipino contract accounting firm. The company told Forbes in June that it was shutting down, initially blaming the pandemic, but Forbes later reported that its problems predated Covid-19. Investors reportedly came to see the company as more of a services business than a software platform, and pulled funding after a pivot to a marketplace model. No legal or regulatory action is reported.

Company involved
ScaleFactor

10 source articles · read the reporting →

WF-ISGR5Y1 Oct 2021

Gorillas workers strike over algorithmic shift scheduling flaws

Workers at Gorillas delivery service in Berlin went on strike after an algorithm used for shift scheduling failed to comply with legal rest periods, causing shifts to be scheduled too close together. The algorithm, part of Project ACE, also generated incorrect warnings for missed shifts, leading to unfair firings. The workers demand full and timely payment of salaries, improved shift plans, and proper equipment. Management promised negotiations but instead hired private security to lock workers out.

Company involved
Gorillas
AI system involved
Project ACE

10 source articles · read the reporting →

Answer.AI tests Devin and reports 14 failures in 20 tasks

Answer.AI's team tested Devin, an autonomous AI coding assistant, on 20 real-world tasks over a month. Devin succeeded in only 3 tasks, failed 14, and was inconclusive in 3. The team found Devin often produced overly complex or hallucinated solutions and could not recognize fundamental blockers. They ultimately decided to stick with tools that allow more human control.

AI system involved
Devin

5 source articles · read the reporting →

IRCC uses AI triage for Temporary Resident Visa applications

Immigration, Refugees and Citizenship Canada (IRCC) uses an AI system called Advanced Analytics to triage Temporary Resident Visa applications from India and China. The system categorizes applications into tiers, with Tier 1 approved automatically and others sent to human officers. Critics allege the system lacks transparency and may introduce bias, leading to visa refusals without clear rationale. The author, a Canadian immigration lawyer, is filing Federal Court cases on behalf of clients affected by refusals.

Company involved
Immigration, Refugees and Citizenship Canada (IRCC)
AI system involved
Advanced Analytics Triage of Overseas Temporary Resident Visa Applications

10 source articles · read the reporting →

Presto Automation uses off-site human agents to double-check AI drive-thru orders

Presto Automation Inc, which markets an AI voice assistant for drive-thru ordering, used off-site human agents in countries including the Philippines to double-check orders in more than 70% of customer interactions, according to SEC filings reported by Bloomberg. The company told Bloomberg that the process helps train its system and should reduce human intervention over time. Presto's drive-thru AI is used in more than 400 restaurants, including Del Taco, Carl's Jr and Checkers, and its stock fell more than 10% after the reports.

Company involved
Presto Automation Inc.

8 source articles · read the reporting →

Cruise alleged to rely on human operators for autonomous driving

A New York Times article alleged that Cruise's autonomous vehicles rely on human operators to achieve autonomous driving, with 1.5 workers per vehicle intervening every 2.5 to five miles. Cruise CEO Kyle Vogt responded on Hacker News, stating that remote assistance is used 2-4% of the time and that many sessions are resolved by the vehicle itself. The company declined an interview request from the NYT.

Company involved
Cruise
AI system involved
Cruise AV

9 source articles · read the reporting →

Teleperformance deploys AI to neutralise Indian call centre agents' accents

Teleperformance, the world's largest call centre operator, has announced it is using AI from Sanas to modify the accents of its Indian employees in real time. The technology, called accent translation, aims to make agents sound more neutral to native English speakers. The company invested $13 million in Sanas and gained exclusive rights. No specific incident of harm has been reported.

Company involved
Teleperformance
AI system involved
Sanas AI

6 source articles · read the reporting →

WF-N8LQF31 Jan 2020

Uber and Amazon algorithms pay different wages for same work

A study by law professor Veena Dubal alleges that Uber and Amazon use AI algorithms to offer different pay rates to gig workers doing identical work. The algorithms are said to calculate the lowest wage a driver will accept based on personal data. Uber denies tailoring individual fares, and the California Labor Commission's lawsuit against Uber and Lyft is ongoing.

Company involved
Uber

10 source articles · read the reporting →

WF-QW9Z288 Mar 2024

NHS plans AI to listen to appointments and generate notes

The UK's National Health Service announced plans to use AI to automatically generate notes from patient appointments. Health Secretary Victoria Atkins said the scheme would reduce admin time. Privacy campaigners raised concerns about data security and accuracy, citing an incident where AI misheard the chief medical officer's name. The Department of Health and Social Care stated that patient confidentiality remains a top priority.

Company involved
National Health Service (NHS)

3 source articles · read the reporting →

WF-YDH39P1 Jan 2013

Swedish welfare agency's AI system flags marginalized groups for fraud investigations

Försäkringskassan, Sweden's Social Insurance Agency, uses an AI risk-scoring system to flag welfare applicants for fraud investigations. The system disproportionately targets women, individuals with foreign backgrounds, low-income earners, and those without university degrees, according to an investigation by Lighthouse Reports and Svenska Dagbladet. Amnesty International has called for the system to be discontinued, citing violations of the right to equality and non-discrimination.

Company involved
Försäkringskassan (Swedish Social Insurance Agency)

6 source articles · read the reporting →

WF-FA32HO1 Mar 2019

Tesla Model 3 crash with truck kills driver using partial automation

At 6:17 a.m. on March 1, 2019, a Tesla Model 3 operating with partial driving automation struck a semitrailer in Delray Beach, Florida, killing the 50-year-old driver. The NTSB determined the probable cause was the truck driver's failure to yield right of way combined with the car driver's inattention due to overreliance on the automation, with Tesla's system design allowing disengagement as a contributing factor.

Company involved
Tesla

10 source articles · read the reporting →

Slack trains AI features on user messages and files by default

Slack uses user messages, files, and data to train its machine learning features such as channel recommendations and emoji suggestions. Users are opted in by default and cannot individually opt out; only workspace administrators can request exclusion via email. A user publicly criticized the practice, and Slack acknowledged the policy but did not change it.

Company involved
Slack

10 source articles · read the reporting →

WF-14BFFD28 May 2024

Klarna CEO faces backlash for using AI to halve marketing team

Klarna CEO Sebastian Siemiatkowski tweeted that the company saved $10 million by using generative AI to reduce its in-house marketing team to half its previous size and cut external agency spending by 25%. The post sparked criticism on social media, with commenters calling the remarks 'ghoulish' and highlighting concerns about AI's impact on jobs. Klarna previously laid off 700 workers in 2022 and reduced headcount by 23% by end of 2023.

Company involved
Klarna
AI system involved
Midjourney, DALL-E, Adobe Firefly

6 source articles · read the reporting →

LinkedIn removes AI 'co-worker' accounts that were seeking jobs

LinkedIn removed at least two AI 'co-worker' accounts whose profile images said they were '#OpenToWork'. One account named Ella claimed it would outperform any social media team and needed no coffee breaks. The article does not specify further consequences.

Company involved
LinkedIn
AI system involved
AI 'co-worker' account 'Ella'

4 source articles · read the reporting →

WF-RX9YRU9 Jul 2024

Lattice cancels plan to give AI digital workers employee records after backlash

Lattice, an HR software company, announced on July 9th that it would give AI digital workers official employee records. After strong backlash from HR professionals and others on LinkedIn, the company canceled the feature on July 12th, stating it 'will not further pursue digital workers in the product.' The feature was intended to manage AI bots such as Devin and Piper, but the company reversed course.

Company involved
Lattice
AI system involved
Lattice

6 source articles · read the reporting →

Driver relying on "smart driving" on highway crashes after system suddenly disengages near truck

司机依赖“智驾 ”跑高速,逼近大货车时“智驾”突然退出引发车祸 - 手机新浪网

A driver was using a smart driving system on an expressway. As the vehicle approached a large truck, the system unexpectedly disengaged, leading to a crash.

1 source article · read the reporting →

DWP algorithm approved Kickstart gateways with no trading history or based abroad

An FE Week investigation found that the Department for Work and Pensions (DWP) approved dozens of companies as Kickstart gateways through automated due diligence checks using the Cabinet Office Spotlight Tool, although some had little or no trading history or were based abroad. The DWP said gateways were subject to stringent checks and later said human checks were also used. After the findings were shared with the Treasury and the DWP, the department stopped taking gateway applications and scrapped the requirement for small employers to use gateways from 3 February.

Company involved
Department for Work and Pensions
AI system involved
Cabinet Office Spotlight Tool

3 source articles · read the reporting →

Amazon cuts 14,000 corporate jobs in bet on AI

Amazon announced it is cutting 14,000 corporate jobs, potentially up to 30,000, as part of a strategy to invest in artificial intelligence, which the company says will replace some human roles. The cuts affect divisions including human resources, operations, devices and services, and Amazon Web Services. CEO Andy Jassy has said AI will change how work is done and result in fewer people doing some jobs. The company has not commented on the impact on its UK workforce.

Company involved
Amazon

5 source articles · read the reporting →

WF-KLSRXK24 Feb 2021

Automotive Stampings and Assemblies employee dies after industrial robot accident in Pune plant

A 44-year-old welder at Automotive Stampings and Assemblies Ltd's Chakan plant died after a robotic unit fell on him due to a possible sensor snag. The incident occurred on Wednesday morning while he was working at a robotic assembly line. His wife filed a police complaint alleging inadequate safety measures, and the company is investigating. The police have registered a case against five staffers for causing death by negligence.

Company involved
Automotive Stampings and Assemblies Ltd

1 source article · read the reporting →

WF-JRPF9K30 Jul 2024

Microsoft Dynamics 365 Field Service AI singles out workers in performance predictions

A report by Cracked Labs found that Microsoft's Dynamics 365 Field Service software uses AI to generate performance metrics and predict task durations, singling out individual workers. The AI predictions can be influenced by the worker's identity, such as increasing or decreasing estimated duration. Microsoft stated the system is not intended for employment decisions and is not a surveillance tool, but the report raises concerns about potential misuse for worker monitoring.

Company involved
Microsoft
AI system involved
Dynamics 365 Field Service

6 source articles · read the reporting →

WF-I4OTS330 Aug 2024

Edinburgh Airport AI trial gives passengers different parking prices

Edinburgh Airport admitted it was trialling an AI system that randomly set higher or lower parking prices for online bookers, with customers receiving different quotes for the same service. The Scottish Passenger Agents Association called for the trial to be conducted in controlled 'lab' conditions rather than on the public. The airport said the AI's pricing closely matched staff-set prices and the findings would evaluate whether to adopt the system.

Company involved
Edinburgh Airport
AI system involved
AI trial for parking pricing

3 source articles · read the reporting →

← Newerpage 4 of 5Older →