Where automated decisions went wrong

Uber fined €825 million for suspending drivers through algorithms without human review - MediaNama

Uber fined €825 million for suspending drivers through algorithms without human review MediaNama

1 source article · read the reporting →

Uber set for €825mn Dutch fine over automating driver suspensions - Financial Times

The system automatically suspended Uber drivers.

Company involved

Uber

1 source article · read the reporting →

Uber fined nearly 825 million euros for automated driver blocking - Autoriteit Persoonsgegevens

Uber's automated system blocked drivers from the platform

Company involved

Uber

1 source article · read the reporting →

Apple, Amazon, Meta, Microsoft, Nvidia, and Samsung face class action suits over AI voice training - cryptobriefing.com

The AI voice models were trained on the plaintiffs' voice recordings without their consent.

Company involved

Apple, Amazon, Meta, Microsoft, Nvidia, Samsung, Alphabet, Adobe, ElevenLabs

1 source article · read the reporting →

Safe City Malta board renewed despite planned shutdown of surveillance programme

The Maltese government has appointed a new board to run Safe City Malta, the Huawei-developed facial-recognition surveillance project, despite saying in January that the company would be wound down. The board, including Omar Debono, Joseph Cauchi and Nadine Mizzi, was appointed for another year until June 2024. The government had earlier dropped the facial-recognition element over concerns that it would breach Maltese and EU law, and the article reports that nothing has been delivered.

Company involved

Safe City Malta

AI system involved

Safe City Malta

10 source articles · read the reporting →

SiriusXM Radio faces class action on AI discrimination

The AI hiring tool screened and rejected job applicants, including the plaintiff, based on race-related data points.

Company involved

SiriusXM Radio

AI system involved

iCims

1 source article · read the reporting →

TRT-RS's Galileu AI Detects Prompt Injection Attempt in Legal Petition

The Galileu AI system, developed by the Tribunal Regional do Trabalho da 4ª Região (TRT-RS) and nationalised by the Conselho Superior da Justiça do Trabalho (CSJT), detected a prompt injection attempt in a petition filed at the 3rd Labour Court of Parauapebas, Pará. The system alerted the magistrate, who reviewed the content and made a decision based on human verification, in line with judicial AI supervision requirements. The court reported that the system prevented the malicious content from being processed and highlighted the importance of institutional AI tools with security measures.

Company involved

Tribunal Regional do Trabalho da 4ª Região

AI system involved

Galileu

1 source article · read the reporting →

Court orders suspension of facial recognition in São Paulo metro

A court in São Paulo ordered the suspension of a facial recognition system in the city's metro stations following a lawsuit by civil society groups. The system, SecurOS by ISS and operated by ViaQuatro, was capturing biometric data of millions of daily users without adequate transparency or risk assessment. The court also barred the metro operator, METRO, from installing new biometric equipment and imposed daily fines for non-compliance. METRO stated it would appeal the ruling and prove compliance with data protection regulations.

Company involved

Companhia do Metropolitano de São Paulo (METRO)

AI system involved

SecurOS

3 source articles · read the reporting →

Ryanair requires facial recognition for customers booking via online travel agents

noyb filed a complaint against Ryanair with the Spanish Data Protection Authority (AEPD) on 27 July 2023. A customer who booked a Ryanair flight through online travel agency eDreams was required to undergo a facial recognition verification process or go to the check-in counter more than two hours before departure. The customer was charged a small fee for the verification. Ryanair outsources the facial recognition to GetID. noyb alleges that the process violates GDPR because consent is not valid and the purpose is to discourage bookings through third-party agents.

Company involved

Ryanair

AI system involved

GetID

10 source articles · read the reporting →

Google AI directs traveller to scam customer service number

Alex Rivlin used Google to find a cruise company's customer service number while booking a shuttle for a European holiday. Google's AI surfaced a number that connected him to a knowledgeable-sounding representative, to whom he gave his credit card details. The number was a scam, representing a new AI-enabled twist on a long-running travel fraud. The article warns readers about this evolving threat.

Company involved

Google

AI system involved

Google AI Overviews

1 source article · read the reporting →

Gamma AI Presentation Tool Exploited in Multi-Stage Phishing Campaign

Threat actors used Gamma, an AI-powered presentation builder, to host a page that redirected recipients to a fake Microsoft SharePoint login portal. Emails sent from compromised legitimate accounts passed authentication checks, while a Cloudflare Turnstile blocked automated security scanners. An adversary-in-the-middle framework validated credentials in real time and captured session cookies, enabling multi-factor authentication bypass on Microsoft accounts. Abnormal reported the campaign on 15 April 2025.

AI system involved

Gamma

7 source articles · read the reporting →

Amnesty International reveals Serbian spyware targeting journalists and activists

Amnesty International's Security Lab found that Serbian authorities used Cellebrite tools and a previously unknown spyware named 'NoviSpy' to covertly infect the phones of independent journalist Slaviša Milanov and several activists. The infections occurred while devices were unattended during police or BIA interviews. The report alleges this is part of a wider crackdown on civil society, violating rights to privacy and free expression.

Company involved

Serbian Security Information Agency (BIA)

AI system involved

NoviSpy

7 source articles · read the reporting →

KPMG drops AI report after false case studies exposed

KPMG removed a global report on AI after multiple case studies were found to be inaccurate and apparently generated from AI hallucinations. The report falsely claimed that UBS, NHS Greater Manchester, Swiss Federal Railways and Transport for London used AI agents in various ways. The companies denied the claims, and KPMG took the report down while reviewing its production.

Company involved

KPMG

3 source articles · read the reporting →

UDR Sued Over Alleged Use of RealPage Algorithm to Set Rents in San Diego

A class action lawsuit alleges that UDR, Inc. used RealPage's YieldStar algorithmic pricing software to set rental rates and occupancy levels for its San Diego properties, in violation of a local ordinance. The suit claims the software relied on nonpublic competitor data, contributing to inflated rents and making housing less affordable. The lawsuit, filed in July 2026, seeks to represent all affected tenants. UDR has previously acknowledged using YieldStar as one tool among others in its decision-making.

Company involved

UDR, Inc.

AI system involved

RealPage YieldStar

1 source article · read the reporting →

EEOC sues iTutorGroup for automated rejection of older tutor applicants

The US Equal Employment Opportunity Commission (EEOC) alleges that tutoring provider iTutorGroup programmed its recruitment software automatically to reject female applicants aged 55 or older and male applicants aged 60 or older. More than 200 qualified US-based applicants were denied work in 2020. The EEOC has filed a lawsuit in the Eastern District of New York seeking back pay and damages.

Company involved

iTutorGroup

10 source articles · read the reporting →

AMS algorithm lacks transparency and may discriminate against job seekers

The Austrian Public Employment Service (AMS) uses an algorithm to classify job seekers into categories A, B, and C, determining their access to benefits and training. Scientists from TU Wien, WU Wien, and University of Vienna have criticised the algorithm for lacking transparency, as only two of 96 model variants have been published. They allege that the system may discriminate against women and people with migration background, and that job seekers are not informed about how the algorithm works or given a chance to appeal.

Company involved

AMS (Arbeitsmarktservice Österreich)

AI system involved

AMS-Algorithmus

10 source articles · read the reporting →

Southwest Airlines holiday meltdown due to crew scheduling software failure

In December 2022, Southwest Airlines experienced a catastrophic operational meltdown after its crew scheduling software failed to assign pilots and flight attendants to flights during a winter storm. The system, SkySolver and Crew Web Access, could not handle the volume of schedule changes, leading to over 15,800 flight cancellations and stranding thousands of passengers and crew. Southwest's CEO acknowledged the technology failure and promised upgrades, but the incident remained unresolved at the time of reporting.

Company involved

Southwest Airlines

AI system involved

SkySolver and Crew Web Access

10 source articles · read the reporting →

Austrian court lifts ban on AMS job-chance prediction algorithm

The Austrian Federal Administrative Court overturned a ban by the Data Protection Authority on the AMS algorithm, which predicts job chances of unemployed people. The system, trained on historical data including age, gender, and nationality, categorises individuals as high, medium or low chance of re-employment. Critics argue it discriminates against women and mothers, and that the data is now outdated due to the COVID-19 pandemic. The court ruled that the algorithm is lawful as long as a human advisor makes the final decision on training support.

Company involved

Arbeitsmarktservice (AMS)

AI system involved

AMS-Algorithmus

5 source articles · read the reporting →

Finnish recruitment company Digital Minds used AI to analyze job applicants' messages, prompting data protection investigation

Digital Minds, a Finnish recruitment company founded by psychologists, used IBM Watson AI to analyze job applicants' social media and email messages for personality assessments. The company obtained written consent but the Finnish Data Protection Ombudsman launched an investigation, suspecting violations of data protection laws and the secrecy of correspondence. The service was used on fewer than ten applicants and has been paused pending the investigation.

Company involved

Digital Minds

AI system involved

IBM Watson

9 source articles · read the reporting →

Teleperformance plans AI webcam surveillance for home-working staff

Teleperformance, a global call centre company, told some staff it would install AI-powered webcams to monitor home-working infractions such as eating, phone use, or leaving desks. The system would randomly scan for breaches and send alerts to managers. After the Guardian inquired, the company said the remote scans would not be used in the UK, but the plan raised concerns from unions and MPs about invasive surveillance.

Company involved

Teleperformance

10 source articles · read the reporting →

Recurso de Suplicación 0005472/2025 (T.S.X. Galicia): AI-hallucinated content in court filing, Bar Referral

The AI generated false legal citations that were included in a court filing, misleading the court and potentially harming the client's case.

1 source article · read the reporting →

ViaQuatro's facial recognition system in São Paulo metro challenged in court

In April 2018, ViaQuatro installed the Digital Interactive Doors System, developed by AdMobilize, on the São Paulo metro's yellow line. The system used cameras to detect passengers' faces and claimed to infer their emotion, age, and gender in order to target advertisements. The Brazilian Institute of Consumer Protection (IDEC) filed a public civil action alleging that the system violated consumer and data protection laws by processing biometric data without consent and making pseudoscientific and discriminatory inferences. A judge ordered the cameras removed in August 2018, and the case is pending a final ruling.

Company involved

ViaQuatro

AI system involved

Digital Interactive Doors System (DID system)

10 source articles · read the reporting →

GSMA fined €200,000 for facial recognition privacy violation at MWC

In 2021, Dr Anastasia Dedyukhina was invited to speak at Mobile World Congress in Barcelona. The organizers required her to upload biometric data (passport) online for identity verification, which she refused. She was then forced to attend virtually. She filed a complaint with the Spanish data protection agency (AEPD) along with Adam Leon Smith. Two years later, the AEPD fined GSMA €200,000 for infringing the privacy of nearly 20,000 attendees through the use of facial recognition without proper justification.

Company involved

GSMA

10 source articles · read the reporting →

Italian privacy regulator fines Foodinho €2.6 million for algorithmic discrimination against riders

The Italian Data Protection Authority (Garante) fined Foodinho S.r.l., a subsidiary of GlovoApp23, €2.6 million for privacy violations related to its algorithmic management of food delivery riders. The Garante found that the company's digital platform used algorithms to assign orders and rate riders without adequate transparency, human oversight, or the right to contest decisions. The system allegedly penalized riders who did not accept orders quickly, leading to reduced work opportunities and exclusion from the platform. The Garante ordered Foodinho to implement corrective measures within 60 days and to overhaul the algorithms within 90 days.

Company involved

Foodinho S.r.l.

10 source articles · read the reporting →