Where automated decisions went wrong

AMS algorithm lacks transparency and may discriminate against job seekers

The Austrian Public Employment Service (AMS) uses an algorithm to classify job seekers into categories A, B, and C, determining their access to benefits and training. Scientists from TU Wien, WU Wien, and University of Vienna have criticised the algorithm for lacking transparency, as only two of 96 model variants have been published. They allege that the system may discriminate against women and people with migration background, and that job seekers are not informed about how the algorithm works or given a chance to appeal.

Company involved

AMS (Arbeitsmarktservice Österreich)

AI system involved

AMS-Algorithmus

10 source articles · read the reporting →

CNIL says facial recognition trial in French high schools is illegal

The French data protection authority (CNIL) issued a non-binding opinion that a planned trial of facial recognition at two high schools in Nice and Marseille is illegal under the GDPR. The trial, approved by the South Region in December 2018, involved biometric portals to identify students. Digital rights groups, parents and teachers' unions opposed the experiment and filed a lawsuit. The CNIL stated that the system was too intrusive and that less intrusive methods could achieve the same purpose.

Company involved

South Region (Région Sud)

AI system involved

Cisco facial recognition system

10 source articles · read the reporting →

Facebook approved harmful ads targeting teens in TTP experiment

The Tech Transparency Project (TTP) submitted six test ads to Facebook promoting pill parties, anorexia, alcohol, smoking, gambling, and dating services, targeting users aged 13-17. Facebook approved all ads within hours, despite its policies prohibiting such content. The ads never ran as TTP did not allow them to be published. Facebook stated it has measures in place to review ads.

Company involved

Facebook

AI system involved

Facebook Ads

10 source articles · read the reporting →

UberEats algorithm change causes pay cuts and financial hardship for car drivers

A survey by the Transport Workers' Union found that UberEats car drivers experienced a 50% pay cut after the company changed its delivery allocation algorithm to preference cyclists and scooter riders. Drivers reported bankruptcy, homelessness, and inability to support their families. The TWU has called for regulation of the gig economy and delivered a letter to the NSW shadow minister.

Company involved

UberEats

AI system involved

UberEats platform

10 source articles · read the reporting →

Shipt's new V2 algorithm reduces pay for 41% of workers, study finds

A study by MIT Media Lab and Coworker.org found that Shipt's new V2 payment algorithm reduced pay for 41% of workers by an average of 11% per shop. The algorithm, rolled out in September 2020, replaced a transparent payment system with a black-box model. Workers pooled data to analyze the impact, revealing uneven distribution of earnings.

Company involved

Shipt

AI system involved

V2

10 source articles · read the reporting →

Belgian political party sp.a created deep fake video of President Trump

In May 2018, the Belgian political party sp.a released a deep fake video appearing to show President Donald Trump telling Belgium to withdraw from the Paris Climate Agreement. The video was created using lip-synching technology and was intended as a parody to promote the party's climate petition. The party confirmed the video was a deep fake and said it was legally sound.

Company involved

sp.a (Socialistische Partij Anders)

10 source articles · read the reporting →

Finnish recruitment company Digital Minds used AI to analyze job applicants' messages, prompting data protection investigation

Digital Minds, a Finnish recruitment company founded by psychologists, used IBM Watson AI to analyze job applicants' social media and email messages for personality assessments. The company obtained written consent but the Finnish Data Protection Ombudsman launched an investigation, suspecting violations of data protection laws and the secrecy of correspondence. The service was used on fewer than ten applicants and has been paused pending the investigation.

Company involved

Digital Minds

AI system involved

IBM Watson

9 source articles · read the reporting →

Teleperformance plans AI webcam surveillance for home-working staff

Teleperformance, a global call centre company, told some staff it would install AI-powered webcams to monitor home-working infractions such as eating, phone use, or leaving desks. The system would randomly scan for breaches and send alerts to managers. After the Guardian inquired, the company said the remote scans would not be used in the UK, but the plan raised concerns from unions and MPs about invasive surveillance.

Company involved

Teleperformance

10 source articles · read the reporting →

ViaQuatro's facial recognition system in São Paulo metro challenged in court

In April 2018, ViaQuatro installed the Digital Interactive Doors System, developed by AdMobilize, on the São Paulo metro's yellow line. The system used cameras to detect passengers' faces and claimed to infer their emotion, age, and gender in order to target advertisements. The Brazilian Institute of Consumer Protection (IDEC) filed a public civil action alleging that the system violated consumer and data protection laws by processing biometric data without consent and making pseudoscientific and discriminatory inferences. A judge ordered the cameras removed in August 2018, and the case is pending a final ruling.

Company involved

ViaQuatro

AI system involved

Digital Interactive Doors System (DID system)

10 source articles · read the reporting →

Italian privacy regulator fines Foodinho €2.6 million for algorithmic discrimination against riders

The Italian Data Protection Authority (Garante) fined Foodinho S.r.l., a subsidiary of GlovoApp23, €2.6 million for privacy violations related to its algorithmic management of food delivery riders. The Garante found that the company's digital platform used algorithms to assign orders and rate riders without adequate transparency, human oversight, or the right to contest decisions. The system allegedly penalized riders who did not accept orders quickly, leading to reduced work opportunities and exclusion from the platform. The Garante ordered Foodinho to implement corrective measures within 60 days and to overhaul the algorithms within 90 days.

Company involved

Foodinho S.r.l.

10 source articles · read the reporting →

Plastic Forte fined for using facial recognition on workers without notice

The Spanish data protection agency AEPD fined Plastic Forte, a plastics manufacturer in Alicante, €20,000 for using facial recognition to record employees' working hours without informing them. A worker requested information about his personal data and discovered the biometric processing. The company initially argued it was only for time tracking but later acknowledged responsibility, resulting in a reduced fine of €12,000. The AEPD deemed facial recognition for time control as highly intrusive and requiring prior impact assessment.

Company involved

Plastic Forte

7 source articles · read the reporting →

Company fires HR team after ATS auto-rejects manager's CV due to filtering error

A company's applicant tracking system (ATS) auto-rejected qualified candidates' resumes for three months because it was filtering for the outdated framework AngularJS instead of the required Angular framework. The manager discovered the flaw by submitting his own CV under a pseudonym and found it was rejected within seconds. After the manager reported the issue to upper management, the company investigated and dismissed half of its HR team. No legal action or regulatory involvement is reported.

4 source articles · read the reporting →

ScaleFactor reportedly failed to deliver promised automated bookkeeping software

ScaleFactor, an Austin-based startup, is reported to have failed to deliver the automated, real-time bookkeeping tools it promised customers, instead relying on human bookkeepers and a Filipino contract accounting firm. The company told Forbes in June that it was shutting down, initially blaming the pandemic, but Forbes later reported that its problems predated Covid-19. Investors reportedly came to see the company as more of a services business than a software platform, and pulled funding after a pivot to a marketplace model. No legal or regulatory action is reported.

Company involved

ScaleFactor

10 source articles · read the reporting →

Stable Diffusion amplifies racial and gender stereotypes in generated images

An analysis by Bloomberg of over 5,000 images generated by Stability AI's Stable Diffusion found that the text-to-image model amplifies racial and gender stereotypes. The model overrepresented lighter-skinned men in high-paying jobs and darker-skinned people in low-paying jobs, and underrepresented women in positions of power. Stability AI acknowledged the inherent biases in its models and stated it is working on mitigation.

Company involved

Stability AI

AI system involved

Stable Diffusion

8 source articles · read the reporting →

Spanish Supreme Court orders release of BOSCO algorithm code for social electricity bonus

The Spanish NGO Civio won a Supreme Court case forcing the government to release the source code of BOSCO, the algorithm that decides eligibility for the social electricity bonus (bono social eléctrico). Civio had demonstrated in 2019 that BOSCO contained serious errors that denied the benefit to vulnerable people who met the requirements. The government had refused to disclose the code, citing intellectual property. The Supreme Court ruled that transparency must prevail, setting a precedent for public access to automated decision-making systems.

Company involved

Ministerio para la Transición Ecológica (Gobierno de España)

AI system involved

BOSCO

10 source articles · read the reporting →

Gorillas workers strike over algorithmic shift scheduling flaws

Workers at Gorillas delivery service in Berlin went on strike after an algorithm used for shift scheduling failed to comply with legal rest periods, causing shifts to be scheduled too close together. The algorithm, part of Project ACE, also generated incorrect warnings for missed shifts, leading to unfair firings. The workers demand full and timely payment of salaries, improved shift plans, and proper equipment. Management promised negotiations but instead hired private security to lock workers out.

Company involved

Gorillas

AI system involved

Project ACE

10 source articles · read the reporting →

Presto Automation uses off-site human agents to double-check AI drive-thru orders

Presto Automation Inc, which markets an AI voice assistant for drive-thru ordering, used off-site human agents in countries including the Philippines to double-check orders in more than 70% of customer interactions, according to SEC filings reported by Bloomberg. The company told Bloomberg that the process helps train its system and should reduce human intervention over time. Presto's drive-thru AI is used in more than 400 restaurants, including Del Taco, Carl's Jr and Checkers, and its stock fell more than 10% after the reports.

Company involved

Presto Automation Inc.

8 source articles · read the reporting →

Teleperformance deploys AI to neutralise Indian call centre agents' accents

Teleperformance, the world's largest call centre operator, has announced it is using AI from Sanas to modify the accents of its Indian employees in real time. The technology, called accent translation, aims to make agents sound more neutral to native English speakers. The company invested $13 million in Sanas and gained exclusive rights. No specific incident of harm has been reported.

Company involved

Teleperformance

AI system involved

Sanas AI

6 source articles · read the reporting →

Amazon France fined €32m for illegal worker surveillance

Amazon France Logistique was fined €32m (£27m) by France's data protection authority, the CNIL, for illegally monitoring warehouse workers through handheld scanners. The system tracked scanning speed, break durations, and stored data for 31 days, leading to workers potentially having to justify every brief interruption. The CNIL found the surveillance to be excessive and in breach of GDPR. Amazon disputes the findings and stated it may appeal.

Company involved

Amazon France Logistique

AI system involved

handheld scanner monitoring system

9 source articles · read the reporting →

Serco Leisure issued enforcement notices for unlawful biometric monitoring of employees

Serco Leisure and associated trusts used facial recognition and fingerprint scanning to monitor employee attendance at 38 leisure facilities. The ICO found they unlawfully processed biometric data of over 2,000 employees. Enforcement notices were issued ordering them to stop.

Company involved

Serco Leisure Operating Limited

8 source articles · read the reporting →

Klarna CEO faces backlash for using AI to halve marketing team

Klarna CEO Sebastian Siemiatkowski tweeted that the company saved $10 million by using generative AI to reduce its in-house marketing team to half its previous size and cut external agency spending by 25%. The post sparked criticism on social media, with commenters calling the remarks 'ghoulish' and highlighting concerns about AI's impact on jobs. Klarna previously laid off 700 workers in 2022 and reduced headcount by 23% by end of 2023.

Company involved

Klarna

AI system involved

Midjourney, DALL-E, Adobe Firefly

6 source articles · read the reporting →

French law legalizes algorithmic surveillance for 2024 Olympics

The French Parliament passed a law on March 23, 2023, that legalizes algorithmic video surveillance for the 2024 Olympic and Paralympic Games. Civil society organizations, including ECNL, La Quadrature du Net, and Amnesty International France, argue that the measures violate international human rights law and risk conflicting with the future EU AI Act. The law allows invasive surveillance under the pretext of securing big events, marking France as the first EU member state to explicitly legalize such practices.

Company involved

French government

AI system involved

Algorithmic video surveillance

10 source articles · read the reporting →

Lattice cancels plan to give AI digital workers employee records after backlash

Lattice, an HR software company, announced on July 9th that it would give AI digital workers official employee records. After strong backlash from HR professionals and others on LinkedIn, the company canceled the feature on July 12th, stating it 'will not further pursue digital workers in the product.' The feature was intended to manage AI bots such as Devin and Piper, but the company reversed course.

Company involved

Lattice

AI system involved

Lattice

6 source articles · read the reporting →

French police secretly used Briefcam facial recognition since 2015

The French National Police has been using Briefcam's Video Synopsis software since 2015, according to internal documents obtained by Disclose. The software, which includes facial recognition capabilities, was deployed without the required data protection impact assessment or notification to the CNIL. The Ministry of the Interior concealed the use of this tool for eight years. The police hierarchy allegedly used the software for facial recognition without judicial requisition, and the DGPN did not respond to requests for comment.

Company involved

French National Police

AI system involved

Briefcam Video Synopsis

10 source articles · read the reporting →