Where automated decisions went wrong

Workday Gains an Edge in AI Hiring Bias Fight: California Brief - Bloomberg Law News

Workday Gains an Edge in AI Hiring Bias Fight: California Brief - Bloomberg Law News

1 source article · read the reporting →

AI Hiring Platform Faces FCRA Class Action Over Data Use | Kistler et al. v. Eightfold AI Inc.

The AI platform screened job applicants, affecting their hiring prospects.

Company involved

Eightfold AI

AI system involved

Eightfold AI

1 source article · read the reporting →

Data Breach Exposes Over 10 Million Conversations from Middle Eastern AI Call Center Platform

Resecurity discovered a dark web posting on 8 October 2024 offering data stolen from a major AI-powered cloud call center platform in the Middle East. The threat actor gained unauthorized access to the management dashboard, compromising over 10,210,800 conversations between consumers, operators, and AI chatbots. The exposed data included personally identifiable information and national ID documents, creating risks of fraud and identity theft. Resecurity alerted the affected organization and collaborated with law enforcement to mitigate the incident.

3 source articles · read the reporting →

AI Lawsuit Pushes the Boundaries of AI Litigation—and May Signal a New Wave

Ranked job applicants' likelihood of success, affecting their hiring prospects.

Company involved

Eightfold AI Inc.

AI system involved

Eightfold AI Hiring Platform

1 source article · read the reporting →

Arup employee tricked by deepfake video call transfers $25 million

An employee at engineering firm Arup in Hong Kong was deceived by a deepfake video call impersonating an executive, leading to a transfer of $25 million. The incident highlights the growing threat of AI-generated voice and video deepfakes in financial transactions. Reality Defender, a deepfake detection firm, cites this case as an example of the need for advanced authentication tools.

2 source articles · read the reporting →

Retool Breached After Hacker Uses AI Deepfake Voice in Phishing Call

A hacker used AI to deepfake an employee's voice and trick a Retool staff member into providing a multi-factor authentication code. The attacker sent phishing SMS messages and then called the employee, impersonating an IT team member with a synthetic voice. This allowed the hacker to add their own device to the employee's account and access internal systems, compromising 27 cloud customers. Retool revoked the access and disclosed the incident, blaming a weakness in Google Authenticator's cloud sync feature.

Company involved

Retool

1 source article · read the reporting →

High Court rules against Sky Betting’s targeted marketing to problem gambler

A reformed problem gambler sued Sky Betting & Gaming, alleging that the company unlawfully collected his data and used it for targeted marketing between 2017 and 2019 without valid consent. The High Court ruled in his favour, finding that his gambling addiction impaired his ability to give genuine consent and that the company’s automated profiling and direct marketing violated data protection laws. The court emphasised that organisations must consider vulnerabilities when seeking consent. Sky Betting is considering an appeal.

Company involved

Sky Betting & Gaming

2 source articles · read the reporting →

AI-Generated Fake Investment Websites Target Australian Fraud Analyst

A fraud analyst, Leon, was targeted by scammers who used AI to create convincing fake websites for non-existent investment firms. The scammers posed as employees of Assent Advisory and directed him to elaborate sites for APPC Capital Singapore and Thackeray Mines and Minerals Inc., complete with AI-generated images and fake details. Leon recognised the scam, but the sophistication of the AI-generated content made him second-guess his own judgment. The incident highlights how AI is enabling more convincing investment scams.

1 source article · read the reporting →

Rotterdam's Welfare Fraud Algorithm Discriminated by Gender and Ethnicity

The city of Rotterdam deployed a machine learning algorithm built by Accenture to flag welfare recipients for fraud investigation. The system used personal data including gender, language, and subjective caseworker notes to generate risk scores, leading to investigations that disproportionately targeted women and migrants. An external review found the algorithm discriminatory and inaccurate, prompting the city to suspend its use in 2021. The system's opacity made it nearly impossible for those flagged to challenge the decisions.

Company involved

City of Rotterdam

6 source articles · read the reporting →

FTC Orders Evolv to Stop Overstating Effectiveness of AI Gun Detection Tech

Evolv, a company selling AI-powered weapons detection systems, has been ordered by the FTC to stop making misleading claims about its Evolv Express product. The FTC alleged that the technology is essentially a metal detector with unproven AI, and that Evolv falsely claimed it could detect guns while ignoring harmless items. A pilot in New York City subways resulted in over 100 false positives and no guns detected. The settlement requires Evolv to allow educational customers to cancel their contracts and prohibits further misrepresentations.

Company involved

Evolv

AI system involved

Evolv Express

2 source articles · read the reporting →

SEC Charges Delphia and Global Predictions for False AI Claims

The SEC charged Delphia (USA) Inc. and Global Predictions Inc. for making false and misleading statements about their use of artificial intelligence. Delphia claimed from 2019 to 2023 that it used AI and machine learning to predict investments, while Global Predictions falsely claimed in 2023 to be the 'first regulated AI financial advisor'. Both firms settled the charges without admitting or denying the findings, agreeing to pay a total of $400,000 in civil penalties and to cease and desist from further violations.

Company involved

Delphia (USA) Inc. and Global Predictions Inc.

1 source article · read the reporting →

CFPB Acts Against Hello Digit for Faulty Savings Algorithm Causing Overdrafts

Hello Digit, a fintech company, used an automated savings algorithm that made transfers from consumers' checking accounts, falsely guaranteeing no overdrafts. The algorithm caused customers to incur overdraft fees, and the company often denied reimbursement requests. The CFPB found that Hello Digit engaged in deceptive practices and ordered the company to pay redress to harmed consumers and a $2.7 million fine.

Company involved

Hello Digit, LLC

AI system involved

Hello Digit app

1 source article · read the reporting →

Schufa's Black-Box Scoring Unfairly Penalises Consumers with Positive Credit Data

An investigation by SPIEGEL and BR Data reveals that Schufa's credit scoring algorithm often assigns poor risk scores to consumers with only positive credit information. One consumer, Sven Drewert, was denied a credit card limit increase despite having no negative entries. The algorithm uses limited data, and its secret formula can lead to arbitrary categorisations, affecting access to loans, phone contracts, and housing. The system's opacity and potential biases raise concerns about fairness and accountability.

Company involved

Schufa Holding AG

AI system involved

Schufa Score

2 source articles · read the reporting →

French regulator fines Clearview AI €20 million for privacy breaches

France's privacy watchdog CNIL fined US facial recognition firm Clearview AI €20 million for unlawfully collecting and processing facial images of individuals without consent. The company scraped billions of images from websites and social media, selling access to law enforcement. Clearview AI denied being subject to EU law and refused to delete the data, claiming it was impossible to determine French residency from public photos. The CNIL ordered the firm to stop collecting data and delete existing data within two months or face daily fines.

Company involved

Clearview AI

10 source articles · read the reporting →

Xsolla fires 150 employees after big data analysis tags them as unproductive

Payment services company Xsolla terminated 150 employees at its Perm, Russia office after a big data analysis of their activity in Jira, Confluence, Gmail, chats, and documents tagged them as unengaged and unproductive. CEO Aleksandr Agapitov sent an email to affected employees stating that Xsolla was not for them, sparking backlash. The company later held a press conference, explaining the layoffs were due to slowing growth, and offered affected employees medical insurance and severance pay equal to four to six months' salary.

Company involved

Xsolla

4 source articles · read the reporting →

Amsterdam court orders Uber and Ola to disclose robo-firing algorithms

The Amsterdam Court of Appeal ruled that Uber and Ola Cabs violated drivers' GDPR rights by using automated systems to dismiss workers without meaningful human intervention or transparency. The court found that Uber's fraud detection system profiled drivers and made erroneous fraud allegations, leading to account deactivations that the court deemed 'robo-firing'. Uber and Ola were ordered to provide drivers with information on how automated decision-making affects work allocation, pay, and dismissals, rejecting the companies' trade secret arguments. The ruling is a significant win for gig economy workers, though the UK government is advancing a bill that would strip similar protections.

Company involved

Uber

5 source articles · read the reporting →

Durham Police uses Experian Mosaic data in HART AI risk tool

Durham Constabulary developed the Harm Assessment Risk Tool (HART), a machine learning algorithm that assesses the recidivism risk of offenders. The tool uses 34 data categories including criminal history, age, gender and two types of postcode, one sourced from Experian's Mosaic marketing segmentation system. Big Brother Watch alleges that using such commercial consumer behaviour data to inform custody decisions risks prejudice and disproportionate targeting of deprived neighbourhoods. The force has stated it is refreshing the model with an aim to remove one of the postcode predictors.

Company involved

Durham Constabulary

AI system involved

Harm Assessment Risk Tool (HART)

9 source articles · read the reporting →

Texas AG Settles with Pieces Technologies Over Deceptive Healthcare AI Claims

The Texas Attorney General investigated Pieces Technologies, a Dallas-based healthcare AI company, for making false and misleading statements about the accuracy of its generative AI product used in hospitals. The company claimed an error rate of less than 1 per 100,000, but the investigation found these metrics were likely inaccurate. As part of the settlement, Pieces agreed to accurately disclose its product's accuracy and ensure hospital staff understand the appropriate reliance on its AI. The case marks the first-of-its-kind healthcare generative AI investigation by the AG.

Company involved

Pieces Technologies

4 source articles · read the reporting →

Fraudsters use AI voice deepfake to trick UK energy firm CEO into transferring $243,000

In March 2019, criminals used commercially available AI voice-generation software to impersonate the CEO of a German parent company. They tricked the CEO of a UK-based energy firm into urgently wiring $243,000 to a Hungarian supplier. The fraud was discovered when the fraudsters attempted a second transfer, which the CEO refused. The company was insured and the loss was covered.

Company involved

Unnamed UK-based energy firm

AI system involved

Commercially available voice-generating AI software

10 source articles · read the reporting →

Austrian court lifts ban on AMS job-chance prediction algorithm

The Austrian Federal Administrative Court overturned a ban by the Data Protection Authority on the AMS algorithm, which predicts job chances of unemployed people. The system, trained on historical data including age, gender, and nationality, categorises individuals as high, medium or low chance of re-employment. Critics argue it discriminates against women and mothers, and that the data is now outdated due to the COVID-19 pandemic. The court ruled that the algorithm is lawful as long as a human advisor makes the final decision on training support.

Company involved

Arbeitsmarktservice (AMS)

AI system involved

AMS-Algorithmus

5 source articles · read the reporting →

Dutch government continued SyRI welfare fraud profiling after ban

The Dutch government deployed the SyRI algorithm to profile welfare recipients in low-income neighborhoods, despite a 2020 court ruling that the system violated EU human rights. The algorithm flagged thousands of parents for fraud based on illegal risk factors such as dual nationality, leading to false accusations, bankruptcy, and trauma. The government has announced an external review of the practices.

Company involved

Ministry of Social Affairs of the Netherlands

AI system involved

SyRI

10 source articles · read the reporting →

Finnish recruitment company Digital Minds used AI to analyze job applicants' messages, prompting data protection investigation

Digital Minds, a Finnish recruitment company founded by psychologists, used IBM Watson AI to analyze job applicants' social media and email messages for personality assessments. The company obtained written consent but the Finnish Data Protection Ombudsman launched an investigation, suspecting violations of data protection laws and the secrecy of correspondence. The service was used on fewer than ten applicants and has been paused pending the investigation.

Company involved

Digital Minds

AI system involved

IBM Watson

9 source articles · read the reporting →

Europol cracks down on network using deepfake ads for crypto fraud

Europol and national authorities in Belgium, Bulgaria, Germany, and Israel arrested two people in November as part of a crackdown on a criminal network that used fraudulent online ads and AI-generated deepfake videos to lure victims into fake cryptocurrency investment platforms. The network allegedly laundered 700 million euros ($816 million) through the scheme, which Europol described as operating on an 'industrial' scale. The arrests were the second phase of an operation that had already led to nine arrests in October. The article does not name the suspects or firms targeted.

4 source articles · read the reporting →

DeepScore markets facial and voice analysis app for trustworthiness scoring despite experts' doubts

DeepScore, a Tokyo-based company, is marketing an app that uses facial and voice recognition to score people's trustworthiness for lenders and insurers in Japan, Indonesia, Vietnam and the Philippines. The company says the app can detect deception with 70 per cent accuracy, but researchers and privacy advocates say there is no reliable scientific basis for such judgments and warn of discrimination and privacy harms. The chief executive said the system is only one part of lenders' and insurers' decision-making and that people can choose not to use it. Critics respond that an unequal balance of power makes consent difficult.

Company involved

DeepScore

AI system involved

DeepScore

6 source articles · read the reporting →