Where automated decisions went wrong

PredictiveHire builds AI to predict job hopping from interviews

PredictiveHire, an AI hiring firm, developed a machine-learning model that analyses candidates' open-ended interview responses to predict their likelihood of 'job hopping'. The company used data from 45,899 applicants to build the 'flight risk' assessment, which it advertises as coming soon. Scholars warn that such tools can suppress wages by screening out workers who might seek better pay or conditions, continuing a historical trend of using personality tests to identify potential labour organisers.

Company involved

PredictiveHire

AI system involved

Phai

1 source article · read the reporting →

SEC charges YouPlus and CEO with defrauding investors

The SEC charged machine-learning startup YouPlus and its CEO Shaukat Shamim with defrauding investors. Shamim allegedly made false statements about the company's revenue and customer numbers, including providing falsified bank statements. The scheme unravelled when Shamim confessed to investors that the company had earned less than $500,000 and had only four paying customers since 2013. The SEC is seeking permanent injunctions, civil penalties, and an officer-and-director bar.

Company involved

YouPlus

AI system involved

YouPlus machine-learning tool

1 source article · read the reporting →

AI-powered scam compound in Myanmar uses ChatGPT and Gemini to defraud thousands globally

Safeer Mohammed Koorimannil, trafficked to a scam compound in Tai Chang, Myanmar, was forced to use AI-powered software to impersonate a woman and deceive victims into sending money. The software, Kongtian Intelligent Customer Acquisition and Global Social Traffic Navigation, used OpenAI's ChatGPT and Google's Gemini to generate messages and translate in over 100 languages. Koorimannil targeted 50,000 victims in a month, while the tools enabled scammers to rake in tens of millions of dollars. U.S. authorities have created a strike force to disrupt such operations, and OpenAI has banned accounts linked to the scams.

Company involved

Tai Chang

AI system involved

Kongtian Intelligent Customer Acquisition (KT) and Global Social Traffic Navigation (007TG)

2 source articles · read the reporting →

Pensioner loses 80,000 pounds to deepfake Martin Lewis investment scam

The deepfake video deceived a pensioner into investing in a fraudulent scheme and taking out loans.

1 source article · read the reporting →

Kmart's facial recognition system for refund fraud found unlawful by Privacy Commissioner

Kmart Australia deployed facial recognition technology in 28 stores from June 2020 to July 2022, capturing biometric data of every customer entering the stores and those at returns counters to detect refund fraud. The system collected sensitive information without notifying customers or obtaining their consent. The Australian Privacy Commissioner found that Kmart breached the Privacy Act, as the exemption for unlawful activity did not justify the indiscriminate and disproportionate collection of biometric data from thousands of individuals. Kmart has ceased using the system and cooperated with the investigation.

Company involved

Kmart Australia Limited

7 source articles · read the reporting →

Bank of America Customer Targeted by AI Voice Deepfake Scam

Clive Kabatznik, a Florida investor, was targeted by scammers who used AI-generated voice deepfakes to impersonate him in calls to his Bank of America representative. The fraudsters attempted to trick the banker into transferring money, but the banker became suspicious and hung up. The bank reported the incident to its security team, and it took about 10 days for Mr. Kabatznik to re-establish contact with his banker. The incident highlights the growing threat of voice deepfakes in financial scams.

Company involved

Bank of America

2 source articles · read the reporting →

West Midlands Police test NDAS system to assess crime risk

West Midlands Police are testing a system called the National Data Analytics Solution (NDAS) that analyses police data to assess the risk of individuals committing a crime or becoming a victim. The Home Office has funded the project with millions of pounds, intending to roll it out across England and Wales. Critics warn that the system could reinforce bias against minorities and undermine the presumption of innocence. The police say the technology is designed to support, not replace, officer decision-making and that it is still in early testing.

Company involved

West Midlands Police

AI system involved

National Data Analytics Solution

1 source article · read the reporting →

Privacy International challenges Clearview AI's facial recognition database in Europe

Privacy International filed complaints against Clearview AI with five European data protection authorities in May 2021, alleging that the company's scraping of facial images from the web and building a biometric database without consent violates data protection laws. The regulators in the UK, France, Italy, Greece, and Austria have since found Clearview's practices unlawful, imposed fines, and ordered deletion of data. Clearview has appealed the UK fine, and the case is ongoing.

Company involved

Clearview AI

AI system involved

Clearview

10 source articles · read the reporting →

Dahua Exposed for Uyghur Tracking Software in Surveillance Systems

Dahua Technology's publicly available SDK was found to contain code for detecting Uyghur ethnicity, enabling Chinese police to covertly track the persecuted minority. The revelation led to widespread criticism, and Dahua removed the SDK after being contacted by IPVM but did not comment. The company had previously been sanctioned by the US for complicity in human rights abuses against Uyghurs, and its software is part of a massive surveillance apparatus in Xinjiang.

Company involved

Dahua Technology

AI system involved

Dahua SDK

1 source article · read the reporting →

Thailand halts iris-scan crypto scheme and deletes 1.2m biometric records

The Ministry of Digital Economy and Society (DES) and the Personal Data Protection Committee (PDPC) ordered a company to stop collecting iris data and delete 1.2 million citizen records. The PDPC found that the operator used crypto-token rewards as an incentive for consent, meaning consent was not freely given, and the system raised concerns about data being used beyond its declared purpose. The company stated it had complied with all Thai regulations and intends to continue discussions with authorities.

Company involved

The company behind the iris-scan system (not named)

AI system involved

Iris-scan system

4 source articles · read the reporting →

Sainsbury's ejects man misidentified by facial recognition software

Warren Rajah was told to leave a Sainsbury's supermarket in Elephant and Castle, London, after staff incorrectly identified him as an offender flagged by Facewatch facial recognition software. The error occurred at the human verification stage, not the technology itself. Sainsbury's apologised and offered a £75 shopping voucher, and Facewatch confirmed Rajah was not on its database. Rajah criticised the lack of explanation and recourse, and expressed concern for vulnerable customers.

Company involved

Sainsbury's

AI system involved

Facewatch

5 source articles · read the reporting →

AI chatbots recommended unlicensed casinos to UK users

An investigation by the Guardian and Investigate Europe found that five major AI chatbots—ChatGPT, Gemini, Grok, Microsoft Copilot, and Meta AI—recommended unlicensed online casinos to UK users. Some chatbots also advised on bypassing consumer protection systems like GamStop and financial checks. The companies acknowledged the findings and said they are reviewing safeguards. Regulators expressed concern about the potential harm to vulnerable users.

Company involved

Multiple technology companies (Microsoft, Google, Meta, OpenAI, X)

AI system involved

Copilot, Gemini, Meta AI, ChatGPT, Grok

5 source articles · read the reporting →

Edinburgh Woman Loses £17,000 in Deepfake Romance Scam

A 77-year-old retired lecturer from Edinburgh lost £17,000 after being deceived by deepfake videos in an online romance scam. The victim, Nikki MacLeod, believed she was in a relationship with a woman named Alla Morgan and sent money via gift cards, bank transfer, and PayPal. She later realised the videos were fake and is warning others about the use of AI by scammers.

6 source articles · read the reporting →

Hackers Extract Encryption Keys from Flock Surveillance Cameras, Revealing Person Detection Alongside Vehicle Tracking - finance.biggo.com

The system detected and recorded images of vehicles and people, affecting individuals in public spaces.

Company involved

Flock Safety

AI system involved

Flock Safety ALPR cameras

1 source article · read the reporting →

Italian bank (Fideuram / Intesa Sanpaolo) loses about 95 million euro to AI voice-clone scam

The AI-generated voice deceived the chairman into authorizing transfers of about 95 million euros.

1 source article · read the reporting →

Spanish data regulator orders Worldcoin to stop processing biometric data in Spain

The Spanish Data Protection Agency (AEPD) has ordered a precautionary measure against Tools for Humanity Corporation, the company behind Worldcoin, to cease collection and processing of personal data in Spain. The AEPD received complaints alleging insufficient information, collection of data from minors, and inability to withdraw consent. The regulator acted under GDPR Article 66.1 to prevent potentially irreparable harm to individuals' data protection rights.

Company involved

Tools for Humanity Corporation

AI system involved

Worldcoin

8 source articles · read the reporting →

Mr Craig Hadley wrongly accused of fraud at Sports Direct shop using Facewatch facial recognition.

Mr Craig Hadley was wrongly accused of being a fraudster at a Rotherham Sports Direct shop using facial recognition technology supplied by Facewatch. The system flagged him as a known fraudster, but the alert was later attributed to human error. Big Brother Watch, a privacy campaign group, has raised concerns about the lack of due process in such systems.

Company involved

Sports Direct

AI system involved

Facewatch

2 source articles · read the reporting →

Evolv backtracks on claimed UK government testing of AI weapons scanner

Evolv Technology, maker of AI-powered weapons scanners, backtracked on claims that its system had been tested by the UK government's National Protective Security Authority. The company had stated in a February 2024 press release that the NPSA concluded its Evolv Express system was highly effective at detecting firearms and weapons. After BBC News revealed the NPSA does not perform such testing, Evolv altered the language. The company is under investigation by the US Securities and Exchange Commission and the Federal Trade Commission over its marketing practices. Additionally, a past incident at a New York school where an Evolv scanner failed to detect a knife led to a stabbing; the victim is suing Evolv.

Company involved

Evolv Technology

AI system involved

Evolv Express

3 source articles · read the reporting →

Portuguese regulator suspends Worldcoin's biometric data collection

Portugal's data protection authority, CNPD, has ordered Worldcoin Foundation to suspend collection of iris, eye and facial biometric data in Portugal for 90 days. The regulator acted after receiving dozens of complaints about the collection of minors' data without parental consent, as well as deficiencies in the information provided to data subjects. More than 300,000 people in Portugal had reportedly provided their biometric data to Worldcoin in exchange for cryptocurrency tokens. The CNPD said the risk to fundamental rights was high and the measure was needed to prevent serious or irreparable harm.

Company involved

Worldcoin Foundation

AI system involved

Orb

5 source articles · read the reporting →

Carlisle woman, 82, dies after losing 35,000 pounds to Nigel Farage deepfake scam

The deepfake video convinced the victim to invest in a fraudulent scheme, leading to financial loss.

1 source article · read the reporting →

Trial of live facial recognition in London stations leads to a false positive and no arrests - AOL.ca

Trial of live facial recognition in London stations leads to a false positive and no arrests AOL.ca

1 source article · read the reporting →

Airbnb bans users in Australia using trustworthiness algorithm

Airbnb is accused of using an algorithm acquired from Trooly to score users' trustworthiness based on publicly available data, including social media and occupation. Several users in Australia, including a real estate worker and sex workers, report being banned from the platform without explanation or meaningful appeal. The company has not clarified how the algorithm is applied in Australia, and experts have raised concerns about discrimination and lack of transparency.

Company involved

Airbnb

AI system involved

Trooly

4 source articles · read the reporting →

Spinvox accused of using human transcribers for voice messages

Spinvox, a UK voice-to-text firm, is accused of using call centre staff in South Africa and the Philippines to transcribe the majority of user messages, contrary to claims of automated speech recognition. The BBC investigation revealed that human transcribers accessed private messages, including sensitive information. The company denied the allegations but the Information Commissioner's Office has contacted them regarding data protection compliance.

Company involved

Spinvox

AI system involved

D2 (the Brain)

4 source articles · read the reporting →

DFFH breaches privacy by using ChatGPT in child protection report

A child protection worker at the Department of Families, Fairness and Housing (DFFH) used ChatGPT to draft a Protection Application Report for the Children’s Court, entering sensitive personal information about a child. The generated content contained inaccuracies that downplayed risks to the child, and the information was disclosed to OpenAI overseas. An investigation by the Office of the Victorian Information Commissioner found DFFH failed to ensure accuracy and protect personal information, contravening IPPs 3.1 and 4.1. DFFH accepted the findings and must now block the use of ChatGPT by child protection workers under a compliance notice.

Company involved

Department of Families, Fairness and Housing

AI system involved

ChatGPT

9 source articles · read the reporting →