Where automated decisions went wrong

Danish law enables algorithm to profile unemployed for long-term risk

The Danish parliament passed a law in April 2019 that allows an algorithm to analyse personal data of all unemployed individuals to identify those at risk of long-term unemployment. The system uses data on ethnicity, age, health, and education history to produce a risk score for social workers. Critics, including legal experts and unemployment insurance funds, argue the profiling is stigmatising, lacks transparency, and may violate GDPR. The Danish Data Protection Authority has reopened its review of the law.

Company involved

Danish Ministry of Employment

10 source articles · read the reporting →

FTC settles with IntelliVision over deceptive facial recognition claims

The Federal Trade Commission (FTC) took action against IntelliVision Technologies Corp. for making false, misleading, or unsubstantiated claims about its AI-powered facial recognition software. The company allegedly claimed its software had one of the highest accuracy rates on the market and was free of gender or racial bias, without supporting evidence. The FTC also alleged that IntelliVision did not train its software on millions of faces as claimed, but on images of approximately 100,000 individuals. Under a proposed consent order, IntelliVision is prohibited from making such misrepresentations without competent and reliable testing.

Company involved

IntelliVision Technologies Corp.

AI system involved

IntelliVision facial recognition software

9 source articles · read the reporting →

ScaleFactor reportedly failed to deliver promised automated bookkeeping software

ScaleFactor, an Austin-based startup, is reported to have failed to deliver the automated, real-time bookkeeping tools it promised customers, instead relying on human bookkeepers and a Filipino contract accounting firm. The company told Forbes in June that it was shutting down, initially blaming the pandemic, but Forbes later reported that its problems predated Covid-19. Investors reportedly came to see the company as more of a services business than a software platform, and pulled funding after a pivot to a marketplace model. No legal or regulatory action is reported.

Company involved

ScaleFactor

10 source articles · read the reporting →

DeepSeek exposed user data via open ClickHouse database

Cloud security firm Wiz discovered a ClickHouse database belonging to DeepSeek that was open to the internet without authentication, containing over a million lines of logs with chat histories, secret keys and backend details. Wiz disclosed the breach to DeepSeek, which promptly locked down the database. The incident highlights security risks in rapidly deploying AI services.

Company involved

DeepSeek

AI system involved

DeepSeek-R1

5 source articles · read the reporting →

Bavarian police test Palantir data mining with real personal data

The Bavarian State Criminal Police Office (LKA) has been testing Palantir's data mining software, called VeRa, with real personal data for months. The Bavarian data protection commissioner only learned of the test through a media inquiry and has announced a review. The Interior Ministry claims the test is lawful under current law, but critics argue a legal basis is missing.

Company involved

Bayerisches Landeskriminalamt

AI system involved

VeRa

7 source articles · read the reporting →

Presto Automation uses off-site human agents to double-check AI drive-thru orders

Presto Automation Inc, which markets an AI voice assistant for drive-thru ordering, used off-site human agents in countries including the Philippines to double-check orders in more than 70% of customer interactions, according to SEC filings reported by Bloomberg. The company told Bloomberg that the process helps train its system and should reduce human intervention over time. Presto's drive-thru AI is used in more than 400 restaurants, including Del Taco, Carl's Jr and Checkers, and its stock fell more than 10% after the reports.

Company involved

Presto Automation Inc.

8 source articles · read the reporting →

Chattr.ai exposed job applicant data due to insecure Firebase rules

A security researcher discovered that Chattr.ai, an AI hiring system used by many fast food chains, had a Firebase database with insecure security rules. By registering a new user, the researcher gained full read/write access to the database, exposing personal data of job applicants and employees, including names, phone numbers, emails, and some plaintext passwords. The vulnerability was reported to Chattr.ai on 9 January and patched the next day, but the company did not provide further contact or thanks.

Company involved

Chattr.ai

AI system involved

Chattr.ai

6 source articles · read the reporting →

N-Tech.lab's FindFace used to identify St Petersburg metro passengers without consent

Egor Tsvetkov photographed passengers on the St Petersburg metro without their permission and used N-Tech.lab's facial recognition service FindFace to match their faces to public Vkontakte profiles. He published the results in an art project called 'Your Face is Big Data', saying he wanted to show how 'digital narcissism' can lead to stalking. Privacy advocates said the project was ethically problematic because the subjects had not consented and their identities were exposed. FindFace had been launched by N-Tech.lab in February 2016.

Company involved

N-Tech.lab

AI system involved

FindFace

8 source articles · read the reporting →

Teleperformance deploys AI to neutralise Indian call centre agents' accents

Teleperformance, the world's largest call centre operator, has announced it is using AI from Sanas to modify the accents of its Indian employees in real time. The technology, called accent translation, aims to make agents sound more neutral to native English speakers. The company invested $13 million in Sanas and gained exclusive rights. No specific incident of harm has been reported.

Company involved

Teleperformance

AI system involved

Sanas AI

6 source articles · read the reporting →

FTC settles with DoNotPay over deceptive AI lawyer claims

The FTC took action against DoNotPay, a company that claimed to offer an AI service that was 'the world's first robot lawyer.' The company promised to generate legal documents and replace human lawyers, but the FTC alleged it failed to test its AI output and did not hire any attorneys. DoNotPay agreed to a settlement requiring it to pay $193,000 and notify consumers about the limitations of its service.

Company involved

DoNotPay

AI system involved

DoNotPay

7 source articles · read the reporting →

Spanish police bust $20M AI-powered investment scam

Spanish law enforcement, collaborating with international authorities, dismantled a $20 million investment scam that used AI-driven algorithms to deceive individuals and organizations. Six suspects were detained and assets, including luxury cars and cryptocurrency, were seized. The article does not report any compensation for victims.

5 source articles · read the reporting →

OnlyFake site uses neural networks to generate fake IDs, bypasses OKX verification

An underground website called OnlyFake uses neural networks to generate realistic photos of fake IDs for $15. The journalist tested the service and obtained a convincing California driver's license. They then used another fake ID to successfully bypass the identity verification process on OKX, a cryptocurrency exchange. The article alleges that this technology could streamline bank fraud and money laundering, but reports no actual financial loss.

Company involved

OKX

AI system involved

OnlyFake

10 source articles · read the reporting →

Swedish welfare agency's AI system flags marginalized groups for fraud investigations

Försäkringskassan, Sweden's Social Insurance Agency, uses an AI risk-scoring system to flag welfare applicants for fraud investigations. The system disproportionately targets women, individuals with foreign backgrounds, low-income earners, and those without university degrees, according to an investigation by Lighthouse Reports and Svenska Dagbladet. Amnesty International has called for the system to be discontinued, citing violations of the right to equality and non-discrimination.

Company involved

Försäkringskassan (Swedish Social Insurance Agency)

6 source articles · read the reporting →

Queensland police trial AI to predict domestic violence risk

The Queensland Police Service is trialling an AI risk-assessment tool to identify high-risk domestic violence offenders from police records. Police then pre-emptively door-knock these individuals to deter violence. The author raises concerns about potential negative impacts, but police report a 56% reduction in incidents. The AI was developed in-house to increase transparency.

Company involved

Queensland Police Service

9 source articles · read the reporting →

CJEU rules Dun & Bradstreet must explain automated credit decisions under GDPR

A customer was refused a mobile phone contract because of an automated credit assessment by Dun & Bradstreet Austria. The customer took the case to court, which found that Dun & Bradstreet had infringed the GDPR by failing to provide meaningful information about the logic involved. The CJEU ruled that data controllers must explain automated decisions and that trade secrets cannot automatically override the right of access.

Company involved

Dun & Bradstreet Austria GmbH

7 source articles · read the reporting →

PimEyes faces fine proceedings over biometric facial recognition in Baden-Württemberg

PimEyes, a facial recognition search engine, is accused of scraping images from the internet and processing biometric data without a valid legal basis under the GDPR. The data protection authority of Baden-Württemberg (LfDI) opened fine proceedings after it found PimEyes's response to its questions inadequate. PimEyes argues that the images it processes are publicly available and not personal data. The LfDI says the processing endangers citizens' rights and freedoms and is not covered by the GDPR exceptions.

Company involved

PimEyes

AI system involved

PimEyes

9 source articles · read the reporting →

Audit of RisCanvi finds biases and reliability issues in criminal justice system

Eticas conducted an adversarial audit of RisCanvi, an AI risk assessment tool used in Catalonia's criminal justice system. The audit uncovered biases in risk classifications against specific demographics and significant reliability issues. The findings call for fairer practices in criminal justice AI.

Company involved

Catalonia's criminal justice system

AI system involved

RisCanvi

4 source articles · read the reporting →

Microsoft Dynamics 365 Field Service AI singles out workers in performance predictions

A report by Cracked Labs found that Microsoft's Dynamics 365 Field Service software uses AI to generate performance metrics and predict task durations, singling out individual workers. The AI predictions can be influenced by the worker's identity, such as increasing or decreasing estimated duration. Microsoft stated the system is not intended for employment decisions and is not a surveillance tool, but the report raises concerns about potential misuse for worker monitoring.

Company involved

Microsoft

AI system involved

Dynamics 365 Field Service

6 source articles · read the reporting →

Singapore Firm Defrauded of $499K in Deepfake CEO Video Call Scam

In March 2025, a finance director at a multinational firm in Singapore authorised a US$499,000 payment during a Zoom call that appeared to include the company's CFO and other executives. The call was a deepfake, with AI-generated likenesses and voices. The fraudsters used the impersonation to request an urgent fund transfer for a purported acquisition. The company later discovered the deception and reported it to Singapore police.

2 source articles · read the reporting →

Scammers Use AI to Create Fake Joann Fabrics Websites to Steal Credit Card Data

After Joann Fabrics filed for bankruptcy in January 2025, scammers used AI to create impostor websites mimicking the retailer's site. These fake sites offered deep discounts to trick shoppers into providing credit-card information and personal data. Customers who placed orders never received products and had their payment information compromised. The incident highlights the growing use of AI by cybercriminals to create convincing fake websites.

2 source articles · read the reporting →

Palantir secretly tested predictive policing in New Orleans

Beginning in 2012, Palantir Technologies secretly partnered with the New Orleans Police Department to deploy a predictive policing system. The programme analysed gang affiliations, social media and criminal histories to forecast individuals’ likelihood of committing or becoming victims of violence, operating without public knowledge or city council oversight. Researchers and law enforcement officials raised concerns about systemic bias and civil liberties. As of 2018, the city and Palantir had not disclosed the programme’s status.

Company involved

New Orleans Police Department

1 source article · read the reporting →

Adobe Firefly trained on thousands of Midjourney images, Bloomberg reports

Bloomberg has reported that Adobe's Firefly image generator was trained using thousands of images from competitor Midjourney. Adobe says these made up about 5% of the training data and were part of the Adobe Stock library. The company has marketed Firefly as ethically trained and offered enterprise customers indemnity against copyright claims. Adobe responded that all Adobe Stock images undergo moderation, but the report has raised questions about Firefly's copyright safety.

Company involved

Adobe

AI system involved

Firefly

7 source articles · read the reporting →

PredictiveHire builds AI to predict job hopping from interviews

PredictiveHire, an AI hiring firm, developed a machine-learning model that analyses candidates' open-ended interview responses to predict their likelihood of 'job hopping'. The company used data from 45,899 applicants to build the 'flight risk' assessment, which it advertises as coming soon. Scholars warn that such tools can suppress wages by screening out workers who might seek better pay or conditions, continuing a historical trend of using personality tests to identify potential labour organisers.

Company involved

PredictiveHire

AI system involved

Phai

1 source article · read the reporting →

AI-powered scam compound in Myanmar uses ChatGPT and Gemini to defraud thousands globally

Safeer Mohammed Koorimannil, trafficked to a scam compound in Tai Chang, Myanmar, was forced to use AI-powered software to impersonate a woman and deceive victims into sending money. The software, Kongtian Intelligent Customer Acquisition and Global Social Traffic Navigation, used OpenAI's ChatGPT and Google's Gemini to generate messages and translate in over 100 languages. Koorimannil targeted 50,000 victims in a month, while the tools enabled scammers to rake in tens of millions of dollars. U.S. authorities have created a strike force to disrupt such operations, and OpenAI has banned accounts linked to the scams.

Company involved

Tai Chang

AI system involved

Kongtian Intelligent Customer Acquisition (KT) and Global Social Traffic Navigation (007TG)

2 source articles · read the reporting →