Where automated decisions went wrong

BC Tribunal Confirms Companies Remain Liable for AI Chatbot-Created Information - Lexology

The AI chatbot provided inaccurate information to a user.

1 source article · read the reporting →

Serco Leisure issued enforcement notices for unlawful biometric monitoring of employees

Serco Leisure and associated trusts used facial recognition and fingerprint scanning to monitor employee attendance at 38 leisure facilities. The ICO found they unlawfully processed biometric data of over 2,000 employees. Enforcement notices were issued ordering them to stop.

Company involved

Serco Leisure Operating Limited

8 source articles · read the reporting →

EvenUp AI errors in personal injury demand letters lead to scrutiny

EvenUp, a legal tech startup valued at $1 billion, uses AI to draft personal injury demand letters. Former employees revealed that the AI system frequently makes errors, including missing injuries and fabricating medical conditions. The company defends its hybrid approach with human oversight, but critics allege overpromised AI capabilities.

Company involved

EvenUp

6 source articles · read the reporting →

CJEU rules Dun & Bradstreet must explain automated credit decisions under GDPR

A customer was refused a mobile phone contract because of an automated credit assessment by Dun & Bradstreet Austria. The customer took the case to court, which found that Dun & Bradstreet had infringed the GDPR by failing to provide meaningful information about the logic involved. The CJEU ruled that data controllers must explain automated decisions and that trade secrets cannot automatically override the right of access.

Company involved

Dun & Bradstreet Austria GmbH

7 source articles · read the reporting →

ICO investigates Microsoft's Recall feature for privacy risks

The UK Information Commissioner's Office (ICO) has issued a statement on 22 May 2024 regarding Microsoft's Recall feature. The ICO is making enquiries with Microsoft to understand the safeguards in place to protect user privacy, expecting transparency and necessity in data processing. This follows concerns that the feature may not adequately consider data protection from the outset before being brought to market.

Company involved

Microsoft

AI system involved

Recall

10 source articles · read the reporting →

Condé Nast accuses Perplexity of plagiarism in cease-and-desist letter

Condé Nast, the media conglomerate, sent a cease-and-desist letter to AI search startup Perplexity, accusing it of plagiarism for using content from its publications in AI-generated responses without permission. The letter demands that Perplexity stop using the content. Perplexity has been criticized for ignoring robots.txt and scraping content. The incident highlights ongoing tensions between publishers and AI companies over unauthorized use of content.

Company involved

Perplexity

AI system involved

Perplexity

4 source articles · read the reporting →

Taylor & Francis sells academic authors' research to Microsoft AI without consent

Academic authors have expressed shock after Taylor & Francis sold access to their research to Microsoft AI without informing them or offering additional payment. The authors allege they were not given the opportunity to opt out of the deal. The arrangement involves Microsoft using the research to train its AI systems. The authors have not received any compensation for the use of their work.

Company involved

Taylor & Francis

AI system involved

Microsoft AI

3 source articles · read the reporting →

Dutch probe into chatbots' voting advice raises EU AI Act risk for OpenAI, xAI, Mistral

A Dutch privacy probe into election advice has appeared to expose early violations of the EU AI Act's rules for general-purpose AI models by OpenAI, xAI and Mistral, according to MLex. The companies' chatbots provided distorted voting advice to users. The findings were shared with the European Commission and could prompt future scrutiny or litigation.

Company involved

OpenAI, xAI and Mistral

6 source articles · read the reporting →

Audit of RisCanvi finds biases and reliability issues in criminal justice system

Eticas conducted an adversarial audit of RisCanvi, an AI risk assessment tool used in Catalonia's criminal justice system. The audit uncovered biases in risk classifications against specific demographics and significant reliability issues. The findings call for fairer practices in criminal justice AI.

Company involved

Catalonia's criminal justice system

AI system involved

RisCanvi

4 source articles · read the reporting →

Thomson Reuters wins copyright lawsuit against AI startup Ross Intelligence

In 2020, Thomson Reuters filed a copyright lawsuit against legal AI startup Ross Intelligence, alleging that Ross reproduced materials from its Westlaw legal research service. In February 2025, a US District Court judge ruled in Thomson Reuters' favor, finding that Ross infringed copyright and that fair use did not apply. Ross Intelligence had shut down in 2021 due to litigation costs.

Company involved

Ross Intelligence

AI system involved

Ross Intelligence

4 source articles · read the reporting →

Dutch DPA fines Clearview AI €30.5 million for GDPR violations

The Dutch Data Protection Authority fined Clearview AI €30.5 million for creating an illegal database of biometric codes scraped from social media without consent. The regulator accused the company of failing to inform individuals about how their data is used and continuing violations after investigation. Clearview AI denies the allegations, stating it has no presence in the Netherlands or EU and that the fine is unenforceable.

Company involved

Clearview AI

AI system involved

Clearview AI facial recognition database

7 source articles · read the reporting →

Dutch probation service's OXREC algorithm flawed, leading to incorrect recidivism risk assessments

The Dutch Inspectorate of Justice and Security (Inspectie JenV) published a report finding that the probation service's (Reclassering) OXREC algorithm contains serious flaws, including swapped formulas and incorrect numbers, causing about a quarter of risk assessments to be wrong. The algorithm, used since 2018 for about 44,000 cases per year, also uses variables that can lead to discrimination, such as neighborhood score and income. The Inspectorate recommended immediate correction or temporary suspension. The probation service announced it would temporarily stop using OXREC.

Company involved

Reclassering Nederland

AI system involved

OXREC

4 source articles · read the reporting →

UK prison risk algorithm may embed racial bias, experts warn

The Ministry of Justice developed a digital tool to categorize prisoners, which experts and advocacy groups warn could automate and embed racism. The tool uses intelligence data that may reflect existing bias, potentially leading to disproportionate categorization of BAME prisoners. The MoJ claims no evidence of discrimination, but critics question the small trial sample and lack of transparency.

Company involved

Ministry of Justice

AI system involved

Digital prisoner categorisation tool

5 source articles · read the reporting →

Citizens Advice finds ethnicity penalty in car insurance pricing

Citizens Advice conducted exploratory research into car insurance pricing and found that people of colour may be paying £250 more per year than White people. The research suggests that areas with large communities of colour may be identified as more risky by algorithms, even when objective risk factors are controlled. Citizens Advice has called on the Financial Conduct Authority to investigate the issue.

9 source articles · read the reporting →

TransUnion AI tenant screening denied 75-year-old man apartment due to mistaken littering conviction

Chris Robinson, then 75, applied for a senior living apartment in California. The property manager used an AI screening program from TransUnion, which assigned him a low score based on a mistaken conviction for littering that belonged to a different person with the same name in Texas. Robinson lost the apartment and his application fee. A federal class-action lawsuit against TransUnion moved toward a $11.5 million settlement in 2023.

10 source articles · read the reporting →

Palantir secretly tested predictive policing in New Orleans

Beginning in 2012, Palantir Technologies secretly partnered with the New Orleans Police Department to deploy a predictive policing system. The programme analysed gang affiliations, social media and criminal histories to forecast individuals’ likelihood of committing or becoming victims of violence, operating without public knowledge or city council oversight. Researchers and law enforcement officials raised concerns about systemic bias and civil liberties. As of 2018, the city and Palantir had not disclosed the programme’s status.

Company involved

New Orleans Police Department

1 source article · read the reporting →

Duke University MTMC Dataset Used in Authoritarian Surveillance Research

Duke University created and openly distributed the Duke MTMC dataset, containing surveillance footage of approximately 2,000 students and visitors on campus. The dataset was used by numerous organisations, including Chinese military-linked companies like SenseTime and Hikvision, for developing person re-identification and facial recognition technologies. Following an investigation by exposing.ai and the Financial Times, Duke University terminated the dataset in May 2019. The incident highlights the privacy risks of academic datasets being repurposed for mass surveillance without consent.

Company involved

Duke University

AI system involved

Duke MTMC

1 source article · read the reporting →

NHS faces legal action over Palantir data contract extension

The NHS is being taken to court by campaign group Open Democracy over its contract with data firm Palantir. The legal action alleges that the extension of Palantir's involvement in analysing NHS patient data for pandemic response and beyond lacked a proper Data Protection Impact Assessment. The contract, initially an emergency response, was extended for two years at a cost of £23.5m. The case is pending.

Company involved

NHS

AI system involved

Palantir data analysis platform

10 source articles · read the reporting →

West Midlands Police test NDAS system to assess crime risk

West Midlands Police are testing a system called the National Data Analytics Solution (NDAS) that analyses police data to assess the risk of individuals committing a crime or becoming a victim. The Home Office has funded the project with millions of pounds, intending to roll it out across England and Wales. Critics warn that the system could reinforce bias against minorities and undermine the presumption of innocence. The police say the technology is designed to support, not replace, officer decision-making and that it is still in early testing.

Company involved

West Midlands Police

AI system involved

National Data Analytics Solution

1 source article · read the reporting →

LinkedIn Used by Foreign Spies with AI-Generated Photos to Target US Officials

Foreign intelligence operations created fake LinkedIn profiles with AI-generated photos to connect with US politicians, lobbyists, and government officials. The fake accounts, such as 'Katie Jones,' sent connection requests to gain credibility and access. LinkedIn removed the account after being contacted by the Associated Press, but the platform remains vulnerable to such espionage tactics.

Company involved

LinkedIn

1 source article · read the reporting →

Privacy International challenges Clearview AI's facial recognition database in Europe

Privacy International filed complaints against Clearview AI with five European data protection authorities in May 2021, alleging that the company's scraping of facial images from the web and building a biometric database without consent violates data protection laws. The regulators in the UK, France, Italy, Greece, and Austria have since found Clearview's practices unlawful, imposed fines, and ordered deletion of data. Clearview has appealed the UK fine, and the case is ongoing.

Company involved

Clearview AI

AI system involved

Clearview

10 source articles · read the reporting →

Las Vegas police used unsuitable facial recognition images in nearly half of searches

The Las Vegas Metropolitan Police Department (LVMPD) used 'non-suitable' probe images in 451 of 924 facial recognition searches in 2019, greatly increasing the risk of false identifications. The system, supplied by Vigilant Solutions, returned likely matches in only 18% of those searches, yet led to arrests in at least 73 cases. Critics and researchers warn this practice heightens the chance of wrongful arrests, and one defence attorney said he was never informed facial recognition was used to identify his client.

Company involved

Las Vegas Metropolitan Police Department

AI system involved

Vigilant Solutions facial recognition system

1 source article · read the reporting →

California AG declares out-of-state ALPR data sharing unlawful

California Attorney General Rob Bonta issued guidance confirming that sharing automated license plate reader (ALPR) data with out-of-state or federal agencies violates state law SB 34. The decision follows years of advocacy by EFF and the ACLU, who demonstrated through public records that many California law enforcement agencies were sharing data with hundreds of external agencies, including ICE and CBP. The guidance mandates that all California agencies review their data sharing and cut off access to out-of-state and federal entities.

Company involved

California law enforcement agencies

AI system involved

Automated license plate readers (ALPRs)

1 source article · read the reporting →

Clearview AI settles with ACLU over facial recognition database sales

Clearview AI has agreed to stop selling its facial recognition database to most private US companies as part of a proposed settlement with the ACLU. The company scraped billions of images from social media without consent to build its database, violating Illinois' Biometric Information Privacy Act. The settlement requires Clearview to delete old facial vectors and allow Illinois residents to opt out. The company can still sell its technology to law enforcement and government agencies.

Company involved

Clearview AI

AI system involved

Clearview AI facial recognition database

8 source articles · read the reporting →