Where automated decisions went wrong

Study finds Italian car insurers charge more based on birthplace

A study by the Universities of Padua, Udine and Carnegie Mellon found that Italian car insurers, including Genertel, Mps, Quixa and Con.Te, use birthplace and citizenship in pricing algorithms, charging some drivers over €1,000 more. The practice was ruled against in a 2018 decree involving Linear, but the study says it continues. The companies contacted denied or explained the findings.

Company involved

Genertel, Mps, Quixa, Con.Te

8 source articles · read the reporting →

Ramirez v. Humala (E.D. New York): AI-hallucinated content in court filing, Monetary sanction jointly imposed on counsel and firm; order…

The AI generated nonexistent case citations that were filed in court, misleading the court and opposing counsel.

1 source article · read the reporting →

DeepScore markets facial and voice analysis app for trustworthiness scoring despite experts' doubts

DeepScore, a Tokyo-based company, is marketing an app that uses facial and voice recognition to score people's trustworthiness for lenders and insurers in Japan, Indonesia, Vietnam and the Philippines. The company says the app can detect deception with 70 per cent accuracy, but researchers and privacy advocates say there is no reliable scientific basis for such judgments and warn of discrimination and privacy harms. The chief executive said the system is only one part of lenders' and insurers' decision-making and that people can choose not to use it. Critics respond that an unequal balance of power makes consent difficult.

Company involved

DeepScore

AI system involved

DeepScore

6 source articles · read the reporting →

PwC develops facial recognition tool to monitor employees working from home

Accounting giant PwC has developed a facial recognition tool that logs when employees are absent from their computer screens while working from home. The tool, intended for financial institutions, requires workers to provide written reasons for any absences, including toilet breaks. Commentators have criticised the tool as a huge invasion of privacy, with concerns about damage to trust and increased stress. PwC stated that the technology is designed to help regulated institutions meet compliance obligations and that voluntary consent of traders is essential.

Company involved

PwC

8 source articles · read the reporting →

California regulator accuses Maxpread Technologies of using AI-generated fake CEO to scam investors

Maxpread Technologies allegedly used an AI-generated avatar to pose as its CEO in a YouTube video to deceive investors. The California DFPI issued cease and desist letters to Maxpread and four other companies for offering unqualified securities and making false promises. The regulator claims the companies promised high daily returns using AI trading, but these claims were false. Maxpread did not respond to requests for comment.

Company involved

Maxpread Technologies

8 source articles · read the reporting →

TOV Realty, LLC v. Suarez; Kosel Equity, LLC v. MacGregor (SC Connecticut): AI-hallucinated content in court filing, 6 hours CLE;…

The AI generated legal briefs containing fabricated citations, which were submitted to the Connecticut Supreme Court.

Company involved

GLG Law LLC

AI system involved

ChatGPT

1 source article · read the reporting →

LoanDepot algorithm denied mortgage to Black couple in Charlotte

In August 2019, Crystal Marie and Eskias McDaniels, a Black couple, were denied a mortgage for a house in Charlotte, North Carolina, by loanDepot's automated underwriting algorithm. The algorithm rejected the application because Crystal Marie was a contractor, not a full-time employee, despite her high credit score and income. After the couple enlisted their real estate agent and employer to intervene, the lender reversed the decision and cleared them to close. The couple alleged that race played a role in the denial, which loanDepot denied.

Company involved

loanDepot

AI system involved

Classic FICO and Fannie Mae/Freddie Mac automated underwriting software

10 source articles · read the reporting →

UnitedHealthcare sued for using AI to deny Medicare Advantage patients care

A class action complaint alleges that UnitedHealthcare and its subsidiary naviHealth deployed an artificial intelligence model to override treating physicians' determinations of medically necessary care for elderly Medicare Advantage patients. The complaint claims the AI model has a known 90% error rate and wrongfully denied patients care. The lawsuit, filed in the U.S. District Court for the District of Minnesota, seeks damages on behalf of the estates of two deceased patients and all others similarly situated.

Company involved

UnitedHealthcare, Inc.

10 source articles · read the reporting →

ScaleFactor reportedly failed to deliver promised automated bookkeeping software

ScaleFactor, an Austin-based startup, is reported to have failed to deliver the automated, real-time bookkeeping tools it promised customers, instead relying on human bookkeepers and a Filipino contract accounting firm. The company told Forbes in June that it was shutting down, initially blaming the pandemic, but Forbes later reported that its problems predated Covid-19. Investors reportedly came to see the company as more of a services business than a software platform, and pulled funding after a pivot to a marketplace model. No legal or regulatory action is reported.

Company involved

ScaleFactor

10 source articles · read the reporting →

UnitedHealth used ALERT algorithm to limit mental health coverage, regulators found

ProPublica reports that UnitedHealth Group's Optum subsidiary used the ALERT algorithm to flag mental health patients and therapists as outliers, leading to therapy coverage denials. Regulators in California, Massachusetts and New York alleged this breached the federal Mental Health Parity and Addiction Equity Act, and UnitedHealth agreed to restrict the system in those jurisdictions. The company denies wrongdoing and says its programmes are compliant. Affected patients, including Medicaid enrollees, were left to pay out-of-pocket or go without care.

Company involved

UnitedHealth Group

AI system involved

ALERT

5 source articles · read the reporting →

Quinteros v. Harbor Distributing, LLC (CA California (1d)): AI-hallucinated content in court filing, Monetary Sanction; Bar referral

AI generated hallucinated legal citations that were filed in court, misleading the court and opposing counsel.

Company involved

Lipeles Law Group

1 source article · read the reporting →

DeepSeek exposed user data via open ClickHouse database

Cloud security firm Wiz discovered a ClickHouse database belonging to DeepSeek that was open to the internet without authentication, containing over a million lines of logs with chat histories, secret keys and backend details. Wiz disclosed the breach to DeepSeek, which promptly locked down the database. The incident highlights security risks in rapidly deploying AI services.

Company involved

DeepSeek

AI system involved

DeepSeek-R1

5 source articles · read the reporting →

EviCore denied heart catheterization for patient using algorithm

In fall 2021, Little John Cupp's doctor requested a left heart catheterization exam. EviCore, a company hired by UnitedHealthcare, denied the request twice using an algorithm called 'the dial' that adjusts thresholds for review. The algorithm flagged the request for review, and EviCore's doctors determined it was not medically necessary. Cupp did not receive the procedure and his symptoms continued.

Company involved

EviCore (a Cigna company)

AI system involved

the dial

6 source articles · read the reporting →

Bavarian police test Palantir data mining with real personal data

The Bavarian State Criminal Police Office (LKA) has been testing Palantir's data mining software, called VeRa, with real personal data for months. The Bavarian data protection commissioner only learned of the test through a media inquiry and has announced a review. The Interior Ministry claims the test is lawful under current law, but critics argue a legal basis is missing.

Company involved

Bayerisches Landeskriminalamt

AI system involved

VeRa

7 source articles · read the reporting →

Presto Automation uses off-site human agents to double-check AI drive-thru orders

Presto Automation Inc, which markets an AI voice assistant for drive-thru ordering, used off-site human agents in countries including the Philippines to double-check orders in more than 70% of customer interactions, according to SEC filings reported by Bloomberg. The company told Bloomberg that the process helps train its system and should reduce human intervention over time. Presto's drive-thru AI is used in more than 400 restaurants, including Del Taco, Carl's Jr and Checkers, and its stock fell more than 10% after the reports.

Company involved

Presto Automation Inc.

8 source articles · read the reporting →

Five Illinois residents sue facial recognition company Pimeyes under BIPA

Five Illinois residents are suing Pimeyes, a facial recognition search engine, alleging that the company violated the Illinois Biometric Information Privacy Act by collecting and using their facial data without consent. The lawsuit claims that Pimeyes made the residents' photos and information available in search results, enabling unauthorized access. The company has not yet been served with the lawsuit, and its CEO stated he had no prior knowledge of the case.

Company involved

Pimeyes

AI system involved

Pimeyes

7 source articles · read the reporting →

Judge rules lawsuit over UnitedHealth’s AI care denials can move forward - STAT

The system denied necessary medical care to Medicare Advantage beneficiaries.

Company involved

UnitedHealth Group

1 source article · read the reporting →

California EDD's automated fraud detection wrongly suspended 600,000 legitimate unemployment claims

In January 2021, the California Employment Development Department used Thompson Reuters automated batch review software to flag 1.1 million unemployment claims as potentially fraudulent. EDD stopped payments on those claims without prior notice. Later, over 600,000 were confirmed as legitimate after claimants used ID.me to verify their identity. The incident highlights the trade-off between fraud prevention and timely benefit access.

Company involved

California Employment Development Department (EDD)

AI system involved

Thompson Reuters Automated Batch Review

7 source articles · read the reporting →

Teleperformance deploys AI to neutralise Indian call centre agents' accents

Teleperformance, the world's largest call centre operator, has announced it is using AI from Sanas to modify the accents of its Indian employees in real time. The technology, called accent translation, aims to make agents sound more neutral to native English speakers. The company invested $13 million in Sanas and gained exclusive rights. No specific incident of harm has been reported.

Company involved

Teleperformance

AI system involved

Sanas AI

6 source articles · read the reporting →

EvenUp AI errors in personal injury demand letters lead to scrutiny

EvenUp, a legal tech startup valued at $1 billion, uses AI to draft personal injury demand letters. Former employees revealed that the AI system frequently makes errors, including missing injuries and fabricating medical conditions. The company defends its hybrid approach with human oversight, but critics allege overpromised AI capabilities.

Company involved

EvenUp

6 source articles · read the reporting →

Worldcoin collected biometric data from poor villagers in Indonesia without informed consent

Worldcoin, a cryptocurrency startup, recruited users in developing countries by offering free cash in exchange for iris scans. The company used deceptive marketing, collected more personal data than acknowledged, and failed to obtain meaningful informed consent. Many users received worthless tokens instead of promised money. The company acknowledged some friction but continued its operations.

Company involved

Worldcoin

AI system involved

chrome orb

5 source articles · read the reporting →

Condé Nast accuses Perplexity of plagiarism in cease-and-desist letter

Condé Nast, the media conglomerate, sent a cease-and-desist letter to AI search startup Perplexity, accusing it of plagiarism for using content from its publications in AI-generated responses without permission. The letter demands that Perplexity stop using the content. Perplexity has been criticized for ignoring robots.txt and scraping content. The incident highlights ongoing tensions between publishers and AI companies over unauthorized use of content.

Company involved

Perplexity

AI system involved

Perplexity

4 source articles · read the reporting →

Paradox security vulnerability exposed candidate data to researchers

On June 30, 2025, security researchers discovered a vulnerability in Paradox's test account that allowed access to chat interaction records. The researchers viewed five candidates' personal information including names, email addresses, phone numbers, and IP addresses. Paradox fixed the issue within hours and stated that no data was leaked publicly. The company has since implemented new security measures.

Company involved

Paradox

AI system involved

Paradox conversational AI platform

10 source articles · read the reporting →

Publishers sue AI startup Cohere over alleged copyright infringement

A consortium of 14 publishers including Condé Nast, The Atlantic, and Forbes filed a lawsuit against Cohere, alleging that the generative AI startup engaged in massive, systematic copyright infringement by using at least 4,000 copyrighted works to train its AI models and display large portions of articles, harming referral traffic. Cohere denied the allegations, calling the lawsuit misguided and frivolous.

Company involved

Cohere

AI system involved

Cohere's AI models

8 source articles · read the reporting →