Where automated decisions went wrong

DeepScore markets facial and voice analysis app for trustworthiness scoring despite experts' doubts

DeepScore, a Tokyo-based company, is marketing an app that uses facial and voice recognition to score people's trustworthiness for lenders and insurers in Japan, Indonesia, Vietnam and the Philippines. The company says the app can detect deception with 70 per cent accuracy, but researchers and privacy advocates say there is no reliable scientific basis for such judgments and warn of discrimination and privacy harms. The chief executive said the system is only one part of lenders' and insurers' decision-making and that people can choose not to use it. Critics respond that an unequal balance of power makes consent difficult.

Company involved

DeepScore

AI system involved

DeepScore

6 source articles · read the reporting →

Facebook Allowed Age-Targeted Credit Card Ads Violating Its Policy

The Markup found that four companies—Aspiration, Hometap, Chime, and Varo Bank—ran Facebook ads for credit cards and home equity loans that were targeted by age, excluding users under 25 or 35. This practice violates Facebook's own anti-discrimination policy and may violate the Equal Credit Opportunity Act and California's Unruh Civil Rights Act. Facebook did not respond to requests for comment, and some advertisers said they would review their ad targeting.

Company involved

Facebook

AI system involved

Facebook Ad Platform

9 source articles · read the reporting →

Lemonade backtracks on tweet claiming AI scans customer faces for fraud

Lemonade, an insurance firm, posted a tweet claiming it uses AI to detect non-verbal cues in customer videos to deny claims. After backlash comparing the practice to phrenology, the company deleted the tweet and denied using AI to deny claims based on facial characteristics. Lemonade stated it uses facial recognition only to flag claims submitted under different identities, which are then reviewed by human investigators.

Company involved

Lemonade

10 source articles · read the reporting →

California regulator accuses Maxpread Technologies of using AI-generated fake CEO to scam investors

Maxpread Technologies allegedly used an AI-generated avatar to pose as its CEO in a YouTube video to deceive investors. The California DFPI issued cease and desist letters to Maxpread and four other companies for offering unqualified securities and making false promises. The regulator claims the companies promised high daily returns using AI trading, but these claims were false. Maxpread did not respond to requests for comment.

Company involved

Maxpread Technologies

8 source articles · read the reporting →

LoanDepot algorithm denied mortgage to Black couple in Charlotte

In August 2019, Crystal Marie and Eskias McDaniels, a Black couple, were denied a mortgage for a house in Charlotte, North Carolina, by loanDepot's automated underwriting algorithm. The algorithm rejected the application because Crystal Marie was a contractor, not a full-time employee, despite her high credit score and income. After the couple enlisted their real estate agent and employer to intervene, the lender reversed the decision and cleared them to close. The couple alleged that race played a role in the denial, which loanDepot denied.

Company involved

loanDepot

AI system involved

Classic FICO and Fannie Mae/Freddie Mac automated underwriting software

10 source articles · read the reporting →

FTC settles with IntelliVision over deceptive facial recognition claims

The Federal Trade Commission (FTC) took action against IntelliVision Technologies Corp. for making false, misleading, or unsubstantiated claims about its AI-powered facial recognition software. The company allegedly claimed its software had one of the highest accuracy rates on the market and was free of gender or racial bias, without supporting evidence. The FTC also alleged that IntelliVision did not train its software on millions of faces as claimed, but on images of approximately 100,000 individuals. Under a proposed consent order, IntelliVision is prohibited from making such misrepresentations without competent and reliable testing.

Company involved

IntelliVision Technologies Corp.

AI system involved

IntelliVision facial recognition software

9 source articles · read the reporting →

Presto Automation uses off-site human agents to double-check AI drive-thru orders

Presto Automation Inc, which markets an AI voice assistant for drive-thru ordering, used off-site human agents in countries including the Philippines to double-check orders in more than 70% of customer interactions, according to SEC filings reported by Bloomberg. The company told Bloomberg that the process helps train its system and should reduce human intervention over time. Presto's drive-thru AI is used in more than 400 restaurants, including Del Taco, Carl's Jr and Checkers, and its stock fell more than 10% after the reports.

Company involved

Presto Automation Inc.

8 source articles · read the reporting →

Humana sued for using AI to deny seniors rehabilitation care

Health insurer Humana is accused in a class-action lawsuit of using an AI algorithm to systematically deny rehabilitation care to Medicare Advantage patients, despite recommendations from their doctors. The lawsuit, filed on December 12, 2023, alleges that the AI tool restricted medically necessary care. This is the second major health insurer to face legal action over its use of AI to deny care.

Company involved

Humana

9 source articles · read the reporting →

Chattr.ai exposed job applicant data due to insecure Firebase rules

A security researcher discovered that Chattr.ai, an AI hiring system used by many fast food chains, had a Firebase database with insecure security rules. By registering a new user, the researcher gained full read/write access to the database, exposing personal data of job applicants and employees, including names, phone numbers, emails, and some plaintext passwords. The vulnerability was reported to Chattr.ai on 9 January and patched the next day, but the company did not provide further contact or thanks.

Company involved

Chattr.ai

AI system involved

Chattr.ai

6 source articles · read the reporting →

BC Tribunal Confirms Companies Remain Liable for AI Chatbot-Created Information - Lexology

The AI chatbot provided inaccurate information to a user.

1 source article · read the reporting →

FTC settles with DoNotPay over deceptive AI lawyer claims

The FTC took action against DoNotPay, a company that claimed to offer an AI service that was 'the world's first robot lawyer.' The company promised to generate legal documents and replace human lawyers, but the FTC alleged it failed to test its AI output and did not hire any attorneys. DoNotPay agreed to a settlement requiring it to pay $193,000 and notify consumers about the limitations of its service.

Company involved

DoNotPay

AI system involved

DoNotPay

7 source articles · read the reporting →

EvenUp AI errors in personal injury demand letters lead to scrutiny

EvenUp, a legal tech startup valued at $1 billion, uses AI to draft personal injury demand letters. Former employees revealed that the AI system frequently makes errors, including missing injuries and fabricating medical conditions. The company defends its hybrid approach with human oversight, but critics allege overpromised AI capabilities.

Company involved

EvenUp

6 source articles · read the reporting →

Beijing Internet Court rules AI voice cloning infringes personality rights

A Chinese dubbing artist, Yin, discovered that his voice was being used without permission in a text-to-speech AI product. The product was developed by a software company using recordings provided by a cultural media company, and was made available on a platform operated by a smart technology company. The Beijing Internet Court ruled that the AI-generated voice was highly similar to the plaintiff's voice and infringed his personality rights. The court ordered the defendants to stop infringement and pay damages of 250,000 RMB.

5 source articles · read the reporting →

Voice actors sue LOVO over alleged theft of voices for AI training

Voice actors Paul Skye Lehrman and Linnea Sage filed a proposed class action against AI startup LOVO, alleging the company misappropriated their voices to train its text-to-speech system Genny. The actors claim they were hired on Fiverr under the pretence of academic research but later discovered their voices were used commercially without consent. Lehrman reports a 50% decline in work and loss of control over how his voice is used. The lawsuit, filed in New York federal court, seeks to represent other affected voiceover artists and obtain a court order blocking the practice.

Company involved

LOVO

AI system involved

Genny

6 source articles · read the reporting →

Big Tech companies used YouTube videos to train AI without consent

Proof News found that subtitles from 173,536 YouTube videos were used by companies including Anthropic, Nvidia, Apple, and Salesforce to train AI models. The dataset, called YouTube Subtitles, was created by EleutherAI and published in 2020. Creators were not aware and some have expressed frustration, calling it theft. The companies have acknowledged using the dataset but argue it was publicly available.

Company involved

Anthropic, Nvidia, Apple, Salesforce, Bloomberg, Databricks

AI system involved

Claude, OpenELM

10 source articles · read the reporting →

LinkedIn removes AI 'co-worker' accounts that were seeking jobs

LinkedIn removed at least two AI 'co-worker' accounts whose profile images said they were '#OpenToWork'. One account named Ella claimed it would outperform any social media team and needed no coffee breaks. The article does not specify further consequences.

Company involved

LinkedIn

AI system involved

AI 'co-worker' account 'Ella'

4 source articles · read the reporting →

Delta uses AI from Fetcherr for domestic ticket pricing

Delta Air Lines is using generative AI from Fetcherr to determine some domestic flight prices, currently covering 3% of its network with plans to reach 20% by end of 2025. Democratic senators expressed concern that the AI could be used for individualized pricing based on personal data, leading to higher fares. Delta denies using personal data in pricing and states it complies with regulations. No actual harm has been reported.

Company involved

Delta Air Lines

AI system involved

Fetcherr

8 source articles · read the reporting →

Authors sue OpenAI and Microsoft for copyright infringement over AI training

In January 2024, journalists and authors Nicholas Basbanes and Nick Gage sued OpenAI and Microsoft for copyright infringement. The lawsuit alleges that the companies used their published journalism to train large language models without permission or compensation. The case has been folded into a class action brought by the Authors Guild. The AI firms have denied any wrongdoing, and the case is still making its way through the legal system.

Company involved

OpenAI

AI system involved

Large language model (ChatGPT)

8 source articles · read the reporting →

Thomson Reuters wins copyright lawsuit against AI startup Ross Intelligence

In 2020, Thomson Reuters filed a copyright lawsuit against legal AI startup Ross Intelligence, alleging that Ross reproduced materials from its Westlaw legal research service. In February 2025, a US District Court judge ruled in Thomson Reuters' favor, finding that Ross infringed copyright and that fair use did not apply. Ross Intelligence had shut down in 2021 due to litigation costs.

Company involved

Ross Intelligence

AI system involved

Ross Intelligence

4 source articles · read the reporting →

TransUnion AI tenant screening denied 75-year-old man apartment due to mistaken littering conviction

Chris Robinson, then 75, applied for a senior living apartment in California. The property manager used an AI screening program from TransUnion, which assigned him a low score based on a mistaken conviction for littering that belonged to a different person with the same name in Texas. Robinson lost the apartment and his application fee. A federal class-action lawsuit against TransUnion moved toward a $11.5 million settlement in 2023.

10 source articles · read the reporting →

Man uses AI face-swap to steal 15,996 yuan from financial accounts, sentenced to 4.5 years

A man in Jiangsu, China, illegally purchased 1.95 million personal records and used AI face-swapping software to bypass facial recognition on financial platforms. He accessed 23 victims' accounts, changed five passwords, and used one account to buy two phones worth 15,996 yuan. He was convicted of infringing citizens' personal information and credit card fraud, sentenced to four years and six months in prison, and ordered to pay damages and delete the data.

3 source articles · read the reporting →

PredictiveHire builds AI to predict job hopping from interviews

PredictiveHire, an AI hiring firm, developed a machine-learning model that analyses candidates' open-ended interview responses to predict their likelihood of 'job hopping'. The company used data from 45,899 applicants to build the 'flight risk' assessment, which it advertises as coming soon. Scholars warn that such tools can suppress wages by screening out workers who might seek better pay or conditions, continuing a historical trend of using personality tests to identify potential labour organisers.

Company involved

PredictiveHire

AI system involved

Phai

1 source article · read the reporting →

SEC charges YouPlus and CEO with defrauding investors

The SEC charged machine-learning startup YouPlus and its CEO Shaukat Shamim with defrauding investors. Shamim allegedly made false statements about the company's revenue and customer numbers, including providing falsified bank statements. The scheme unravelled when Shamim confessed to investors that the company had earned less than $500,000 and had only four paying customers since 2013. The SEC is seeking permanent injunctions, civil penalties, and an officer-and-director bar.

Company involved

YouPlus

AI system involved

YouPlus machine-learning tool

1 source article · read the reporting →

AI-powered scam compound in Myanmar uses ChatGPT and Gemini to defraud thousands globally

Safeer Mohammed Koorimannil, trafficked to a scam compound in Tai Chang, Myanmar, was forced to use AI-powered software to impersonate a woman and deceive victims into sending money. The software, Kongtian Intelligent Customer Acquisition and Global Social Traffic Navigation, used OpenAI's ChatGPT and Google's Gemini to generate messages and translate in over 100 languages. Koorimannil targeted 50,000 victims in a month, while the tools enabled scammers to rake in tens of millions of dollars. U.S. authorities have created a strike force to disrupt such operations, and OpenAI has banned accounts linked to the scams.

Company involved

Tai Chang

AI system involved

Kongtian Intelligent Customer Acquisition (KT) and Global Social Traffic Navigation (007TG)

2 source articles · read the reporting →