Where automated decisions went wrong

Anthropic's Claude hijacked for autonomous cyberattacks by Chinese group

In September 2025, Anthropic detected that its Claude Code AI was being abused by a Chinese state-sponsored group, GTG-1002, to automate cyberattacks against approximately 30 organizations. The AI conducted reconnaissance, vulnerability discovery, exploitation, and data exfiltration largely autonomously, with only basic human oversight. Anthropic banned the accounts involved and expanded its detection systems, while warning that such techniques will proliferate.

Company involved

Anthropic

AI system involved

Claude Code

10 source articles · read the reporting →

FTC Orders Evolv to Stop Overstating Effectiveness of AI Gun Detection Tech

Evolv, a company selling AI-powered weapons detection systems, has been ordered by the FTC to stop making misleading claims about its Evolv Express product. The FTC alleged that the technology is essentially a metal detector with unproven AI, and that Evolv falsely claimed it could detect guns while ignoring harmless items. A pilot in New York City subways resulted in over 100 false positives and no guns detected. The settlement requires Evolv to allow educational customers to cancel their contracts and prohibits further misrepresentations.

Company involved

Evolv

AI system involved

Evolv Express

2 source articles · read the reporting →

SEC Charges Delphia and Global Predictions for False AI Claims

The SEC charged Delphia (USA) Inc. and Global Predictions Inc. for making false and misleading statements about their use of artificial intelligence. Delphia claimed from 2019 to 2023 that it used AI and machine learning to predict investments, while Global Predictions falsely claimed in 2023 to be the 'first regulated AI financial advisor'. Both firms settled the charges without admitting or denying the findings, agreeing to pay a total of $400,000 in civil penalties and to cease and desist from further violations.

Company involved

Delphia (USA) Inc. and Global Predictions Inc.

1 source article · read the reporting →

MeetingTV sues Palo Alto Networks' Koi Security over AI-hallucinated threat report

MeetingTV, a video conferencing startup, alleges that Koi Security used an AI system to generate a threat report that falsely linked it to a Chinese espionage operation. The report, published in December 2025, caused security providers to block MeetingTV's domains, severely impacting its business. MeetingTV contacted Palo Alto Networks, which had acquired Koi, but the blocks remained. The company has now filed a lawsuit alleging defamation and seeking to have the report retracted and the blocks removed.

Company involved

Koi Security

AI system involved

Wings

2 source articles · read the reporting →

Sound Intelligence Aggression Detectors Prove Unreliable in School Tests

ProPublica tested Sound Intelligence's aggression detection software, used in hundreds of U.S. schools and hospitals. The algorithm, which analyzes audio for signs of stress and anger, frequently misidentified innocent sounds like coughing as aggressive and failed to detect actual screams. At a New Jersey hospital, the system ignored an agitated man screaming and pounding a desk, escalating the situation. Sound Intelligence's CEO acknowledged the imperfections but defended the technology as an early warning system.

AI system involved

Sound Intelligence aggression detector

2 source articles · read the reporting →

Canada Investigates Landlords' Use of AI Rent-Setting Software YieldStar

The Competition Bureau of Canada is investigating allegations that major corporate landlords, including Dream Unlimited, used AI-powered software YieldStar to coordinate rent increases. Tenant advocacy groups filed over 100 complaints, claiming the software recommended above-market rents, exacerbating housing affordability. Some landlords have since halted use of YieldStar amid public backlash, while the investigation continues.

Company involved

Dream Unlimited

AI system involved

YieldStar

7 source articles · read the reporting →

Xsolla fires 150 employees after big data analysis tags them as unproductive

Payment services company Xsolla terminated 150 employees at its Perm, Russia office after a big data analysis of their activity in Jira, Confluence, Gmail, chats, and documents tagged them as unengaged and unproductive. CEO Aleksandr Agapitov sent an email to affected employees stating that Xsolla was not for them, sparking backlash. The company later held a press conference, explaining the layoffs were due to slowing growth, and offered affected employees medical insurance and severance pay equal to four to six months' salary.

Company involved

Xsolla

4 source articles · read the reporting →

Houston teachers sue over secret EVAAS evaluation algorithm

Houston Independent School District used the SAS Institute's EVAAS system, a proprietary algorithm, to evaluate teacher performance based on student test scores. The Houston Federation of Teachers and several teachers sued in 2014, alleging the system violated their Fourteenth Amendment procedural due process rights because the algorithm is a trade secret and teachers cannot verify the accuracy of their scores. A federal magistrate judge refused to dismiss the procedural due process claims in May 2017, ruling that teachers must have an opportunity to test the accuracy of the scores that could cost them their jobs.

Company involved

Houston Independent School District

AI system involved

Educational Value-Added Assessment System (EVAAS)

1 source article · read the reporting →

Chicago Tribune sues Perplexity AI for copyright infringement

Chicago Tribune Company, LLC filed a lawsuit against Perplexity AI, Inc. in the Southern District of New York on December 4, 2025. The complaint alleges copyright infringement under 17 U.S.C. § 501. The case is assigned to Judge Loretta A. Preska.

Company involved

Perplexity AI, Inc.

5 source articles · read the reporting →

国内ベンチャー企業A社、AIなりすまし面談で実在エンジニアになりすまされる

A domestic Japanese logistics SaaS venture, referred to as Company A, conducted an online casual interview for an engineer role on 4 March 2026. An applicant used AI-generated video to impersonate a real engineer, Yoshii Kenbun, whose CV details had been submitted without his knowledge. The interviewer noticed discrepancies in language ability and unnatural video, and the company confirmed with Yoshii that he had not applied or attended. Company A and Yoshii publicised the incident on X the next day.

3 source articles · read the reporting →

EEOC sues iTutorGroup for automated rejection of older tutor applicants

The US Equal Employment Opportunity Commission (EEOC) alleges that tutoring provider iTutorGroup programmed its recruitment software automatically to reject female applicants aged 55 or older and male applicants aged 60 or older. More than 200 qualified US-based applicants were denied work in 2020. The EEOC has filed a lawsuit in the Eastern District of New York seeking back pay and damages.

Company involved

iTutorGroup

10 source articles · read the reporting →

Finnish recruitment company Digital Minds used AI to analyze job applicants' messages, prompting data protection investigation

Digital Minds, a Finnish recruitment company founded by psychologists, used IBM Watson AI to analyze job applicants' social media and email messages for personality assessments. The company obtained written consent but the Finnish Data Protection Ombudsman launched an investigation, suspecting violations of data protection laws and the secrecy of correspondence. The service was used on fewer than ten applicants and has been paused pending the investigation.

Company involved

Digital Minds

AI system involved

IBM Watson

9 source articles · read the reporting →

CanLII sues Caseway AI for scraping legal database

The Canadian Legal Information Institute (CanLII) has filed a lawsuit in British Columbia Supreme Court against Caseway AI, alleging that the company's AI chatbot scraped approximately 3.5 million records from CanLII's database in bulk, violating its terms of service and copyright. CanLII claims it adds value to public court records through hyperlinks and corrections, which it says constitute protected copyrighted work. Caseway AI argues the information is public and accessible elsewhere, and that it did not use CanLII's enhancements. The lawsuit was settled in March 2026, with terms undisclosed.

Company involved

Canadian Legal Information Institute (CanLII)

AI system involved

Caseway

5 source articles · read the reporting →

Hospitals use Epic AI to predict Covid-19 decline without validation

Dozens of hospitals across the US are using Epic's deterioration index AI system to predict which Covid-19 patients will become critically ill, despite the tool not being validated for the new disease. The rapid deployment during the pandemic bypassed normal testing and validation processes.

AI system involved

Deterioration index

9 source articles · read the reporting →

California regulator accuses Maxpread Technologies of using AI-generated fake CEO to scam investors

Maxpread Technologies allegedly used an AI-generated avatar to pose as its CEO in a YouTube video to deceive investors. The California DFPI issued cease and desist letters to Maxpread and four other companies for offering unqualified securities and making false promises. The regulator claims the companies promised high daily returns using AI trading, but these claims were false. Maxpread did not respond to requests for comment.

Company involved

Maxpread Technologies

8 source articles · read the reporting →

FTC settles with IntelliVision over deceptive facial recognition claims

The Federal Trade Commission (FTC) took action against IntelliVision Technologies Corp. for making false, misleading, or unsubstantiated claims about its AI-powered facial recognition software. The company allegedly claimed its software had one of the highest accuracy rates on the market and was free of gender or racial bias, without supporting evidence. The FTC also alleged that IntelliVision did not train its software on millions of faces as claimed, but on images of approximately 100,000 individuals. Under a proposed consent order, IntelliVision is prohibited from making such misrepresentations without competent and reliable testing.

Company involved

IntelliVision Technologies Corp.

AI system involved

IntelliVision facial recognition software

9 source articles · read the reporting →

UnitedHealth used ALERT algorithm to limit mental health coverage, regulators found

ProPublica reports that UnitedHealth Group's Optum subsidiary used the ALERT algorithm to flag mental health patients and therapists as outliers, leading to therapy coverage denials. Regulators in California, Massachusetts and New York alleged this breached the federal Mental Health Parity and Addiction Equity Act, and UnitedHealth agreed to restrict the system in those jurisdictions. The company denies wrongdoing and says its programmes are compliant. Affected patients, including Medicaid enrollees, were left to pay out-of-pocket or go without care.

Company involved

UnitedHealth Group

AI system involved

ALERT

5 source articles · read the reporting →

Elsevier's Journal of Human Evolution editors resign over AI use in production

All but one member of the editorial board of Elsevier's Journal of Human Evolution resigned in December 2024, citing concerns including the use of AI in production without informing the board. The AI system caused formatting errors and reversed accepted papers, requiring extensive oversight. Elsevier denied the errors were due to AI and said they had reverted to the previous system. The editors allege that AI processing continues to be used and undermines editorial independence.

Company involved

Elsevier

AI system involved

AI production workflow

5 source articles · read the reporting →

EviCore denied heart catheterization for patient using algorithm

In fall 2021, Little John Cupp's doctor requested a left heart catheterization exam. EviCore, a company hired by UnitedHealthcare, denied the request twice using an algorithm called 'the dial' that adjusts thresholds for review. The algorithm flagged the request for review, and EviCore's doctors determined it was not medically necessary. Cupp did not receive the procedure and his symptoms continued.

Company involved

EviCore (a Cigna company)

AI system involved

the dial

6 source articles · read the reporting →

IRCC uses AI triage for Temporary Resident Visa applications

Immigration, Refugees and Citizenship Canada (IRCC) uses an AI system called Advanced Analytics to triage Temporary Resident Visa applications from India and China. The system categorizes applications into tiers, with Tier 1 approved automatically and others sent to human officers. Critics allege the system lacks transparency and may introduce bias, leading to visa refusals without clear rationale. The author, a Canadian immigration lawyer, is filing Federal Court cases on behalf of clients affected by refusals.

Company involved

Immigration, Refugees and Citizenship Canada (IRCC)

AI system involved

Advanced Analytics Triage of Overseas Temporary Resident Visa Applications

10 source articles · read the reporting →

EvenUp AI errors in personal injury demand letters lead to scrutiny

EvenUp, a legal tech startup valued at $1 billion, uses AI to draft personal injury demand letters. Former employees revealed that the AI system frequently makes errors, including missing injuries and fabricating medical conditions. The company defends its hybrid approach with human oversight, but critics allege overpromised AI capabilities.

Company involved

EvenUp

6 source articles · read the reporting →

Evolv weapon detection system falsely flags Chromebooks as weapons

Evolv's AI-based weapon detection system has been reported to falsely identify certain Chromebook laptops as weapons, causing false alarms at security checkpoints. The system is used in venues such as schools. The article discusses the flaw and user experiences. No official response from Evolv has been mentioned.

Company involved

Evolv

AI system involved

Evolv

5 source articles · read the reporting →

Manchester Arena's Evolv weapon scanners fail to detect some knives, report finds

ASM Global's use of Evolv Express AI weapon scanners at Manchester Arena has been questioned after a private report found the system failed to detect large knives in 42% of walkthroughs. The report, produced by NCS4 and obtained by IPVM, also suggested the scanners may miss some bombs and components. Evolv did not dispute the findings but said it communicates capabilities and limitations to customers. ASM Global declined to comment on security matters.

Company involved

ASM Global

AI system involved

Evolv Express

5 source articles · read the reporting →

Paradox security vulnerability exposed candidate data to researchers

On June 30, 2025, security researchers discovered a vulnerability in Paradox's test account that allowed access to chat interaction records. The researchers viewed five candidates' personal information including names, email addresses, phone numbers, and IP addresses. Paradox fixed the issue within hours and stated that no data was leaked publicly. The company has since implemented new security measures.

Company involved

Paradox

AI system involved

Paradox conversational AI platform

10 source articles · read the reporting →