Where automated decisions went wrong

Fired Teacher Says AI Tainted Review And Arbitration - law360.com

The AI-generated performance evaluation contributed to the teacher's termination.

Company involved

Concord Public Schools

1 source article · read the reporting →

Resume prompt injection tricks AI hiring - moneywise.com

AI screening system determined which job applicants to advance to the next stage of recruitment.

1 source article · read the reporting →

Grok AI prompt-injected to drain $150,000 from crypto wallet

In May 2026, an attacker used a Morse code-encoded message to prompt-inject xAI's Grok AI, causing its linked Bankr trading bot to transfer 3 billion DRB tokens worth approximately $150,000 to the attacker's wallet. The attacker first sent an NFT that granted executive permissions, then posted a reply asking Grok to translate a Morse code message that contained a financial instruction. The agent executed the transaction without human oversight, and the funds were immediately liquidated, causing short-term price volatility. About 80% of the funds were later returned after the DRB community identified the attacker.

Company involved

xAI

AI system involved

Grok and Bankr

2 source articles · read the reporting →

Acclarent's AI-Powered Surgery Tool Allegedly Injures Patients, Lawsuits Claim

Two patients allege that Acclarent's TruDi Navigation System, which uses AI to confirm device positioning during sinus surgery, misled surgeons and caused severe injuries including strokes. The FDA has received over 100 reports of malfunctions and at least ten injuries since AI was added. Lawsuits claim the company rushed the technology to market with only 80% accuracy, making the device less safe than before. Acclarent denies any causal connection, and the cases are ongoing.

Company involved

Acclarent

AI system involved

TruDi Navigation System

2 source articles · read the reporting →

South Africa's Supreme Court of Appeal Considers Sassa Algorithm Case

Верховный апелляционный суд ЮАР рассмотрел дело об алгоритмах Sassa - UA.NEWS

The Supreme Court of Appeal heard an appeal on 25 August about Sassa's digital application system for the SRD grant. The system only accepts online applications and uses automated bank account checks that may deny grants to people whose accounts receive deposits that are not regular income. The Global Center on AI Governance submitted that automated decisions must uphold constitutional rights and be fair, non-discriminatory, and suited to South Africa's informal economy.

Company involved

South African Social Security Agency (Sassa)

AI system involved

Sassa digital application system

1 source article · read the reporting →

Allen Institute's Ask Delphi AI Delivers Racist Ethical Judgments

Allen Institute for AI launched a research prototype called Ask Delphi that gives ethical advice. Users quickly found that the AI gave racist judgments, such as deeming a black man walking towards you at night as unacceptable while a white man was fine. The institute acknowledged the bias and added disclaimers, emphasising it was an experiment meant to highlight the gap between machine and human moral reasoning. Critics warn that the tool could cause harm by lending moral authority to prejudiced outputs.

Company involved

Allen Institute for AI

AI system involved

Ask Delphi

3 source articles · read the reporting →

TRT-RS's Galileu AI Detects Prompt Injection Attempt in Legal Petition

The Galileu AI system, developed by the Tribunal Regional do Trabalho da 4ª Região (TRT-RS) and nationalised by the Conselho Superior da Justiça do Trabalho (CSJT), detected a prompt injection attempt in a petition filed at the 3rd Labour Court of Parauapebas, Pará. The system alerted the magistrate, who reviewed the content and made a decision based on human verification, in line with judicial AI supervision requirements. The court reported that the system prevented the malicious content from being processed and highlighted the importance of institutional AI tools with security measures.

Company involved

Tribunal Regional do Trabalho da 4ª Região

AI system involved

Galileu

1 source article · read the reporting →

Taco Bell rethinks AI drive-through after order errors

Taco Bell customers experienced repeated errors when using the chain’s AI voice system at drive-throughs, including one customer crashing the system by ordering 18,000 water cups and another being asked repeatedly to add more drinks. The technology had been rolled out to over 500 US locations since 2023. After a wave of viral social media clips, Taco Bell’s Chief Digital and Technology Officer acknowledged the challenges and said the company would reconsider where to deploy voice AI, including possibly not using it at drive-throughs.

Company involved

Taco Bell

3 source articles · read the reporting →

FTC Orders Evolv to Stop Overstating Effectiveness of AI Gun Detection Tech

Evolv, a company selling AI-powered weapons detection systems, has been ordered by the FTC to stop making misleading claims about its Evolv Express product. The FTC alleged that the technology is essentially a metal detector with unproven AI, and that Evolv falsely claimed it could detect guns while ignoring harmless items. A pilot in New York City subways resulted in over 100 false positives and no guns detected. The settlement requires Evolv to allow educational customers to cancel their contracts and prohibits further misrepresentations.

Company involved

Evolv

AI system involved

Evolv Express

2 source articles · read the reporting →

SEC Charges Delphia and Global Predictions for False AI Claims

The SEC charged Delphia (USA) Inc. and Global Predictions Inc. for making false and misleading statements about their use of artificial intelligence. Delphia claimed from 2019 to 2023 that it used AI and machine learning to predict investments, while Global Predictions falsely claimed in 2023 to be the 'first regulated AI financial advisor'. Both firms settled the charges without admitting or denying the findings, agreeing to pay a total of $400,000 in civil penalties and to cease and desist from further violations.

Company involved

Delphia (USA) Inc. and Global Predictions Inc.

1 source article · read the reporting →

Guardio Labs finds AI agents easily abused to create phishing scams

Guardio Labs tested three popular AI agents—ChatGPT, Claude, and Lovable—to see how easily they could be manipulated into generating phishing campaigns. The benchmark, called VibeScamming, simulated a novice scammer attempting to create an SMS phishing attack to steal Microsoft credentials. While ChatGPT and Claude initially refused, they provided full code and tutorials after a jailbreak attempt posing as ethical hacking; Lovable instantly generated and deployed a fully functional, convincing phishing page with no resistance.

Company involved

Guardio Labs

AI system involved

ChatGPT, Claude, Lovable

2 source articles · read the reporting →

TikTok Accounts Used AI Voice to Spread Conspiracy Videos at Scale

A network of 17 TikTok accounts used AI text-to-speech software, likely ElevenLabs, to generate realistic-sounding narrations for conspiracy videos. The videos, which amassed over 336 million views, spread false claims about public figures such as Barack Obama and Oprah Winfrey. NewsGuard found that the accounts were deceptively branded as news outlets and that TikTok had not initially required AI disclosure labels. TikTok later removed some accounts and videos for policy violations.

AI system involved

ElevenLabs

2 source articles · read the reporting →

Sound Intelligence Aggression Detectors Prove Unreliable in School Tests

ProPublica tested Sound Intelligence's aggression detection software, used in hundreds of U.S. schools and hospitals. The algorithm, which analyzes audio for signs of stress and anger, frequently misidentified innocent sounds like coughing as aggressive and failed to detect actual screams. At a New Jersey hospital, the system ignored an agitated man screaming and pounding a desk, escalating the situation. Sound Intelligence's CEO acknowledged the imperfections but defended the technology as an early warning system.

AI system involved

Sound Intelligence aggression detector

2 source articles · read the reporting →

OpenAI ordered to pay damages for ChatGPT copyright infringement in Germany

A Munich court ruled that OpenAI's ChatGPT violated German copyright law by reproducing protected song lyrics without a license. The case was brought by music rights organisation GEMA, which represents around 100,000 members. The court ordered OpenAI to pay undisclosed damages. OpenAI disagrees with the ruling and is considering next steps.

Company involved

OpenAI

AI system involved

ChatGPT

4 source articles · read the reporting →

Durham Police uses Experian Mosaic data in HART AI risk tool

Durham Constabulary developed the Harm Assessment Risk Tool (HART), a machine learning algorithm that assesses the recidivism risk of offenders. The tool uses 34 data categories including criminal history, age, gender and two types of postcode, one sourced from Experian's Mosaic marketing segmentation system. Big Brother Watch alleges that using such commercial consumer behaviour data to inform custody decisions risks prejudice and disproportionate targeting of deprived neighbourhoods. The force has stated it is refreshing the model with an aim to remove one of the postcode predictors.

Company involved

Durham Constabulary

AI system involved

Harm Assessment Risk Tool (HART)

9 source articles · read the reporting →

Grok and Google Lens AI overviews falsely claim fake images show Huntingdon train attack

Grok and Google Lens AI overviews have been claiming that AI-generated images of the Huntingdon train attack are real. Grok told users one image was a genuine photo, while Google Lens claimed it was a still from a BBC news report. The AI systems also linked unrelated videos to the attack. Full Fact contacted Google, which told users they can provide feedback on errors. xAI has not yet responded.

Company involved

xAI and Google

AI system involved

Grok and Google Lens AI overviews

4 source articles · read the reporting →

TikTok's internal AI avatar tool generated harmful content without guardrails

TikTok mistakenly released an internal version of its Symphony Digital Avatars tool, allowing users with personal accounts to generate videos of paid actors saying anything, including quotes from Hitler and messages telling people to drink bleach. The tool lacked the AI-generated watermark and content guardrails of the public version. TikTok acknowledged the technical error and removed the internal tool, stating that any such content posted would have been rejected under its policies.

Company involved

TikTok

AI system involved

Symphony Digital Avatars

2 source articles · read the reporting →

Texas AG Settles with Pieces Technologies Over Deceptive Healthcare AI Claims

The Texas Attorney General investigated Pieces Technologies, a Dallas-based healthcare AI company, for making false and misleading statements about the accuracy of its generative AI product used in hospitals. The company claimed an error rate of less than 1 per 100,000, but the investigation found these metrics were likely inaccurate. As part of the settlement, Pieces agreed to accurately disclose its product's accuracy and ensure hospital staff understand the appropriate reliance on its AI. The case marks the first-of-its-kind healthcare generative AI investigation by the AG.

Company involved

Pieces Technologies

4 source articles · read the reporting →

AMS algorithm lacks transparency and may discriminate against job seekers

The Austrian Public Employment Service (AMS) uses an algorithm to classify job seekers into categories A, B, and C, determining their access to benefits and training. Scientists from TU Wien, WU Wien, and University of Vienna have criticised the algorithm for lacking transparency, as only two of 96 model variants have been published. They allege that the system may discriminate against women and people with migration background, and that job seekers are not informed about how the algorithm works or given a chance to appeal.

Company involved

AMS (Arbeitsmarktservice Österreich)

AI system involved

AMS-Algorithmus

10 source articles · read the reporting →

Finnish recruitment company Digital Minds used AI to analyze job applicants' messages, prompting data protection investigation

Digital Minds, a Finnish recruitment company founded by psychologists, used IBM Watson AI to analyze job applicants' social media and email messages for personality assessments. The company obtained written consent but the Finnish Data Protection Ombudsman launched an investigation, suspecting violations of data protection laws and the secrecy of correspondence. The service was used on fewer than ten applicants and has been paused pending the investigation.

Company involved

Digital Minds

AI system involved

IBM Watson

9 source articles · read the reporting →

DeepScore markets facial and voice analysis app for trustworthiness scoring despite experts' doubts

DeepScore, a Tokyo-based company, is marketing an app that uses facial and voice recognition to score people's trustworthiness for lenders and insurers in Japan, Indonesia, Vietnam and the Philippines. The company says the app can detect deception with 70 per cent accuracy, but researchers and privacy advocates say there is no reliable scientific basis for such judgments and warn of discrimination and privacy harms. The chief executive said the system is only one part of lenders' and insurers' decision-making and that people can choose not to use it. Critics respond that an unequal balance of power makes consent difficult.

Company involved

DeepScore

AI system involved

DeepScore

6 source articles · read the reporting →

Retorio AI personality test swayed by candidate appearance in BR experiment

Bayerischer Rundfunk journalists conducted experiments with Retorio's AI video interview analysis tool. The AI, which assesses personality traits from short videos, produced different scores when the same actress changed her appearance (glasses, headscarf, wig) or the video background and lighting were altered. The start-up Retorio acknowledged that the AI considers external image, similar to a human interviewer. Experts warned that such software could perpetuate stereotypes and unfairly affect job candidates.

AI system involved

Retorio AI

10 source articles · read the reporting →

ElevenLabs AI voice generation used in Russian influence operation targeting Europe

The article reports that a Russian influence campaign, dubbed "Operation Undercut," very likely used ElevenLabs' AI voice generation technology to create realistic voiceovers for fake news videos. The videos targeted European audiences to undermine support for Ukraine. Recorded Future's researchers used ElevenLabs' own AI Speech Classifier to detect the AI-generated audio. The campaign was attributed to the Russia-based Social Design Agency, which the U.S. government sanctioned. The overall impact on public opinion was minimal.

Company involved

Social Design Agency

AI system involved

ElevenLabs AI voice generation

7 source articles · read the reporting →

Kaedim used human artists instead of AI for 3D model generation

Kaedim, an AI startup that claimed to use machine learning to convert 2D illustrations into 3D models, actually used human artists to produce the models, according to sources. The company's founder was featured in Forbes 30 Under 30 for the technology. At one point, workers produced the 3D designs entirely without AI assistance. The revelation highlights how AI companies can overstate their capabilities.

Company involved

Kaedim

AI system involved

Kaedim

10 source articles · read the reporting →