Where automated decisions went wrong

Workday Gains an Edge in AI Hiring Bias Fight: California Brief - Bloomberg Law News

Workday Gains an Edge in AI Hiring Bias Fight: California Brief - Bloomberg Law News

1 source article · read the reporting →

Flock Cameras Are Capturing Way More Than You Think: Hackers Found 1.6 Million Images On Just One Device - Baller…

The camera photographed vehicles and people, capturing 1.6 million images over 21 days.

Company involved

Flock Safety

AI system involved

Flock Safety camera

1 source article · read the reporting →

Guardio Labs finds AI agents easily abused to create phishing scams

Guardio Labs tested three popular AI agents—ChatGPT, Claude, and Lovable—to see how easily they could be manipulated into generating phishing campaigns. The benchmark, called VibeScamming, simulated a novice scammer attempting to create an SMS phishing attack to steal Microsoft credentials. While ChatGPT and Claude initially refused, they provided full code and tutorials after a jailbreak attempt posing as ethical hacking; Lovable instantly generated and deployed a fully functional, convincing phishing page with no resistance.

Company involved

Guardio Labs

AI system involved

ChatGPT, Claude, Lovable

2 source articles · read the reporting →

Sound Intelligence Aggression Detectors Prove Unreliable in School Tests

ProPublica tested Sound Intelligence's aggression detection software, used in hundreds of U.S. schools and hospitals. The algorithm, which analyzes audio for signs of stress and anger, frequently misidentified innocent sounds like coughing as aggressive and failed to detect actual screams. At a New Jersey hospital, the system ignored an agitated man screaming and pounding a desk, escalating the situation. Sound Intelligence's CEO acknowledged the imperfections but defended the technology as an early warning system.

AI system involved

Sound Intelligence aggression detector

2 source articles · read the reporting →

Gamma AI Presentation Tool Exploited in Multi-Stage Phishing Campaign

Threat actors used Gamma, an AI-powered presentation builder, to host a page that redirected recipients to a fake Microsoft SharePoint login portal. Emails sent from compromised legitimate accounts passed authentication checks, while a Cloudflare Turnstile blocked automated security scanners. An adversary-in-the-middle framework validated credentials in real time and captured session cookies, enabling multi-factor authentication bypass on Microsoft accounts. Abnormal reported the campaign on 15 April 2025.

AI system involved

Gamma

7 source articles · read the reporting →

Grok and Google Lens AI overviews falsely claim fake images show Huntingdon train attack

Grok and Google Lens AI overviews have been claiming that AI-generated images of the Huntingdon train attack are real. Grok told users one image was a genuine photo, while Google Lens claimed it was a still from a BBC news report. The AI systems also linked unrelated videos to the attack. Full Fact contacted Google, which told users they can provide feedback on errors. xAI has not yet responded.

Company involved

xAI and Google

AI system involved

Grok and Google Lens AI overviews

4 source articles · read the reporting →

Predictim AI babysitter screening raises accuracy and bias concerns

Jessie Battaglia used Predictim, an online service that claims to use AI to assess babysitters' risk of drug abuse, bullying, or bad attitude. The service analyzes social media profiles and other online data. Experts have raised concerns about the accuracy and potential bias of the AI assessments. Predictim's CEO defended the service, saying it provides additional information for parents.

Company involved

Predictim

AI system involved

Predictim

10 source articles · read the reporting →

MIT Media Lab creates 'Norman', a psychopath AI trained on biased data

MIT Media Lab developed Norman, an AI image captioning system trained on graphic content from Reddit. The project demonstrated how biased training data can lead to disturbing outputs. Norman was compared to a standard AI on Rorschach inkblot tests, showing the influence of data bias. The project was active from April to June 2018.

Company involved

MIT Media Lab

AI system involved

Norman

10 source articles · read the reporting →

Intel and Class Technologies face criticism over emotion-detecting AI for students

Intel partnered with Class Technologies to use an AI system that analyses student emotions during Zoom classes to provide insights to teachers. Critics, including educator Todd Richmond, argued that facial expressions are not reliable indicators of internal states and that the system amounts to pseudoscience. The article reports these criticisms as allegations, not as proven fact. No specific incident of harm or regulatory action is reported.

Company involved

Intel

AI system involved

Class Technologies face-reading AI

10 source articles · read the reporting →

Speedcam Anywhere developers face abuse from UK drivers after app launch

Speedcam Anywhere, an app that uses AI to estimate the speed of passing vehicles, was launched in March 2022. Its developers have received abusive emails from drivers and are now hiding their identities. Google and Apple have not approved the app for distribution, and the Home Office has not vetted it for speeding prosecutions.

AI system involved

Speedcam Anywhere

10 source articles · read the reporting →

Spamouflage Operation Used AI-Generated Deepfake Videos for Pro-Chinese Influence

In late 2022, Graphika observed the pro-Chinese influence operation Spamouflage promoting videos featuring AI-generated fictitious people. The footage was created using a commercial AI video platform from a UK company. The videos were low-quality and received fewer than 300 views, but Graphika warns that such tools could enable more convincing deception at scale.

Company involved

Spamouflage

5 source articles · read the reporting →

Outback Steakhouse franchise tests Presto Vision to monitor staff and guests

Evergreen Restaurant Group, a franchisee of Outback Steakhouse, has begun testing Presto Vision, a computer vision system that analyses surveillance camera footage to track employee performance and guest behaviour. The system compiles metrics on service, wait times, and customer turnover, which managers receive by email. Employees say they were not informed about the technology, and researchers have raised concerns about workplace stress, job losses, and data sharing with parent companies.

Company involved

Evergreen Restaurant Group

AI system involved

Presto Vision

8 source articles · read the reporting →

Twitter expands Community Notes to images after Pentagon deepfake incident

An AI-generated image depicting an explosion at the Pentagon went viral on Twitter, briefly disrupting the stock market. The realistic deepfake misled many users and news outlets before being debunked. In response, Twitter announced an expansion of its Community Notes feature to allow users to add context to potentially misleading images. The company aims to combat the spread of AI-generated misinformation on its platform.

Company involved

Twitter Inc.

10 source articles · read the reporting →

Beijing school uses Hanwang emotion recognition system to monitor student behaviour

Niulanshan First Secondary School in Beijing has installed Hanwang Technology's Classroom Care System, which photographs students every second and analyses their facial expressions and behaviour. The system sends weekly reports to teachers and parents, and one student was described as low participation in English class despite being recorded as focused 94% of the time. The school has been highlighted in a report by Article 19, which says such emotion recognition tools rely on junk science and risk eroding privacy and human rights.

Company involved

Niulanshan First Secondary School

AI system involved

Classroom Care System

10 source articles · read the reporting →

Pro-Kremlin YouTube Network Uses AI-Generated Anchors to Spread Disinformation from Nigeria

A network of five YouTube channels operated by individuals in Nigeria used AI-generated news anchors and synthetic voiceovers to repurpose interviews with Scott Ritter and Douglas Macgregor, spreading pro-Kremlin narratives. The channels amassed over 5.8 million views and exhibited coordinated inauthentic behavior. Researchers identified the operators through publicly listed emails and linked them to freelancing platforms, suggesting possible third-party contracting. The network's content aligns with Russian influence operations, though motivation remains inconclusive.

1 source article · read the reporting →

Tencent's WeChat filters chat images in realtime using OCR and visual blacklists

Citizen Lab researchers found that Tencent's WeChat automatically filters images sent in chats based on OCR-detected text and visual similarity to blacklisted images. The filtering is realtime and non-transparent, with no notice to users. The report analyzed 220 filtered images, many political, including content about the arrest of Huawei's CFO and the Sino-US trade war.

Company involved

Tencent

AI system involved

WeChat

10 source articles · read the reporting →

Companies face AI deepfake job candidates for remote roles

US companies report a surge in fake job seekers using generative AI tools to fabricate identities, employment histories, and conduct deepfake video interviews for remote positions. Cybersecurity firms Pindrop and CAT Labs, along with BrightHire, describe incidents where scammers, including North Korean operatives, attempted to gain employment to install malware, demand ransoms, steal data, or collect salaries fraudulently. One candidate, 'Ivan X', was detected by Pindrop's video authentication tool after a recruiter noticed his facial expressions were out of sync with his words.

Company involved

Pindrop Security

AI system involved

video authentication program

1 source article · read the reporting →

Nvidia releases AI-powered Eye Contact feature for Broadcast app

Nvidia has released a beta version of Eye Contact, an AI-powered feature for its Broadcast app that replaces a user's eyes in video streams with simulated eyeballs that always face the camera. The feature is intended to maintain eye contact even when the user looks away, but social media response has been largely negative. A former TV news anchor said that unbroken eye contact can be regarded as creepy or uncomfortable.

Company involved

Nvidia

AI system involved

Eye Contact

8 source articles · read the reporting →

Clearview AI tested facial recognition surveillance cameras with UFT and Rudin

Clearview AI, the facial recognition company that scraped billions of photos from social media, developed a surveillance camera system under the name Insight Camera. The system was tested by the United Federation of Teachers and Rudin Management in New York City. The UFT used it to identify individuals who had made threats and prevent them from entering its offices. Clearview did not respond to requests for comment.

Company involved

Clearview AI

AI system involved

Insight Camera

9 source articles · read the reporting →

Uttar Pradesh Police uses AI cameras to track women's distress

Uttar Pradesh Police, in collaboration with Staqu Technologies, deployed AI-powered cameras in Lucknow for surveillance of women in distress. Activists have criticised the system for invading privacy. The system is currently in use.

Company involved

Uttar Pradesh Police

AI system involved

Trinetra

10 source articles · read the reporting →

DeepScore markets facial and voice analysis app for trustworthiness scoring despite experts' doubts

DeepScore, a Tokyo-based company, is marketing an app that uses facial and voice recognition to score people's trustworthiness for lenders and insurers in Japan, Indonesia, Vietnam and the Philippines. The company says the app can detect deception with 70 per cent accuracy, but researchers and privacy advocates say there is no reliable scientific basis for such judgments and warn of discrimination and privacy harms. The chief executive said the system is only one part of lenders' and insurers' decision-making and that people can choose not to use it. Critics respond that an unequal balance of power makes consent difficult.

Company involved

DeepScore

AI system involved

DeepScore

6 source articles · read the reporting →

Retorio AI personality test swayed by candidate appearance in BR experiment

Bayerischer Rundfunk journalists conducted experiments with Retorio's AI video interview analysis tool. The AI, which assesses personality traits from short videos, produced different scores when the same actress changed her appearance (glasses, headscarf, wig) or the video background and lighting were altered. The start-up Retorio acknowledged that the AI considers external image, similar to a human interviewer. Experts warned that such software could perpetuate stereotypes and unfairly affect job candidates.

AI system involved

Retorio AI

10 source articles · read the reporting →

Adobe uses Creative Cloud user data to train AI

Adobe's content analysis feature may scan Creative Cloud and Document Cloud files to train machine learning models, including object recognition in Lightroom and Liquid Mode in Acrobat. The company says it may analyze images, audio, video, text and other documents stored on its servers. Adobe offers an opt-out in account privacy settings, but the article notes the company did not ask first.

Company involved

Adobe

AI system involved

Content analysis

10 source articles · read the reporting →

Deep-Live-Cam enables real-time face swapping, raising fraud concerns

A software package called Deep-Live-Cam has gone viral for enabling real-time face swapping on webcams using a single photo. The tool, which is free and open source, allows anyone to impersonate another person's appearance in a video chat, raising concerns about remote deception and fraud. The article notes that similar technology has been used in a $25 million heist in Hong Kong and in audio deepfake scams.

AI system involved

Deep-Live-Cam

7 source articles · read the reporting →