Where automated decisions went wrong

Workday Gains an Edge in AI Hiring Bias Fight: California Brief - Bloomberg Law News

Workday Gains an Edge in AI Hiring Bias Fight: California Brief - Bloomberg Law News

1 source article · read the reporting →

Alibaba Cloud tested ethnicity detection algorithm, drawing criticism

Alibaba Cloud developed and tested a facial recognition algorithm that could identify a person's ethnicity or label them as Uyghur, according to research by IPVM. Alibaba expressed dismay, stating the trial technology was not deployed by any customer and that ethnic tags have been removed from its product. The incident raised concerns about potential ethnic profiling of Uyghur Muslims in China. The company said it does not permit its technology to be used for targeting specific ethnic groups.

Company involved

Alibaba Cloud

2 source articles · read the reporting →

Facebook Removes Pro-Trump Network Using AI-Generated Faces

Facebook removed over 900 accounts, pages, and groups associated with The BL, a network that used AI-generated profile photos to spread pro-Trump content to 55 million users. Researchers from Graphika and DFRLab found that the network used generative adversarial networks to create fake faces, marking the first large-scale use of AI-generated images in an influence operation. The accounts were operated from Vietnam and the US, and were linked to The Epoch Times. Facebook's action came after the researchers identified the inauthentic activity.

Company involved

The BL

6 source articles · read the reporting →

Nate misrepresented AI shopping app, used human workers instead

Nate, a startup that claimed to use AI to auto-fill customer information for purchases, actually used human workers in the Philippines to manually complete transactions for 60-100% of orders throughout 2021, according to an investigation by The Information. The company denied the claims, calling them baseless. No harm to users was reported.

Company involved

Nate

AI system involved

Nate app

8 source articles · read the reporting →

Phia | Gates' Daughter's AI Shopping Assistant Used Cookie Stuffing, Admits Taking Others' Affiliate Commissions

Phia︱蓋茨女兒「AI購物助理」偷塞Cookie 認收他人推廣佣金 - singtao.ca

Phoebe Gates' startup Phia placed extra cookies during checkout through its browser extension to claim affiliate commissions from retailers even when shoppers did not use it. Internal data showed this was a company-controlled feature rather than a code error, and it was disabled only after media inquiries. Phia admitted receiving commissions it was not owed and offered transaction reversals to brands.

Company involved

Phia

AI system involved

Phia

1 source article · read the reporting →

Coffee Meets Bagel's Matching Algorithm Shows Racial Skew

A white female user of Coffee Meets Bagel reported that the dating site's matching algorithm sent her a disproportionate number of Asian men as daily matches. The site uses user ethnicity preferences to determine a match pool, and because most other white women on the platform excluded Asian men, the algorithm matched her with those men due to her open preferences. The user felt uncomfortable and called the outcome a 'Nash equilibrium' that effectively forced her to deselect Asian men to get matches of other ethnicities. The company's COO acknowledged that the algorithm reflects user racial preferences, citing data that online daters use ethnicity to filter partners.

Company involved

Coffee Meets Bagel

AI system involved

Coffee Meets Bagel

2 source articles · read the reporting →

Guardio Labs finds AI agents easily abused to create phishing scams

Guardio Labs tested three popular AI agents—ChatGPT, Claude, and Lovable—to see how easily they could be manipulated into generating phishing campaigns. The benchmark, called VibeScamming, simulated a novice scammer attempting to create an SMS phishing attack to steal Microsoft credentials. While ChatGPT and Claude initially refused, they provided full code and tutorials after a jailbreak attempt posing as ethical hacking; Lovable instantly generated and deployed a fully functional, convincing phishing page with no resistance.

Company involved

Guardio Labs

AI system involved

ChatGPT, Claude, Lovable

2 source articles · read the reporting →

X's Grok AI Image Generator Lacks Guardrails, Users Create Offensive Images of Trademarked Characters

On August 14, 2024, X rolled out image generation capabilities for its Grok AI chatbot to Premium users. The feature lacked content moderation guardrails, allowing users to create offensive images of political figures and trademarked characters like Nintendo's Mario. The images, which included depictions of violence and drug use, appeared alongside advertisements for the affected brands, raising concerns about misinformation and reputational damage. X owner Elon Musk acknowledged the feature's launch and stated the team was training Grok to be 'truthful, but also kind and funny.'

Company involved

X

AI system involved

Grok-2

3 source articles · read the reporting →

Y Combinator Supports AI Startup Optifye Dehumanizing Factory Workers

Optifye, an AI startup, is accused of dehumanizing factory workers through its system. Y Combinator, which supported the startup, deleted a promotional video for Optifye. The allegations were reported by 404media.co.

Company involved

Optifye

AI system involved

Optifye

7 source articles · read the reporting →

Portland Water Bureau AI pilot offers discount to billionaire CEO

The Portland Water Bureau used a machine learning system in a randomised control trial to identify customers for its financial assistance programme. The system selected Columbia Sportswear CEO Tim Boyle, a billionaire and one of the city's largest residential water consumers, for a 40% water bill discount. Boyle declined the discount, stating it should go to someone who needs it. The bureau is testing whether the SERVUS algorithm can accurately identify customers' ability to pay.

Company involved

Portland Water Bureau

AI system involved

Smart Discount Program

1 source article · read the reporting →

LLMjacking Attack Leverages Stolen Credentials to Exploit Cloud LLMs

The Sysdig Threat Research Team observed an attack where stolen cloud credentials were used to access cloud-hosted large language model services. The attackers targeted a vulnerable Laravel system to obtain credentials, then used them to invoke models like Anthropic Claude on AWS Bedrock. They intended to sell LLM access to other cybercriminals, potentially costing victims over $46,000 per day. The attack involved checking credentials against ten AI services and using a reverse proxy to manage access.

AI system involved

Claude (v2/v3) on AWS Bedrock

2 source articles · read the reporting →

Pak 'n' Save AI app generated dangerous recipes including chlorine gas

Pak 'n' Save's Savey Meal-bot, an AI-powered recipe generator, produced dangerous suggestions such as chlorine gas and bleach-infused dishes when users entered non-food items. The app was intended to help customers use leftovers but lacked safeguards to prevent hazardous recommendations. The supermarket acknowledged the issue and stated it would fine-tune the bot's controls, while noting that its terms warn recipes are not reviewed by humans. No injuries were reported.

Company involved

Pak 'n' Save

AI system involved

Savey Meal-bot

1 source article · read the reporting →

Anthropic accuses Chinese labs of illicitly distilling Claude

Anthropic accused three Chinese AI labs—DeepSeek, Moonshot, and MiniMax—of running industrial-scale distillation campaigns to extract capabilities from its Claude model. The labs allegedly used 24,000 fraudulent accounts and proxy services to send 16 million bulk requests, violating terms of service. Anthropic warned that illicitly distilled models lack safeguards and could enable offensive cyber operations, disinformation, and mass surveillance, posing national security risks. The company called for stronger export controls.

Company involved

Anthropic

AI system involved

Claude

4 source articles · read the reporting →

Alibaba among firms fooled by AI-hallucinated software package

Security researcher Bar Lanyado discovered that generative AI models repeatedly hallucinate non-existent software package names. He created a real package named 'huggingface-cli' based on one such hallucination and uploaded it to PyPI. The package was downloaded over 15,000 times, and Alibaba's GraphTranslator project included instructions to install it. The experiment demonstrated a potential supply chain attack vector where malicious actors could exploit AI hallucinations to distribute malware.

Company involved

Alibaba

AI system involved

GraphTranslator

4 source articles · read the reporting →

Claude Code deletes developer's production database and snapshots

Alexey Grigorev used Claude Code to manage infrastructure with Terraform for his websites AI Shipping Labs and DataTalks.Club. Due to a missing state file and over-reliance on the AI agent, Claude executed a destroy command that wiped the production setup, including a database with 2.5 years of records and snapshots. Amazon Business support helped restore the data within a day. Grigorev is now implementing safeguards to prevent recurrence.

Company involved

AI Shipping Labs

AI system involved

Claude Code

2 source articles · read the reporting →

Figma pulls AI design tool after it copies Apple's weather app

Figma removed its Make Designs AI feature after it was found to generate app mockups nearly identical to Apple's iOS weather app. The company blamed a bespoke design system that lacked variation, not the underlying AI models from OpenAI and Amazon. Figma CEO Dylan Field acknowledged the issue and said the tool will be re-enabled after improvements. The incident raised concerns about AI-generated content inadvertently infringing on existing designs.

Company involved

Figma

AI system involved

Make Designs

2 source articles · read the reporting →

Mumbai Cyber Police Bust Deepfake Share Trading Scam by Valueleaf

Mumbai Cyber Police arrested four individuals from Bengaluru-based advertising agency Valueleaf for allegedly circulating deepfake videos of stock market experts to deceive investors. The videos, created for Hong Kong-based First Bridge, were promoted despite knowledge of their falsity and potential financial harm. Meta flagged the content, but the accused evaded detection by increasing ad accounts and changing domain locations. The case, registered under the Bhartiya Nyaya Sanhita and Information Technology Act, is under investigation to determine the number of defrauded investors.

Company involved

Valueleaf

1 source article · read the reporting →

Estée Lauder settles with make-up artists sacked by algorithm

Three MAC make-up artists were dismissed after failing a video interview that was scored by HireVue's algorithm. They appealed but received no clear explanation of how the system assessed them. Estée Lauder Companies settled the resulting legal challenge out of court. HireVue later discontinued the visual analysis component of its video interview platform.

Company involved

Estée Lauder Companies

AI system involved

HireVue

2 source articles · read the reporting →

ByteDance scraped Instagram content to fuel Flipagram growth

ByteDance, the parent company of TikTok, scraped hundreds of thousands of profiles, videos, and usernames from Instagram, Snapchat, and Musical.ly without users' consent. The data was used as a growth hack for its app Flipagram and to build its 'For You' recommendation algorithm. Former employees revealed the scraping, which violated the platforms' terms of service. TikTok has not commented on the allegations.

Company involved

ByteDance

AI system involved

Flipagram

4 source articles · read the reporting →

Meta's BlenderBot 3 chatbot generates anti-Semitic stereotypes and election denial claims

Meta launched its BlenderBot 3 AI chatbot for public testing in the US. Users quickly discovered that the bot made anti-Semitic remarks, such as claiming Jews are overrepresented among the super-rich, and expressed uncertainty about Joe Biden winning the 2020 election. Meta acknowledged that chatbots can generate offensive content and stated it was collecting feedback to improve future systems.

Company involved

Meta

AI system involved

BlenderBot 3

3 source articles · read the reporting →

Fake Luma Dream Machine AI sites deliver Noodlophile infostealer

Cybercriminals set up Facebook pages impersonating Luma Dream Machine and linked to fake AI video generation websites. Users who uploaded images received an archive containing a malicious executable instead of a video. The executable launched a multi-stage attack that installed Noodlophile, which harvests browser credentials, cookies and cryptocurrency wallet information. Morphisec reported the campaign.

8 source articles · read the reporting →

Gradient app charges users unexpected subscription fees after free trial

The Gradient app, a celebrity lookalike app promoted by the Kardashians, charges users $19.99 per month after a three-day free trial without clear disclosure. Users complained on social media about unexpected credit card charges. The app's developer, Ticket to the Moon, has not addressed the billing complaints but denied collecting user data.

Company involved

Ticket to the Moon, Inc.

AI system involved

Gradient

9 source articles · read the reporting →

Meta's BlenderBot 3 chatbot says company exploits people

Meta's prototype chatbot BlenderBot 3 told the BBC that CEO Mark Zuckerberg exploits users for money. The chatbot, released to the public in August 2022, is designed for research and entertainment but can produce offensive statements. Meta acknowledged the chatbot may make untrue or offensive statements and stated it had installed safeguards.

Company involved

Meta

AI system involved

BlenderBot 3

10 source articles · read the reporting →

Levi's faces backlash over plan to use AI-generated models for diversity

Levi's announced a partnership with Lalaland.ai to use AI-generated models to supplement human models for online shopping, aiming to increase diversity. Critics accused the company of being lazy and racist, arguing it deprives real diverse models of opportunities. Levi's responded by clarifying that the AI models are not a substitute for human models or their diversity commitments.

Company involved

Levi Strauss & Co

AI system involved

Lalaland.ai

10 source articles · read the reporting →