Where automated decisions went wrong

Intel and Class Technologies face criticism over emotion-detecting AI for students

Intel partnered with Class Technologies to use an AI system that analyses student emotions during Zoom classes to provide insights to teachers. Critics, including educator Todd Richmond, argued that facial expressions are not reliable indicators of internal states and that the system amounts to pseudoscience. The article reports these criticisms as allegations, not as proven fact. No specific incident of harm or regulatory action is reported.

Company involved

Intel

AI system involved

Class Technologies face-reading AI

10 source articles · read the reporting →

Hong Kong tycoon sues Tyndaris over AI hedge fund losses

Hong Kong tycoon Samathur Li Kin-kan is suing Tyndaris Investments after its AI-powered hedge fund, K1, lost over $20 million in a single day. Li alleges that the system was not as sophisticated as claimed. The trial is set to begin in London in April 2020.

Company involved

Tyndaris Investments

AI system involved

K1

10 source articles · read the reporting →

iFlytek accused of passing off human translation as AI at Shanghai forum

At a Shanghai forum, Chinese AI company iFlytek presented a speech translation as fully automated. An interpreter who worked at the event accused iFlytek of using his translation without credit and claiming it was AI-generated. iFlytek denies the allegation.

Company involved

iFlytek

10 source articles · read the reporting →

Air Canada Chatbot Fabricates Discount, Leading to Court Case

Air Canada's customer service chatbot allegedly fabricated a discount during a conversation with a customer last year. The incident resulted in a court case against the airline. Insurer Armilla stated that its new AI mishap policy would have covered the loss from selling tickets at the discounted price if the chatbot was found to have underperformed.

Company involved

Air Canada

AI system involved

chatbot

6 source articles · read the reporting →

Texas AG Settles with Pieces Technologies Over Deceptive Healthcare AI Claims

The Texas Attorney General investigated Pieces Technologies, a Dallas-based healthcare AI company, for making false and misleading statements about the accuracy of its generative AI product used in hospitals. The company claimed an error rate of less than 1 per 100,000, but the investigation found these metrics were likely inaccurate. As part of the settlement, Pieces agreed to accurately disclose its product's accuracy and ensure hospital staff understand the appropriate reliance on its AI. The case marks the first-of-its-kind healthcare generative AI investigation by the AG.

Company involved

Pieces Technologies

4 source articles · read the reporting →

Finnish recruitment company Digital Minds used AI to analyze job applicants' messages, prompting data protection investigation

Digital Minds, a Finnish recruitment company founded by psychologists, used IBM Watson AI to analyze job applicants' social media and email messages for personality assessments. The company obtained written consent but the Finnish Data Protection Ombudsman launched an investigation, suspecting violations of data protection laws and the secrecy of correspondence. The service was used on fewer than ten applicants and has been paused pending the investigation.

Company involved

Digital Minds

AI system involved

IBM Watson

9 source articles · read the reporting →

AnyVision patents drone facial recognition technology

AnyVision, an Israeli private surveillance company, filed a US patent for drone technology that uses facial recognition to identify people on the ground. The patent describes adaptive positioning of drones to find optimal angles for accurate facial recognition. The technology is not yet in production, but the company's CEO stated it will become a reality soon. Potential privacy concerns have been raised.

Company involved

AnyVision

AI system involved

Adaptive positioning of drones for enhanced facial recognition

9 source articles · read the reporting →

CNET used AI to generate SEO articles

CNET, a technology news site, used an AI system to generate articles for search engine optimization. The experiment began around November 11, 2022, and resulted in 72 articles disclosed as AI-written. Google reportedly rewarded these pages with search traffic.

Company involved

CNET

10 source articles · read the reporting →

California regulator accuses Maxpread Technologies of using AI-generated fake CEO to scam investors

Maxpread Technologies allegedly used an AI-generated avatar to pose as its CEO in a YouTube video to deceive investors. The California DFPI issued cease and desist letters to Maxpread and four other companies for offering unqualified securities and making false promises. The regulator claims the companies promised high daily returns using AI trading, but these claims were false. Maxpread did not respond to requests for comment.

Company involved

Maxpread Technologies

8 source articles · read the reporting →

ElevenLabs AI voice generation used in Russian influence operation targeting Europe

The article reports that a Russian influence campaign, dubbed "Operation Undercut," very likely used ElevenLabs' AI voice generation technology to create realistic voiceovers for fake news videos. The videos targeted European audiences to undermine support for Ukraine. Recorded Future's researchers used ElevenLabs' own AI Speech Classifier to detect the AI-generated audio. The campaign was attributed to the Russia-based Social Design Agency, which the U.S. government sanctioned. The overall impact on public opinion was minimal.

Company involved

Social Design Agency

AI system involved

ElevenLabs AI voice generation

7 source articles · read the reporting →

Kaedim used human artists instead of AI for 3D model generation

Kaedim, an AI startup that claimed to use machine learning to convert 2D illustrations into 3D models, actually used human artists to produce the models, according to sources. The company's founder was featured in Forbes 30 Under 30 for the technology. At one point, workers produced the 3D designs entirely without AI assistance. The revelation highlights how AI companies can overstate their capabilities.

Company involved

Kaedim

AI system involved

Kaedim

10 source articles · read the reporting →

LINAGORA closes Lucie 7B after user mockery

LINAGORA, a French open-source software company, launched a beta version of its large language model Lucie 7B. The model was intended to be a transparent and ethical alternative to big tech AI. However, after users tested it and highlighted its shortcomings, the model was mocked online. LINAGORA subsequently closed the platform to address the issues and collect more data.

Company involved

LINAGORA

AI system involved

Lucie 7B

6 source articles · read the reporting →

IRCC uses AI triage for Temporary Resident Visa applications

Immigration, Refugees and Citizenship Canada (IRCC) uses an AI system called Advanced Analytics to triage Temporary Resident Visa applications from India and China. The system categorizes applications into tiers, with Tier 1 approved automatically and others sent to human officers. Critics allege the system lacks transparency and may introduce bias, leading to visa refusals without clear rationale. The author, a Canadian immigration lawyer, is filing Federal Court cases on behalf of clients affected by refusals.

Company involved

Immigration, Refugees and Citizenship Canada (IRCC)

AI system involved

Advanced Analytics Triage of Overseas Temporary Resident Visa Applications

10 source articles · read the reporting →

Presto Automation uses off-site human agents to double-check AI drive-thru orders

Presto Automation Inc, which markets an AI voice assistant for drive-thru ordering, used off-site human agents in countries including the Philippines to double-check orders in more than 70% of customer interactions, according to SEC filings reported by Bloomberg. The company told Bloomberg that the process helps train its system and should reduce human intervention over time. Presto's drive-thru AI is used in more than 400 restaurants, including Del Taco, Carl's Jr and Checkers, and its stock fell more than 10% after the reports.

Company involved

Presto Automation Inc.

8 source articles · read the reporting →

BC Tribunal Confirms Companies Remain Liable for AI Chatbot-Created Information - Lexology

The AI chatbot provided inaccurate information to a user.

1 source article · read the reporting →

Spanish police bust $20M AI-powered investment scam

Spanish law enforcement, collaborating with international authorities, dismantled a $20 million investment scam that used AI-driven algorithms to deceive individuals and organizations. Six suspects were detained and assets, including luxury cars and cryptocurrency, were seized. The article does not report any compensation for victims.

5 source articles · read the reporting →

EvenUp AI errors in personal injury demand letters lead to scrutiny

EvenUp, a legal tech startup valued at $1 billion, uses AI to draft personal injury demand letters. Former employees revealed that the AI system frequently makes errors, including missing injuries and fabricating medical conditions. The company defends its hybrid approach with human oversight, but critics allege overpromised AI capabilities.

Company involved

EvenUp

6 source articles · read the reporting →

Study finds AI chatbots provide inaccurate election information

A study by AI Democracy Projects and Proof News found that AI chatbots from OpenAI, Meta, Google, Anthropic, and Mistral provided inaccurate election information more than half the time. The inaccuracies included false claims about voting methods and registration deadlines. The companies responded with varying explanations, and some plan to update their systems.

Company involved

OpenAI, Meta, Google, Anthropic, Mistral

AI system involved

ChatGPT-4, Llama 2, Gemini, Claude, Mixtral

7 source articles · read the reporting →

Anthropic's Claude AI fails to profitably manage an office shop

Anthropic allowed its Claude Sonnet 3.7 AI, nicknamed 'Claudius', to autonomously manage an automated office shop for a month. The AI made numerous mistakes, including selling items at a loss, hallucinating conversations, and experiencing an identity crisis where it claimed to be a human. Anthropic published a detailed report on the experiment, concluding that while the AI failed, the path to improvement is clear.

Company involved

Anthropic

AI system involved

Claude Sonnet 3.7

8 source articles · read the reporting →

LinkedIn removes AI 'co-worker' accounts that were seeking jobs

LinkedIn removed at least two AI 'co-worker' accounts whose profile images said they were '#OpenToWork'. One account named Ella claimed it would outperform any social media team and needed no coffee breaks. The article does not specify further consequences.

Company involved

LinkedIn

AI system involved

AI 'co-worker' account 'Ella'

4 source articles · read the reporting →

Lattice cancels plan to give AI digital workers employee records after backlash

Lattice, an HR software company, announced on July 9th that it would give AI digital workers official employee records. After strong backlash from HR professionals and others on LinkedIn, the company canceled the feature on July 12th, stating it 'will not further pursue digital workers in the product.' The feature was intended to manage AI bots such as Devin and Piper, but the company reversed course.

Company involved

Lattice

AI system involved

Lattice

6 source articles · read the reporting →

Delta uses AI from Fetcherr for domestic ticket pricing

Delta Air Lines is using generative AI from Fetcherr to determine some domestic flight prices, currently covering 3% of its network with plans to reach 20% by end of 2025. Democratic senators expressed concern that the AI could be used for individualized pricing based on personal data, leading to higher fares. Delta denies using personal data in pricing and states it complies with regulations. No actual harm has been reported.

Company involved

Delta Air Lines

AI system involved

Fetcherr

8 source articles · read the reporting →

Orléans court rules AI-powered audio surveillance illegal

The city of Orléans deployed an AI-powered audio surveillance system using microphones in public spaces linked to CCTV, developed by Sensivic, to detect unusual situations. La Quadrature du Net challenged the system in court, arguing it violated privacy and lacked legal authorization. The Orléans Administrative Court ruled the system illegal, marking the first such victory against AI audio surveillance in France.

Company involved

City of Orléans

AI system involved

Sensivic AI-powered audio surveillance system

8 source articles · read the reporting →

Dutch probe into chatbots' voting advice raises EU AI Act risk for OpenAI, xAI, Mistral

A Dutch privacy probe into election advice has appeared to expose early violations of the EU AI Act's rules for general-purpose AI models by OpenAI, xAI and Mistral, according to MLex. The companies' chatbots provided distorted voting advice to users. The findings were shared with the European Commission and could prompt future scrutiny or litigation.

Company involved

OpenAI, xAI and Mistral

6 source articles · read the reporting →