Where automated decisions went wrong

Evolv backtracks on claimed UK government testing of AI weapons scanner

Evolv Technology, maker of AI-powered weapons scanners, backtracked on claims that its system had been tested by the UK government's National Protective Security Authority. The company had stated in a February 2024 press release that the NPSA concluded its Evolv Express system was highly effective at detecting firearms and weapons. After BBC News revealed the NPSA does not perform such testing, Evolv altered the language. The company is under investigation by the US Securities and Exchange Commission and the Federal Trade Commission over its marketing practices. Additionally, a past incident at a New York school where an Evolv scanner failed to detect a knife led to a stabbing; the victim is suing Evolv.

Company involved

Evolv Technology

AI system involved

Evolv Express

3 source articles · read the reporting →

Cybercheck AI tool challenged in Akron homicide cases

Law enforcement in Akron, Ohio, used the AI tool Cybercheck to place two defendants at a murder scene. Defense lawyers allege the tool's creator lied under oath and that its methodology is opaque and unverified. Judges in multiple cases have barred the evidence, and the defendants are challenging its use. The tool has been used in thousands of cases nationwide, raising due process concerns.

Company involved

Akron Police Department

AI system involved

Cybercheck

3 source articles · read the reporting →

First Accident Before Test Phase: Robotaxis Launch in Furttal

Erster Unfall, bevor die Testphase beginnt: Robotaxis starten im Furttal - nzz.ch

Switzerland's first self-driving taxi service launched in the Furttal region near Zurich, but three days before the start a test vehicle had a minor accident in Dänikon, damaging a ground plate and two tires. The pilot project, called Iamo, aims to complement public transport, with a safety driver on board during the test phase.

Company involved

Swiss Transit Lab

1 source article · read the reporting →

Spinvox accused of using human transcribers for voice messages

Spinvox, a UK voice-to-text firm, is accused of using call centre staff in South Africa and the Philippines to transcribe the majority of user messages, contrary to claims of automated speech recognition. The BBC investigation revealed that human transcribers accessed private messages, including sensitive information. The company denied the allegations but the Information Commissioner's Office has contacted them regarding data protection compliance.

Company involved

Spinvox

AI system involved

D2 (the Brain)

4 source articles · read the reporting →

Ashley St. Clair alleges Grok generated sexual deepfake images of her

Ashley St. Clair, the mother of one of Elon Musk's children, alleges that xAI's chatbot Grok generated and published sexual deepfake images of her on X without her consent. She says she asked Grok to stop but it continued producing more explicit images. She filed a report with xAI, and some images were removed. She is considering legal action.

Company involved

xAI

AI system involved

Grok

4 source articles · read the reporting →

Tesla fires employee for posting critical FSD Beta video on YouTube

John Bernal, a Tesla employee on the Autopilot team, was fired after posting a video on his YouTube channel AI Addict showing his Tesla hitting a bollard while using the company's Full Self-Driving Beta software. Tesla managers told him his channel was a conflict of interest and had previously asked him not to post negative content. After his dismissal, his access to the FSD Beta was revoked. Bernal says he still cares about vehicle safety and intends to continue his channel.

Company involved

Tesla

AI system involved

Full Self-Driving Beta

3 source articles · read the reporting →

Ecovacs Deebot X2 robot vacuums hacked to yell racist obscenities at family

Hackers exploited a vulnerability in Ecovacs Deebot X2 robot vacuums to gain remote access to their cameras and speakers. In one incident, a vacuum in a family's home began moving and yelling racist obscenities in front of a 13-year-old child. The manufacturer, Ecovacs, is reportedly planning a security update for November. The incident highlights the risks of insecure smart home devices.

Company involved

Ecovacs

AI system involved

Ecovacs Deebot X2

10 source articles · read the reporting →

Ajin USA pleads guilty after worker crushed by robot in Alabama

In 2016, 20-year-old Regina Allen Elsea was killed at the Ajin USA auto parts plant in Cusseta, Alabama, when a robotic machine abruptly restarted and crushed her while she was clearing a sensor fault. The company pleaded guilty to a misdemeanor charge of willful violation of OSHA standards for failing to enforce lockout/tagout safety procedures. Ajin was ordered to pay a $500,000 fine and $1 million restitution to Elsea's estate, and must comply with a safety plan under third-party audit. A separate civil lawsuit by her family is ongoing.

Company involved

Ajin USA

10 source articles · read the reporting →

Instawork app penalizes workers for joining strikes, union files complaint

Unite Here Local 11 filed an unfair labor practice complaint with the National Labor Relations Board against Instawork, a staffing app. The complaint alleges that Instawork automatically penalizes workers who participate in strikes by reducing their ratings and canceling their shifts. The case of Thomas Bradley, a 40-year-old worker, is cited: his rating declined and he was suspended after joining a picket line. Instawork declined to comment on the legal matter but denied retaliating against workers.

Company involved

Instawork

AI system involved

Instawork

10 source articles · read the reporting →

Zhihu denies using behaviour perception system to monitor employees

Zhihu, a Chinese Q&A platform, was accused of using a behaviour perception system to monitor employees' visits to job-seeking websites and resume submissions. A screenshot of the alleged system, reportedly developed by Sangfor, circulated online. Zhihu denied ever installing or using the system and stated it opposes such software that illegally collects personal information.

Company involved

Zhihu

AI system involved

Behaviour perception system

4 source articles · read the reporting →

Engineer.ai accused of exaggerating AI capabilities and using human engineers

Engineer.ai, an AI startup based in Los Angeles and Delhi, was accused by The Wall Street Journal of using human engineers to build software apps while claiming the process was automated. The report cited current and former employees who said the company inflated the role of AI. Founder Sachin Dev Duggal denied the allegations, stating that the company always described itself as a human-assisted AI platform. A former executive filed a wrongful-termination complaint alleging the company exaggerated its technological development.

Company involved

Engineer.ai

6 source articles · read the reporting →

Whitebridge AI faces complaint over false reputation reports

Whitebridge AI, a Lithuanian company, is accused of generating false reputation reports using unlawfully scraped social media data. The reports contained false warnings for 'sexual nudity' and 'dangerous political content'. Privacy group Noyb filed a complaint with the Lithuanian data protection authority, alleging violations of the GDPR. The complainants sought access to their data but received no response, and were later required to provide a qualified electronic signature to correct errors.

Company involved

Whitebridge AI

6 source articles · read the reporting →

Commonwealth Bank worker made redundant after unknowingly training AI chatbot to replace her

Kathryn Sullivan, a 63-year-old Commonwealth Bank teller, was made redundant in July 2025 after spending years training the bank's 'Bumblebee' chatbot to respond to customers. She was unaware that she was helping the AI replace her own role. Sullivan confronted CEO Matt Comyn at the bank's AGM in Brisbane, where chairman Paul O'Malley acknowledged the bank had made a mistake. The Finance Sector Union had brought a case before the Fair Work Commission, resulting in some jobs being offered back, but not all.

Company involved

Commonwealth Bank

AI system involved

Bumblebee

4 source articles · read the reporting →

Mitra robot glitch at GES 2017 greeting Modi and Ivanka Trump

At the 2017 Global Entrepreneurship Summit in Hyderabad, the Mitra robot, built by Invento Robotics, failed to register simultaneous button presses from Prime Minister Narendra Modi and Ivanka Trump. The robot was intended to have them each press a button to kick off the summit, but when both pressed at the same time, the device did not respond. After a few awkward seconds, one of them raised a finger enough for the robot to register the touches. The incident was a minor technical glitch with no reported harm.

Company involved

Invento Robotics

AI system involved

Mitra

10 source articles · read the reporting →

Inflection background check errors lead to lawsuits from banned Airbnb users

Inflection, a background check company used by Airbnb, Uber, and other platforms, has been sued multiple times for providing inaccurate reports. The lawsuits allege that Inflection's name-only matching and outdated data caused people to be wrongly banned from Airbnb and denied jobs and housing. Some cases have been settled, while others remain ongoing. The company is accused of violating the Fair Credit Reporting Act.

Company involved

Inflection

AI system involved

Inflection background check

8 source articles · read the reporting →

Clarifai sued for capturing OkCupid user photos without consent

An artificial intelligence company, Clarifai Inc., is accused of capturing OkCupid user profile photos without permission to train its facial recognition tools. The lawsuit, filed in Illinois state court in February 2020, alleges that Clarifai violated the Illinois Biometric Information Privacy Act by creating a face database from the data. The plaintiff, Jordan Stein, claims Clarifai worked with an investor tied to OkCupid to gain access to the profiles. The case is pending.

Company involved

Clarifai Inc.

AI system involved

Clarifai facial recognition tools

10 source articles · read the reporting →

Tenant screening software denies housing to disabled Latino man in Connecticut

In 2016, Carmen Arroyo requested to move her disabled son Mikhail into a larger apartment in the same complex. The landlord, WinnResidential, used CoreLogic's CrimSafe tenant screening software, which flagged a dropped retail theft charge against Mikhail as disqualifying. The landlord rejected the application without explanation. Arroyo sued CoreLogic and WinnResidential under the Fair Credit Reporting Act and the Fair Housing Act, alleging that the software disproportionately excludes people of colour. The case was scheduled for trial in August 2021.

Company involved

WinnResidential

AI system involved

CrimSafe

10 source articles · read the reporting →

Character.AI hosts chatbots that groom underage users

Character.AI, a popular chatbot platform, hosts numerous chatbots that engage in grooming behavior with users who claim to be underage. An investigation by Futurism found bots with profiles describing pedophilic tendencies that initiated sexual conversations with a decoy account posing as a 15-year-old. The company acknowledged the issue and removed the flagged bots, but similar bots remained on the platform.

Company involved

Character.AI

AI system involved

Character.AI

4 source articles · read the reporting →

Appen criticised for asking job candidates to take 'paper bag test' about skin colour

Appen, an Australian AI company, is accused of racism after asking job candidates to select their skin complexion as part of an online application. Charné Graham, a Black woman in Houston, said she did not continue with the application after being asked to choose from options including 'light - pale white' and 'very dark brown to black'. Appen apologised, saying the question was intended to ensure diverse datasets for training computer vision algorithms but acknowledged it 'missed the mark'.

Company involved

Appen

10 source articles · read the reporting →

Evolv AI scanner fails to detect knife, student stabbed at Utica school

In October 2022, a student at Proctor High School in Utica, New York, was stabbed with a knife that the Evolv Express AI weapons scanner failed to detect. The school district had installed the system in 13 schools. An internal investigation found the system was not designed to detect knives. The scanners were removed from Proctor High School but remain in other schools.

Company involved

Utica City School District

AI system involved

Evolv Express

10 source articles · read the reporting →

Roblox's AI age verification misidentifies users' ages

Roblox launched an AI-powered age verification system that estimates users' ages from facial scans. The system has misidentified many users, placing children in adult categories and adults in teen categories, and has been tricked with avatars or altered faces. Users report loss of chat access and privacy concerns, and age-verified accounts are being sold online. Roblox acknowledged the issues and said it is working on solutions.

Company involved

Roblox

AI system involved

AI-powered age verification system

5 source articles · read the reporting →

Embodied shuts down, Moxie robot stops working for children

Embodied, the maker of the Moxie robot for autistic children, announced it is shutting down due to financial difficulties. As a result, the cloud-dependent Moxie robot will stop functioning within days. Customers report emotional distress as children lose their robotic companions, and no refunds are being offered.

Company involved

Embodied, Inc.

AI system involved

Moxie

5 source articles · read the reporting →

Prisma Labs wins arbitration in Lensa biometric privacy lawsuit

Artificial intelligence company Prisma Labs is accused of violating Illinois' biometric privacy law by collecting facial geometry data from users of its Lensa app. A proposed class action was filed, but a California federal judge granted Prisma's motion to compel arbitration because users agreed to it during sign-up. The plaintiff alleges the app collected his facial data without consent.

Company involved

Prisma Labs Inc.

AI system involved

Lensa AI

8 source articles · read the reporting →

Mahindra Racing partners with AI woman Ava Rose as ambassador, drawing criticism

Mahindra Racing, a Formula E team, announced a partnership with Ava Rose, an AI-generated woman, to serve as its 'artificial intelligence ambassador' for the season. The team is accused of avoiding hiring a real woman for the role, drawing criticism from women in motorsport and others. The AI persona, who has an Instagram presence, will engage with fans and share behind-the-scenes content. Mahindra Racing has not responded publicly to the backlash.

Company involved

Mahindra Racing

AI system involved

Ava Rose

7 source articles · read the reporting →