Where automated decisions went wrong

What a $150,000 scam reveals about AI and elder fraud - Rolling Out

AI-generated deepfakes and voice cloning convinced a 63-year-old woman to send $150,000 to a scammer posing as a romantic partner

1 source article · read the reporting →

AI Scammers Clone Exante Broker, Use JPMorgan Account to Defraud US Victim

Scammers used generative AI to create a fake clone of the brokerage firm Exante, including a replicated trading platform and AI-generated passports. They opened a real JPMorgan Chase bank account using a US address and tricked at least one US victim into transferring funds. Exante, which does not serve US clients, discovered the scam when the victim was registered on its real platform and reported the incident to the FBI, SEC, CFTC, and other authorities. The scammers remain unidentified, and the victim's funds have not been recovered.

2 source articles · read the reporting →

AI Hiring Platform Faces FCRA Class Action Over Data Use | Kistler et al. v. Eightfold AI Inc.

The AI platform screened job applicants, affecting their hiring prospects.

Company involved

Eightfold AI

AI system involved

Eightfold AI

1 source article · read the reporting →

Data Breach Exposes Over 10 Million Conversations from Middle Eastern AI Call Center Platform

Resecurity discovered a dark web posting on 8 October 2024 offering data stolen from a major AI-powered cloud call center platform in the Middle East. The threat actor gained unauthorized access to the management dashboard, compromising over 10,210,800 conversations between consumers, operators, and AI chatbots. The exposed data included personally identifiable information and national ID documents, creating risks of fraud and identity theft. Resecurity alerted the affected organization and collaborated with law enforcement to mitigate the incident.

3 source articles · read the reporting →

Istanbul gang used AI to defraud European citizens from call center

A gang in Maslak, Istanbul, set up a call centre using AI-supported software to impersonate police, prosecutors, soldiers, and bank officials. They defrauded many Czech and European citizens, obtaining high amounts of money, which they laundered through bank accounts and crypto platforms. Turkish authorities, in cooperation with Czech judicial authorities, conducted simultaneous raids on three addresses, seizing digital materials and detaining 80 foreign suspects.

1 source article · read the reporting →

AI-Generated Fake Investment Websites Target Australian Fraud Analyst

A fraud analyst, Leon, was targeted by scammers who used AI to create convincing fake websites for non-existent investment firms. The scammers posed as employees of Assent Advisory and directed him to elaborate sites for APPC Capital Singapore and Thackeray Mines and Minerals Inc., complete with AI-generated images and fake details. Leon recognised the scam, but the sophistication of the AI-generated content made him second-guess his own judgment. The incident highlights how AI is enabling more convincing investment scams.

1 source article · read the reporting →

TRT-RS's Galileu AI Detects Prompt Injection Attempt in Legal Petition

The Galileu AI system, developed by the Tribunal Regional do Trabalho da 4ª Região (TRT-RS) and nationalised by the Conselho Superior da Justiça do Trabalho (CSJT), detected a prompt injection attempt in a petition filed at the 3rd Labour Court of Parauapebas, Pará. The system alerted the magistrate, who reviewed the content and made a decision based on human verification, in line with judicial AI supervision requirements. The court reported that the system prevented the malicious content from being processed and highlighted the importance of institutional AI tools with security measures.

Company involved

Tribunal Regional do Trabalho da 4ª Região

AI system involved

Galileu

1 source article · read the reporting →

PredPol crime predictions disproportionately targeted Black and Latino neighborhoods

Between 2018 and 2021, PredPol software sent millions of crime predictions to U.S. police departments. An analysis of unsecured data found that predictions overwhelmingly targeted Black, Latino, and low-income neighborhoods, while whiter, wealthier areas were largely avoided. The company claimed the software was bias-free, but the data showed persistent racial and economic disparities. Many police departments have since stopped using the software.

AI system involved

PredPol

10 source articles · read the reporting →

Scammers use AI-generated identities to steal $5.6 million in FTX debt claims fraud

In June 2024, a scam group posing as FTX debt claimants allegedly used AI-generated identities and manipulated facial appearances to defraud two companies of more than $5.6 million. The perpetrators accessed FTX customer data through public bankruptcy filings or a 2023 data breach at Kroll. Blockchain analysis traced the stolen funds through Binance, CoinEx, and Gate.io. The incident remains unresolved.

6 source articles · read the reporting →

Rotterdam's Welfare Fraud Algorithm Discriminated by Gender and Ethnicity

The city of Rotterdam deployed a machine learning algorithm built by Accenture to flag welfare recipients for fraud investigation. The system used personal data including gender, language, and subjective caseworker notes to generate risk scores, leading to investigations that disproportionately targeted women and migrants. An external review found the algorithm discriminatory and inaccurate, prompting the city to suspend its use in 2021. The system's opacity made it nearly impossible for those flagged to challenge the decisions.

Company involved

City of Rotterdam

6 source articles · read the reporting →

FTC Orders Evolv to Stop Overstating Effectiveness of AI Gun Detection Tech

Evolv, a company selling AI-powered weapons detection systems, has been ordered by the FTC to stop making misleading claims about its Evolv Express product. The FTC alleged that the technology is essentially a metal detector with unproven AI, and that Evolv falsely claimed it could detect guns while ignoring harmless items. A pilot in New York City subways resulted in over 100 false positives and no guns detected. The settlement requires Evolv to allow educational customers to cancel their contracts and prohibits further misrepresentations.

Company involved

Evolv

AI system involved

Evolv Express

2 source articles · read the reporting →

SEC Charges Delphia and Global Predictions for False AI Claims

The SEC charged Delphia (USA) Inc. and Global Predictions Inc. for making false and misleading statements about their use of artificial intelligence. Delphia claimed from 2019 to 2023 that it used AI and machine learning to predict investments, while Global Predictions falsely claimed in 2023 to be the 'first regulated AI financial advisor'. Both firms settled the charges without admitting or denying the findings, agreeing to pay a total of $400,000 in civil penalties and to cease and desist from further violations.

Company involved

Delphia (USA) Inc. and Global Predictions Inc.

1 source article · read the reporting →

Guardio Labs finds AI agents easily abused to create phishing scams

Guardio Labs tested three popular AI agents—ChatGPT, Claude, and Lovable—to see how easily they could be manipulated into generating phishing campaigns. The benchmark, called VibeScamming, simulated a novice scammer attempting to create an SMS phishing attack to steal Microsoft credentials. While ChatGPT and Claude initially refused, they provided full code and tutorials after a jailbreak attempt posing as ethical hacking; Lovable instantly generated and deployed a fully functional, convincing phishing page with no resistance.

Company involved

Guardio Labs

AI system involved

ChatGPT, Claude, Lovable

2 source articles · read the reporting →

Schufa's Black-Box Scoring Unfairly Penalises Consumers with Positive Credit Data

An investigation by SPIEGEL and BR Data reveals that Schufa's credit scoring algorithm often assigns poor risk scores to consumers with only positive credit information. One consumer, Sven Drewert, was denied a credit card limit increase despite having no negative entries. The algorithm uses limited data, and its secret formula can lead to arbitrary categorisations, affecting access to loans, phone contracts, and housing. The system's opacity and potential biases raise concerns about fairness and accountability.

Company involved

Schufa Holding AG

AI system involved

Schufa Score

2 source articles · read the reporting →

Gamma AI Presentation Tool Exploited in Multi-Stage Phishing Campaign

Threat actors used Gamma, an AI-powered presentation builder, to host a page that redirected recipients to a fake Microsoft SharePoint login portal. Emails sent from compromised legitimate accounts passed authentication checks, while a Cloudflare Turnstile blocked automated security scanners. An adversary-in-the-middle framework validated credentials in real time and captured session cookies, enabling multi-factor authentication bypass on Microsoft accounts. Abnormal reported the campaign on 15 April 2025.

AI system involved

Gamma

7 source articles · read the reporting →

Durham Police uses Experian Mosaic data in HART AI risk tool

Durham Constabulary developed the Harm Assessment Risk Tool (HART), a machine learning algorithm that assesses the recidivism risk of offenders. The tool uses 34 data categories including criminal history, age, gender and two types of postcode, one sourced from Experian's Mosaic marketing segmentation system. Big Brother Watch alleges that using such commercial consumer behaviour data to inform custody decisions risks prejudice and disproportionate targeting of deprived neighbourhoods. The force has stated it is refreshing the model with an aim to remove one of the postcode predictors.

Company involved

Durham Constabulary

AI system involved

Harm Assessment Risk Tool (HART)

9 source articles · read the reporting →

Predictim AI babysitter screening raises accuracy and bias concerns

Jessie Battaglia used Predictim, an online service that claims to use AI to assess babysitters' risk of drug abuse, bullying, or bad attitude. The service analyzes social media profiles and other online data. Experts have raised concerns about the accuracy and potential bias of the AI assessments. Predictim's CEO defended the service, saying it provides additional information for parents.

Company involved

Predictim

AI system involved

Predictim

10 source articles · read the reporting →

FinTech and traditional lenders discriminate against minority borrowers in mortgage pricing

A study of mortgage lending from 2012-2018 found that Latinx and African-American borrowers were charged higher interest rates than white borrowers with similar credit risk. FinTech algorithms reduced the disparity by 40% but did not eliminate it. The discrimination costs minority borrowers an estimated $765 million per year in extra interest.

10 source articles · read the reporting →

Montenegro police warn of rising AI identity abuse in online fraud

Montenegrin police say AI is fuelling a rise in identity abuse by online fraudsters, using deepfake videos of public figures to promote bogus remedies and investment schemes. Surgeon and MP Vladimir Dobricanin found a manipulated video of himself apparently endorsing eyedrops and other products, while Bosnian fact-checkers reported similar misuse of statements to lure people into cryptocurrency scams. Victims reported 42 online scams in 2025 with financial losses of around 300,000 euros, but authorities say investigations are complex and often lengthy due to the use of servers abroad and encryption. Experts called for stronger technical capacities, international cooperation, and faster platform responses to tackle the problem.

Company involved

Limited Charm

AI system involved

Deepfake social media advertisements

1 source article · read the reporting →

UIUC researchers use OpenAI API to automate phone scams for under a dollar

Researchers at the University of Illinois Urbana-Champaign used OpenAI's Realtime API to create AI agents that can autonomously execute phone scams. The agents successfully performed bank account transfers and credential theft at an average cost of $0.75 per scam. OpenAI acknowledged the experiment and pointed to its safety policies.

Company involved

University of Illinois Urbana-Champaign

AI system involved

GPT-4o Realtime API

6 source articles · read the reporting →

Austrian court lifts ban on AMS job-chance prediction algorithm

The Austrian Federal Administrative Court overturned a ban by the Data Protection Authority on the AMS algorithm, which predicts job chances of unemployed people. The system, trained on historical data including age, gender, and nationality, categorises individuals as high, medium or low chance of re-employment. Critics argue it discriminates against women and mothers, and that the data is now outdated due to the COVID-19 pandemic. The court ruled that the algorithm is lawful as long as a human advisor makes the final decision on training support.

Company involved

Arbeitsmarktservice (AMS)

AI system involved

AMS-Algorithmus

5 source articles · read the reporting →

Dutch police predictive policing project in Roermond faces discrimination allegations

Amnesty International's report alleges that the Dutch police's Sensing predictive policing project in Roermond uses algorithms that result in automated discrimination and mass surveillance. The system assesses crime risk for individuals and locations, potentially targeting marginalized communities. The report calls for accountability and human rights safeguards.

Company involved

Dutch police

AI system involved

Sensing project

10 source articles · read the reporting →

UK councils use Covid OneView AI to harvest personal data for risk scoring

UK local authorities are using a system called Covid OneView, developed by data analytics firm Xantura, to harvest millions of personal details from council records. The system uses predictive analytics and AI to assign risk scores to households and individuals, aiming to identify those vulnerable to Covid or likely to break lockdown rules. Privacy campaigners and MPs have criticised the lack of transparency and the extent of data collection, which includes sensitive information such as debt levels, living arrangements, and even notes on unfaithful sex. Xantura and Barking and Dagenham Council have defended the system as compliant with data protection rules and focused on providing support.

Company involved

UK local authorities

AI system involved

Covid OneView

6 source articles · read the reporting →

Europol cracks down on network using deepfake ads for crypto fraud

Europol and national authorities in Belgium, Bulgaria, Germany, and Israel arrested two people in November as part of a crackdown on a criminal network that used fraudulent online ads and AI-generated deepfake videos to lure victims into fake cryptocurrency investment platforms. The network allegedly laundered 700 million euros ($816 million) through the scheme, which Europol described as operating on an 'industrial' scale. The arrests were the second phase of an operation that had already led to nine arrests in October. The article does not name the suspects or firms targeted.

4 source articles · read the reporting →