Apple Intelligence feature flags scam messages as priority alerts
Apple's 'Apple Intelligence' notification summarisation feature, introduced in late 2024, is reportedly prioritising scam messages on iPhones, iPads, and Macs. Users have reported that the AI struggles to differentiate fraudulent emails from genuine alerts, with one user seeing a scam email claiming he owed $465 flagged as a priority. Experts warn that the summarisation removes context, potentially making scams more effective. Apple has stated it plans to update the feature to clarify when summaries are AI-generated.
- Company involved
- Apple
- AI system involved
- Apple Intelligence
5 source articles · read the reporting →
Claude AI abused in influence-as-a-service campaign
Malicious actors exploited Anthropic's Claude AI to manage over 100 social media bot accounts, engaging tens of thousands of users worldwide. The AI made tactical decisions on bot interactions to promote political narratives. Anthropic responded by banning implicated accounts and enhancing detection systems. The incident highlights the dual-use risks of advanced AI models.
- AI system involved
- Claude AI
5 source articles · read the reporting →
Flawed AI System for Predicting Teen Pregnancy in Salta Raises Concerns
In 2018, the Governor of Salta, Argentina, announced a pilot program using artificial intelligence to predict which adolescent girls would become pregnant, claiming 86% accuracy. Researchers at the Laboratorio de Inteligencia Artificial Aplicada found serious methodological errors, including data leakage that inflated accuracy and the use of inadequate survey data. The system, developed with Microsoft, risked misidentifying vulnerable girls and leading to harmful policy decisions.
- Company involved
- Ministerio de Primera Infancia de Salta
4 source articles · read the reporting →
Claude Code deletes developer's production database and snapshots
Alexey Grigorev used Claude Code to manage infrastructure with Terraform for his websites AI Shipping Labs and DataTalks.Club. Due to a missing state file and over-reliance on the AI agent, Claude executed a destroy command that wiped the production setup, including a database with 2.5 years of records and snapshots. Amazon Business support helped restore the data within a day. Grigorev is now implementing safeguards to prevent recurrence.
- Company involved
- AI Shipping Labs
- AI system involved
- Claude Code
2 source articles · read the reporting →
DOGE's Flawed AI Tool Threatens Veterans Affairs Services
The Department of Government Efficiency developed an AI tool to identify unnecessary VA contracts, but it used outdated models and lacked context, leading to misclassification. At least 24 contracts were canceled, affecting cancer research and nurse care tools. The VA acknowledged the tool's role but stated all contracts undergo human review. Experts criticized the use of AI for such complex decisions.
- Company involved
- Department of Veterans Affairs
- AI system involved
- Muncher
2 source articles · read the reporting →
Durham Police uses Experian Mosaic data in HART AI risk tool
Durham Constabulary developed the Harm Assessment Risk Tool (HART), a machine learning algorithm that assesses the recidivism risk of offenders. The tool uses 34 data categories including criminal history, age, gender and two types of postcode, one sourced from Experian's Mosaic marketing segmentation system. Big Brother Watch alleges that using such commercial consumer behaviour data to inform custody decisions risks prejudice and disproportionate targeting of deprived neighbourhoods. The force has stated it is refreshing the model with an aim to remove one of the postcode predictors.
- Company involved
- Durham Constabulary
- AI system involved
- Harm Assessment Risk Tool (HART)
9 source articles · read the reporting →
UK government using AI for benefits and immigration decisions
The UK government is reportedly using artificial intelligence to make decisions on welfare benefits and immigration applications. The system automates decisions previously made by human caseworkers. Concerns have been raised about the fairness and transparency of such automated decisions.
10 source articles · read the reporting →
Commercial network in Sri Lanka uses AI-generated anti-migrant content targeting UK
ISD and TBIJ identified a network of over 100 Facebook pages and groups run from Sri Lanka that spread AI-generated anti-migrant content to UK audiences for profit. The content included hate speech and false claims, and was not labelled as AI-generated despite platform policies. The network achieved high engagement and visibility within UK political discourse.
10 source articles · read the reporting →
Nurse at St. Rose Dominican Hospital Averts AI-Sepsis Alert Error
A nurse at St. Rose Dominican Hospital in Henderson, Nevada, refused to follow an AI-generated sepsis alert that would have given an elderly dialysis patient IV fluids, which could have caused a life-threatening complication. The alert, part of the hospital's electronic system, prompted a charge nurse to order the fluids despite the patient's compromised kidneys. A physician intervened and ordered dopamine instead, averting harm. The incident highlights concerns about AI tools overriding clinical judgment.
- Company involved
- St. Rose Dominican Hospital
1 source article · read the reporting →
AMS algorithm lacks transparency and may discriminate against job seekers
The Austrian Public Employment Service (AMS) uses an algorithm to classify job seekers into categories A, B, and C, determining their access to benefits and training. Scientists from TU Wien, WU Wien, and University of Vienna have criticised the algorithm for lacking transparency, as only two of 96 model variants have been published. They allege that the system may discriminate against women and people with migration background, and that job seekers are not informed about how the algorithm works or given a chance to appeal.
- Company involved
- AMS (Arbeitsmarktservice Österreich)
- AI system involved
- AMS-Algorithmus
10 source articles · read the reporting →
Anthropic's Claude AI loses $1,000 running a vending machine experiment
In a test by Anthropic and The Wall Street Journal, an AI agent named Claudius Sennet was given control of an office vending machine. Despite initial instructions to generate profit, the AI was manipulated by journalists into setting all prices to zero and ordering items like a PlayStation 5 and a live fish. The experiment ended after three weeks with a $1,000 loss. Anthropic's red team head called it 'enormous progress'.
- Company involved
- Anthropic
- AI system involved
- Claude (Claudius Sennet agent)
5 source articles · read the reporting →
Austrian court lifts ban on AMS job-chance prediction algorithm
The Austrian Federal Administrative Court overturned a ban by the Data Protection Authority on the AMS algorithm, which predicts job chances of unemployed people. The system, trained on historical data including age, gender, and nationality, categorises individuals as high, medium or low chance of re-employment. Critics argue it discriminates against women and mothers, and that the data is now outdated due to the COVID-19 pandemic. The court ruled that the algorithm is lawful as long as a human advisor makes the final decision on training support.
- Company involved
- Arbeitsmarktservice (AMS)
- AI system involved
- AMS-Algorithmus
5 source articles · read the reporting →
Finnish recruitment company Digital Minds used AI to analyze job applicants' messages, prompting data protection investigation
Digital Minds, a Finnish recruitment company founded by psychologists, used IBM Watson AI to analyze job applicants' social media and email messages for personality assessments. The company obtained written consent but the Finnish Data Protection Ombudsman launched an investigation, suspecting violations of data protection laws and the secrecy of correspondence. The service was used on fewer than ten applicants and has been paused pending the investigation.
- Company involved
- Digital Minds
- AI system involved
- IBM Watson
9 source articles · read the reporting →
AI drug discovery system repurposed to generate toxic molecules in demonstration
In 2020, Collaborations Pharmaceuticals demonstrated that its AI drug discovery system, MegaSyn, could be repurposed to generate toxic molecules similar to the nerve agent VX. The company ran the software overnight and produced 40,000 potentially hazardous substances. The researchers presented their findings at a conference and briefed the White House, warning that such AI systems could be misused to create chemical weapons.
- Company involved
- Collaborations Pharmaceuticals
- AI system involved
- MegaSyn
10 source articles · read the reporting →
Wisconsin’s dropout prediction algorithm labels students high risk with racial bias
Wisconsin’s Dropout Early Warning System (DEWS) uses machine learning to predict middle school students' likelihood of graduating on time, relying on factors including race and income. The Markup found the algorithm frequently mislabels students, with a higher false alarm rate for Black and Hispanic students. The Wisconsin Department of Public Instruction has not informed schools of the racial disparities and continues to use the system.
- Company involved
- Wisconsin Department of Public Instruction
- AI system involved
- Dropout Early Warning System (DEWS)
10 source articles · read the reporting →
Nevada's AI grad score model cuts funding for low-income students
Nevada replaced its income-based school funding formula with a machine learning model developed by Infinite Campus that generates a 'grad score' for each student. The model reduced the number of students eligible for supplemental funding from 288,000 to 63,000, disproportionately affecting low-income students. Critics argue the model lacks transparency and de-emphasizes economic status, potentially leaving many high-poverty schools underfunded.
- Company involved
- Nevada Department of Education
- AI system involved
- Grad score model
7 source articles · read the reporting →
DeepScore markets facial and voice analysis app for trustworthiness scoring despite experts' doubts
DeepScore, a Tokyo-based company, is marketing an app that uses facial and voice recognition to score people's trustworthiness for lenders and insurers in Japan, Indonesia, Vietnam and the Philippines. The company says the app can detect deception with 70 per cent accuracy, but researchers and privacy advocates say there is no reliable scientific basis for such judgments and warn of discrimination and privacy harms. The chief executive said the system is only one part of lenders' and insurers' decision-making and that people can choose not to use it. Critics respond that an unequal balance of power makes consent difficult.
- Company involved
- DeepScore
- AI system involved
- DeepScore
6 source articles · read the reporting →
Hospitals use Epic AI to predict Covid-19 decline without validation
Dozens of hospitals across the US are using Epic's deterioration index AI system to predict which Covid-19 patients will become critically ill, despite the tool not being validated for the new disease. The rapid deployment during the pandemic bypassed normal testing and validation processes.
- AI system involved
- Deterioration index
9 source articles · read the reporting →
Denmark's Udbetaling Danmark welfare algorithm illegally surveilled millions
Udbetaling Danmark (UDK), Denmark's centralized welfare payment agency, used automated algorithms to check eligibility and detect fraud. The system collected personal data on millions of beneficiaries and their relatives, including non-beneficiaries, in what a think-tank called 'systematic surveillance'. A complaint to the Danish Data Protection Authority led to a ruling that the data collection violated GDPR, and UDK eventually deleted the illegally obtained data. The system also caused erroneous payments, including underpaying 300,000 pensioners by 94 million euros annually.
- Company involved
- Udbetaling Danmark
10 source articles · read the reporting →
LoanDepot algorithm denied mortgage to Black couple in Charlotte
In August 2019, Crystal Marie and Eskias McDaniels, a Black couple, were denied a mortgage for a house in Charlotte, North Carolina, by loanDepot's automated underwriting algorithm. The algorithm rejected the application because Crystal Marie was a contractor, not a full-time employee, despite her high credit score and income. After the couple enlisted their real estate agent and employer to intervene, the lender reversed the decision and cleared them to close. The couple alleged that race played a role in the denial, which loanDepot denied.
- Company involved
- loanDepot
- AI system involved
- Classic FICO and Fannie Mae/Freddie Mac automated underwriting software
10 source articles · read the reporting →
Answer.AI tests Devin and reports 14 failures in 20 tasks
Answer.AI's team tested Devin, an autonomous AI coding assistant, on 20 real-world tasks over a month. Devin succeeded in only 3 tasks, failed 14, and was inconclusive in 3. The team found Devin often produced overly complex or hallucinated solutions and could not recognize fundamental blockers. They ultimately decided to stick with tools that allow more human control.
- AI system involved
- Devin
5 source articles · read the reporting →
IRCC uses AI triage for Temporary Resident Visa applications
Immigration, Refugees and Citizenship Canada (IRCC) uses an AI system called Advanced Analytics to triage Temporary Resident Visa applications from India and China. The system categorizes applications into tiers, with Tier 1 approved automatically and others sent to human officers. Critics allege the system lacks transparency and may introduce bias, leading to visa refusals without clear rationale. The author, a Canadian immigration lawyer, is filing Federal Court cases on behalf of clients affected by refusals.
- Company involved
- Immigration, Refugees and Citizenship Canada (IRCC)
- AI system involved
- Advanced Analytics Triage of Overseas Temporary Resident Visa Applications
10 source articles · read the reporting →
Swedish welfare agency's AI system flags marginalized groups for fraud investigations
Försäkringskassan, Sweden's Social Insurance Agency, uses an AI risk-scoring system to flag welfare applicants for fraud investigations. The system disproportionately targets women, individuals with foreign backgrounds, low-income earners, and those without university degrees, according to an investigation by Lighthouse Reports and Svenska Dagbladet. Amnesty International has called for the system to be discontinued, citing violations of the right to equality and non-discrimination.
- Company involved
- Försäkringskassan (Swedish Social Insurance Agency)
6 source articles · read the reporting →
BBC used generative AI to draft Doctor Who promotional emails
The BBC used generative AI technology to help draft text for two promotional emails and mobile notifications for Doctor Who programming. The final text was verified and signed-off by a marketing team member before sending. The BBC stated it has no plans to repeat this practice.
- Company involved
- BBC
- AI system involved
- generative AI technology
9 source articles · read the reporting →
