DOGE's Flawed AI Tool Threatens Veterans Affairs Services
The Department of Government Efficiency developed an AI tool to identify unnecessary VA contracts, but it used outdated models and lacked context, leading to misclassification. At least 24 contracts were canceled, affecting cancer research and nurse care tools. The VA acknowledged the tool's role but stated all contracts undergo human review. Experts criticized the use of AI for such complex decisions.
- Company involved
- Department of Veterans Affairs
- AI system involved
- Muncher
2 source articles · read the reporting →
Teachers find no value in SAS EVAAS system in Southwest School District
A study examined the SAS Education Value-Added Assessment System (EVAAS®) used by the Southwest School District (SSD) to evaluate teacher effectiveness for high-stakes consequences. Teachers reported that the system produced inconsistent results, was biased by student factors, and reduced morale and collaboration. The study found that teachers did not use the data formatively as intended, and unintended consequences included heightened pressure and teaching to the test.
- Company involved
- Southwest School District (SSD)
- AI system involved
- SAS Education Value-Added Assessment System (EVAAS®)
10 source articles · read the reporting →
Boston Public Schools reverses algorithm-generated bus schedule after parent protest
In December 2017, Boston Public Schools announced new school start times and bus schedules generated by an algorithm designed by MIT researchers. Some parents, predominantly white, protested the changes, arguing they were unfair. The city reversed the decision after the protest, despite the algorithm's aim to improve equity and sleep health for the majority of students. The incident highlighted the lack of transparency in algorithmic decision-making.
- Company involved
- Boston Public Schools
10 source articles · read the reporting →
U.S. Department of Homeland Security Uses Secret Algorithm ATLAS to Flag Naturalized Citizens for Denaturalization
The U.S. Department of Homeland Security operates a secret algorithm called ATLAS that screens naturalized citizens for potential fraud or national security concerns, flagging them for denaturalization. The system, hosted on Amazon Web Services, analyzes biometric and other data from various federal databases, and its rules are undisclosed. Human reviewers then decide whether to refer flagged individuals to Immigration and Customs Enforcement for possible deportation. Critics allege the algorithm relies on inaccurate data and lacks transparency, leading to wrongful denaturalization proceedings.
- Company involved
- U.S. Department of Homeland Security
- AI system involved
- ATLAS
2 source articles · read the reporting →
UK government using AI for benefits and immigration decisions
The UK government is reportedly using artificial intelligence to make decisions on welfare benefits and immigration applications. The system automates decisions previously made by human caseworkers. Concerns have been raised about the fairness and transparency of such automated decisions.
10 source articles · read the reporting →
Google Duplex AI assistant to identify itself as robot after criticism
Google demonstrated its Duplex AI assistant making lifelike phone calls without disclosing it was a robot, sparking accusations of deceit. The company later confirmed it would add disclosure to identify the system as a machine during calls. The feature was not yet a finished product at the time.
- Company involved
- Google
- AI system involved
- Google Duplex
10 source articles · read the reporting →
INSS AI Denies Rural Worker's Pension After Misidentifying Her as a Man
A rural worker in Brazil, Josélia de Brito, had her pension application through the Meu INSS app automatically denied after the AI system misidentified her as a man. The INSS deployed the system to speed up benefit decisions, but experts say it struggles with complex rural cases. The case is cited as evidence that automation may exclude vulnerable people with limited digital access.
- Company involved
- Instituto Nacional do Seguro Social (INSS)
- AI system involved
- Meu INSS
4 source articles · read the reporting →
Xinjiang Police App Enables Mass Surveillance and Arbitrary Detention of Uyghurs
Human Rights Watch reverse-engineered a police app used in Xinjiang, China, revealing that the Integrated Joint Operations Platform (IJOP) collects vast personal data and flags individuals as suspicious based on broad criteria. The system targets ethnic Uyghurs and Turkic Muslims, leading to mass arbitrary detention, forced indoctrination, and movement restrictions. The Chinese government operates the system, supplied by a subsidiary of CETC, and has not informed or obtained consent from those surveilled. The report calls for shutting down the system and releasing detainees.
- Company involved
- Chinese government
- AI system involved
- Integrated Joint Operations Platform (IJOP)
2 source articles · read the reporting →
Commercial network in Sri Lanka uses AI-generated anti-migrant content targeting UK
ISD and TBIJ identified a network of over 100 Facebook pages and groups run from Sri Lanka that spread AI-generated anti-migrant content to UK audiences for profit. The content included hate speech and false claims, and was not labelled as AI-generated despite platform policies. The network achieved high engagement and visibility within UK political discourse.
10 source articles · read the reporting →
McDonald's tests AI drive-thru ordering in Chicago
McDonald's is testing automated voice-ordering technology at ten drive-thru locations in Chicago. The system, developed by Apprente, uses artificial intelligence to take customer orders with about 85% accuracy, handling 80% of orders without human intervention. The company's CEO stated that wider implementation faces challenges such as menu and dialect variations. No incidents of harm were reported.
- Company involved
- McDonald's
3 source articles · read the reporting →
Intel and Class Technologies face criticism over emotion-detecting AI for students
Intel partnered with Class Technologies to use an AI system that analyses student emotions during Zoom classes to provide insights to teachers. Critics, including educator Todd Richmond, argued that facial expressions are not reliable indicators of internal states and that the system amounts to pseudoscience. The article reports these criticisms as allegations, not as proven fact. No specific incident of harm or regulatory action is reported.
- Company involved
- Intel
- AI system involved
- Class Technologies face-reading AI
10 source articles · read the reporting →
Ibrahim Diallo fired by automated HR system, locked out for three weeks
Ibrahim Diallo, a software engineer in California, was automatically terminated by an HR system after a script triggered by a missed contract renewal revoked his access and disabled his key card. Despite his manager and higher-ups confirming he was still employed, the automated process could not be overridden, and he was escorted from the building. It took three weeks for the company to resolve the issue by rehiring him as a new employee, after which he quit. The incident highlights the risks of fully automated termination systems without human override.
10 source articles · read the reporting →
UDR Sued Over Alleged Use of RealPage Algorithm to Set Rents in San Diego
A class action lawsuit alleges that UDR, Inc. used RealPage's YieldStar algorithmic pricing software to set rental rates and occupancy levels for its San Diego properties, in violation of a local ordinance. The suit claims the software relied on nonpublic competitor data, contributing to inflated rents and making housing less affordable. The lawsuit, filed in July 2026, seeks to represent all affected tenants. UDR has previously acknowledged using YieldStar as one tool among others in its decision-making.
- Company involved
- UDR, Inc.
- AI system involved
- RealPage YieldStar
1 source article · read the reporting →
AMS algorithm lacks transparency and may discriminate against job seekers
The Austrian Public Employment Service (AMS) uses an algorithm to classify job seekers into categories A, B, and C, determining their access to benefits and training. Scientists from TU Wien, WU Wien, and University of Vienna have criticised the algorithm for lacking transparency, as only two of 96 model variants have been published. They allege that the system may discriminate against women and people with migration background, and that job seekers are not informed about how the algorithm works or given a chance to appeal.
- Company involved
- AMS (Arbeitsmarktservice Österreich)
- AI system involved
- AMS-Algorithmus
10 source articles · read the reporting →
Dutch government continued SyRI welfare fraud profiling after ban
The Dutch government deployed the SyRI algorithm to profile welfare recipients in low-income neighborhoods, despite a 2020 court ruling that the system violated EU human rights. The algorithm flagged thousands of parents for fraud based on illegal risk factors such as dual nationality, leading to false accusations, bankruptcy, and trauma. The government has announced an external review of the practices.
- Company involved
- Ministry of Social Affairs of the Netherlands
- AI system involved
- SyRI
10 source articles · read the reporting →
AI drug discovery system repurposed to generate toxic molecules in demonstration
In 2020, Collaborations Pharmaceuticals demonstrated that its AI drug discovery system, MegaSyn, could be repurposed to generate toxic molecules similar to the nerve agent VX. The company ran the software overnight and produced 40,000 potentially hazardous substances. The researchers presented their findings at a conference and briefed the White House, warning that such AI systems could be misused to create chemical weapons.
- Company involved
- Collaborations Pharmaceuticals
- AI system involved
- MegaSyn
10 source articles · read the reporting →
Wisconsin’s dropout prediction algorithm labels students high risk with racial bias
Wisconsin’s Dropout Early Warning System (DEWS) uses machine learning to predict middle school students' likelihood of graduating on time, relying on factors including race and income. The Markup found the algorithm frequently mislabels students, with a higher false alarm rate for Black and Hispanic students. The Wisconsin Department of Public Instruction has not informed schools of the racial disparities and continues to use the system.
- Company involved
- Wisconsin Department of Public Instruction
- AI system involved
- Dropout Early Warning System (DEWS)
10 source articles · read the reporting →
Nevada's AI grad score model cuts funding for low-income students
Nevada replaced its income-based school funding formula with a machine learning model developed by Infinite Campus that generates a 'grad score' for each student. The model reduced the number of students eligible for supplemental funding from 288,000 to 63,000, disproportionately affecting low-income students. Critics argue the model lacks transparency and de-emphasizes economic status, potentially leaving many high-poverty schools underfunded.
- Company involved
- Nevada Department of Education
- AI system involved
- Grad score model
7 source articles · read the reporting →
ViaQuatro's facial recognition system in São Paulo metro challenged in court
In April 2018, ViaQuatro installed the Digital Interactive Doors System, developed by AdMobilize, on the São Paulo metro's yellow line. The system used cameras to detect passengers' faces and claimed to infer their emotion, age, and gender in order to target advertisements. The Brazilian Institute of Consumer Protection (IDEC) filed a public civil action alleging that the system violated consumer and data protection laws by processing biometric data without consent and making pseudoscientific and discriminatory inferences. A judge ordered the cameras removed in August 2018, and the case is pending a final ruling.
- Company involved
- ViaQuatro
- AI system involved
- Digital Interactive Doors System (DID system)
10 source articles · read the reporting →
Hospitals use Epic AI to predict Covid-19 decline without validation
Dozens of hospitals across the US are using Epic's deterioration index AI system to predict which Covid-19 patients will become critically ill, despite the tool not being validated for the new disease. The rapid deployment during the pandemic bypassed normal testing and validation processes.
- AI system involved
- Deterioration index
9 source articles · read the reporting →
Denmark's Udbetaling Danmark welfare algorithm illegally surveilled millions
Udbetaling Danmark (UDK), Denmark's centralized welfare payment agency, used automated algorithms to check eligibility and detect fraud. The system collected personal data on millions of beneficiaries and their relatives, including non-beneficiaries, in what a think-tank called 'systematic surveillance'. A complaint to the Danish Data Protection Authority led to a ruling that the data collection violated GDPR, and UDK eventually deleted the illegally obtained data. The system also caused erroneous payments, including underpaying 300,000 pensioners by 94 million euros annually.
- Company involved
- Udbetaling Danmark
10 source articles · read the reporting →
Campaigners call for end to Home Office's IPIC 'robo-caseworker' AI tool
Migrants' rights campaigners have called for the UK government to stop using the IPIC (Identify & Prioritise Immigration Cases) AI system, which automatically identifies and recommends migrants for immigration decisions or enforcement action. Privacy International and the Migrants' Rights Network have raised concerns that the system could lead to racial bias and the 'rubberstamping' of algorithmic recommendations by caseworkers. The campaigners argue that people going through the immigration system may not know their information has been processed by an algorithm. The Home Office has not directly responded to the specific concerns, though Science Secretary Peter Kyle has defended the safe development of AI in public services.
- Company involved
- Home Office
- AI system involved
- IPIC (Identify & Prioritise Immigration Cases)
9 source articles · read the reporting →
Unity faces employee concerns over military contract transparency
An investigation by Vice Games reported that Unity Technologies is not transparent with its developers about military contracts. Employees allege some are unaware their work on AI tools may become part of Department of Defense projects. CEO John Riccitiello stated the company will not support programs that violate its principles, but sources say many staff are angry. Unity said it cannot police all uses of its engine as it is a tool available to anyone.
- Company involved
- Unity Technologies
- AI system involved
- Unity engine
8 source articles · read the reporting →
Stable Diffusion amplifies racial and gender stereotypes in generated images
An analysis by Bloomberg of over 5,000 images generated by Stability AI's Stable Diffusion found that the text-to-image model amplifies racial and gender stereotypes. The model overrepresented lighter-skinned men in high-paying jobs and darker-skinned people in low-paying jobs, and underrepresented women in positions of power. Stability AI acknowledged the inherent biases in its models and stated it is working on mitigation.
- Company involved
- Stability AI
- AI system involved
- Stable Diffusion
8 source articles · read the reporting →
