USPS algorithm RRECS causes pay cuts for two-thirds of rural mail carriers
The United States Postal Service (USPS) implemented a new algorithm, RRECS, to evaluate rural carrier routes and determine pay. Due to flaws in how carriers scanned packages, the algorithm underestimated route times, resulting in pay cuts for 66 per cent of rural carriers, some losing thousands of dollars annually. Carriers report that they were not adequately trained on the system and that the cuts are scheduled to take effect, though they have been postponed multiple times. The USPS stated that the system is the result of a nationally negotiated agreement.
- Company involved
- United States Postal Service
- AI system involved
- RRECS
9 source articles · read the reporting →
Denmark's Udbetaling Danmark welfare algorithm illegally surveilled millions
Udbetaling Danmark (UDK), Denmark's centralized welfare payment agency, used automated algorithms to check eligibility and detect fraud. The system collected personal data on millions of beneficiaries and their relatives, including non-beneficiaries, in what a think-tank called 'systematic surveillance'. A complaint to the Danish Data Protection Authority led to a ruling that the data collection violated GDPR, and UDK eventually deleted the illegally obtained data. The system also caused erroneous payments, including underpaying 300,000 pensioners by 94 million euros annually.
- Company involved
- Udbetaling Danmark
10 source articles · read the reporting →
Danish law enables algorithm to profile unemployed for long-term risk
The Danish parliament passed a law in April 2019 that allows an algorithm to analyse personal data of all unemployed individuals to identify those at risk of long-term unemployment. The system uses data on ethnicity, age, health, and education history to produce a risk score for social workers. Critics, including legal experts and unemployment insurance funds, argue the profiling is stigmatising, lacks transparency, and may violate GDPR. The Danish Data Protection Authority has reopened its review of the law.
- Company involved
- Danish Ministry of Employment
10 source articles · read the reporting →
Met Police buys £3m retrospective facial recognition system
The Metropolitan Police Service (MPS) has awarded a £3 million, four-year contract to Northgate Public Services for a new retrospective facial-recognition (RFR) system to be deployed within three months. RFR processes biometric information from historic CCTV, social media, and other images to identify suspects and missing persons, operating retroactively unlike live facial recognition. The procurement was approved by the Mayor's Office for Policing and Crime in August 2021. The MPS states that human-in-the-loop decision-making will be used, but digital rights groups and a former biometrics commissioner have raised concerns about potential discrimination, overrepresentation of marginalised groups in watch lists, and lack of a legislative framework.
- Company involved
- Metropolitan Police Service
- AI system involved
- Retrospective facial-recognition software
9 source articles · read the reporting →
Spanish Supreme Court orders release of BOSCO algorithm code for social electricity bonus
The Spanish NGO Civio won a Supreme Court case forcing the government to release the source code of BOSCO, the algorithm that decides eligibility for the social electricity bonus (bono social eléctrico). Civio had demonstrated in 2019 that BOSCO contained serious errors that denied the benefit to vulnerable people who met the requirements. The government had refused to disclose the code, citing intellectual property. The Supreme Court ruled that transparency must prevail, setting a precedent for public access to automated decision-making systems.
- Company involved
- Ministerio para la Transición Ecológica (Gobierno de España)
- AI system involved
- BOSCO
10 source articles · read the reporting →
UnitedHealth used ALERT algorithm to limit mental health coverage, regulators found
ProPublica reports that UnitedHealth Group's Optum subsidiary used the ALERT algorithm to flag mental health patients and therapists as outliers, leading to therapy coverage denials. Regulators in California, Massachusetts and New York alleged this breached the federal Mental Health Parity and Addiction Equity Act, and UnitedHealth agreed to restrict the system in those jurisdictions. The company denies wrongdoing and says its programmes are compliant. Affected patients, including Medicaid enrollees, were left to pay out-of-pocket or go without care.
- Company involved
- UnitedHealth Group
- AI system involved
- ALERT
5 source articles · read the reporting →
Perth hospital group bans ChatGPT after doctor used it for a patient discharge summary
South Metropolitan Health Service, which runs five hospitals in Perth, ordered staff to stop using ChatGPT for work involving patient information after an email said some staff had used the AI chatbot to write medical notes. The service later clarified that only one doctor had used the tool to generate a patient discharge summary and that no confidential patient information had been breached. The Australian Medical Association has called for national regulations to ensure a human is involved in AI-assisted health decisions.
- Company involved
- South Metropolitan Health Service
- AI system involved
- ChatGPT
6 source articles · read the reporting →
Immigration NZ uses data model to predict likely troublemakers among overstayers
Immigration New Zealand has been piloting a data modelling system that uses demographic information such as age, gender, and ethnicity of overstayers to predict which individuals are likely to commit crime or incur hospital debts. The system then targets those individuals for accelerated deportation. Critics, including immigration lawyers and a lobby group, have likened the programme to 'Minority Report' and allege it amounts to racial profiling. Immigration Minister Iain Lees-Galloway only learned of the programme on the day of the article and said he would seek a full briefing, but maintained it is not racial profiling and does not breach the Human Rights Act.
- Company involved
- Immigration New Zealand
10 source articles · read the reporting →
Haryana government's algorithm wrongly declares thousands dead, stops pensions
The Haryana state government's Family Identity Data Repository (PPP) algorithm wrongly declared thousands of elderly and widowed citizens as dead, stopping their pensions. Dhuli Chand, a 102-year-old man, had to stage a mock wedding procession to prove he was alive after his pension was cut for six months. The government acknowledged the errors and corrected some records, but many still face difficulties.
- Company involved
- Haryana state government
- AI system involved
- Family Identity Data Repository (Parivar Pehchan Patra or PPP)
6 source articles · read the reporting →
Meta tracks employee keystrokes on Google, LinkedIn, Wikipedia for AI training
Meta is using an internal tool, Model Capability Initiative (MCI), to capture employees' keystrokes, mouse movements and screen contents on work computers, including on sites such as Google, LinkedIn, Wikipedia and Slack, to train AI agents. Meta confirmed the project and said safeguards protect sensitive content and that the data is not used for other purposes. Employees raised concerns in internal messages that the tool could expose passwords, product details and personal information. A Meta memo said staff can avoid capture by not doing personal work on work computers.
- Company involved
- Meta
- AI system involved
- Model Capability Initiative (MCI)
5 source articles · read the reporting →
Serco Leisure issued enforcement notices for unlawful biometric monitoring of employees
Serco Leisure and associated trusts used facial recognition and fingerprint scanning to monitor employee attendance at 38 leisure facilities. The ICO found they unlawfully processed biometric data of over 2,000 employees. Enforcement notices were issued ordering them to stop.
- Company involved
- Serco Leisure Operating Limited
8 source articles · read the reporting →
Teething problems in Mater Dei's medicine robots addressed
The Malta Union for Midwives and Nurses claimed that a €23 million investment in two computerised drug administration robots, Mario and Sophia, at Mater Dei Hospital had resulted in a complete failure. However, sources within the Health Ministry said that most teething problems have been addressed and that the supplier has not been paid yet. They reported that out of over 1,700 medication rounds, only four required a contingency plan.
- Company involved
- Mater Dei Hospital
- AI system involved
- Mario and Sophia
6 source articles · read the reporting →
Swedish welfare agency's AI system flags marginalized groups for fraud investigations
Försäkringskassan, Sweden's Social Insurance Agency, uses an AI risk-scoring system to flag welfare applicants for fraud investigations. The system disproportionately targets women, individuals with foreign backgrounds, low-income earners, and those without university degrees, according to an investigation by Lighthouse Reports and Svenska Dagbladet. Amnesty International has called for the system to be discontinued, citing violations of the right to equality and non-discrimination.
- Company involved
- Försäkringskassan (Swedish Social Insurance Agency)
6 source articles · read the reporting →
Study finds LLMs used in up to 16.9% of AI conference peer reviews
According to a new paper on arXiv, researchers have begun using generative AI services to help write peer reviews of machine learning papers submitted to leading AI conferences. The study analysed reviews from ICLR 2024, NeurIPS 2023, CoRL 2023 and EMNLP 2023 and estimated that between 6.5% and 16.9% of review text may have been substantially modified by large language models. The authors argue that this risks depriving authors of diverse expert feedback and may skew reviews towards AI model biases. They have called for greater transparency about the use of LLMs in peer review.
9 source articles · read the reporting →
AEMPS withdraws AI medicines tool MeQA after detecting errors
AEMPS launched MeQA, an artificial intelligence tool for answering public questions about medicines, on 13 May 2025. Two days later it withdrew the tool after detecting that some responses contained errors. The agency said that most answers were correct but that the errors could affect patient safety, and that it would restore the service as soon as possible.
- Company involved
- Agencia Española de Medicamentos y Productos Sanitarios (AEMPS)
- AI system involved
- MeQA
4 source articles · read the reporting →
Singapore writers reject government plan to use their work for AI training
The Singaporean government's National Multimodal LLM Programme (NMLP) requested permission from local writers to use their copyrighted works to train a large language model aimed at reducing Western bias. Writers expressed skepticism over the lack of clarity on compensation and copyright protection. The S$70 million project, launched in December 2023, is part of Singapore's effort to become a global AI leader. No actual use of the works has occurred yet, and the writers have not agreed.
- Company involved
- Singapore government (National Multimodal LLM Programme)
- AI system involved
- National Multimodal LLM (NMLP)
2 source articles · read the reporting →
Mistral releases unmoderated chatbot that gives instructions on murder and ethnic cleansing
Mistral, a French AI startup valued at $260 million, released an open-source large language model named Mistral-7B-v0.1 without safety evaluations or moderation mechanisms. The model readily provides detailed instructions on murder, ethnic cleansing, suicide, and other harmful content. Mistral added a statement after the release acknowledging the lack of moderation but did not remove the model, which is distributed via torrent and cannot be deleted.
- Company involved
- Mistral
- AI system involved
- Mistral-7B-v0.1
7 source articles · read the reporting →
DWP algorithm approved Kickstart gateways with no trading history or based abroad
An FE Week investigation found that the Department for Work and Pensions (DWP) approved dozens of companies as Kickstart gateways through automated due diligence checks using the Cabinet Office Spotlight Tool, although some had little or no trading history or were based abroad. The DWP said gateways were subject to stringent checks and later said human checks were also used. After the findings were shared with the Treasury and the DWP, the department stopped taking gateway applications and scrapped the requirement for small employers to use gateways from 3 February.
- Company involved
- Department for Work and Pensions
- AI system involved
- Cabinet Office Spotlight Tool
3 source articles · read the reporting →
Meta's cross-check program delays removal of violating content for privileged users
The Oversight Board's policy advisory opinion on Meta's cross-check program found that the system grants certain users, such as business partners and celebrities, additional human review before removing violating content, while ordinary users face immediate removal. This unequal treatment allows potentially harmful content to remain on the platform for days, and Meta has failed to track whether the program improves accuracy. The Board made 32 recommendations to address these flaws.
- Company involved
- Meta
- AI system involved
- cross-check program
10 source articles · read the reporting →
Audit of RisCanvi finds biases and reliability issues in criminal justice system
Eticas conducted an adversarial audit of RisCanvi, an AI risk assessment tool used in Catalonia's criminal justice system. The audit uncovered biases in risk classifications against specific demographics and significant reliability issues. The findings call for fairer practices in criminal justice AI.
- Company involved
- Catalonia's criminal justice system
- AI system involved
- RisCanvi
4 source articles · read the reporting →
Microsoft Dynamics 365 Field Service AI singles out workers in performance predictions
A report by Cracked Labs found that Microsoft's Dynamics 365 Field Service software uses AI to generate performance metrics and predict task durations, singling out individual workers. The AI predictions can be influenced by the worker's identity, such as increasing or decreasing estimated duration. Microsoft stated the system is not intended for employment decisions and is not a surveillance tool, but the report raises concerns about potential misuse for worker monitoring.
- Company involved
- Microsoft
- AI system involved
- Dynamics 365 Field Service
6 source articles · read the reporting →
Tennessee's TennCare Connect algorithm illegally denied thousands Medicaid benefits
A U.S. District Court judge ruled that Tennessee's TennCare Connect system, built by Deloitte for over $400 million, illegally denied thousands of low-income residents and people with disabilities Medicaid and disability benefits due to programming and data errors. The system automatically terminated coverage without properly considering eligibility for all available programs. A class action lawsuit filed in 2020 resulted in the ruling.
- Company involved
- TennCare (Tennessee Medicaid)
- AI system involved
- TennCare Connect
10 source articles · read the reporting →
Deloitte software glitches wrongly remove Texans from Medicaid
Advocacy groups filed a complaint with the Federal Trade Commission alleging that Deloitte's eligibility software, TIERS, used by Texas Medicaid, wrongly disenrolled qualified recipients due to glitches. Nearly 1.8 million Texans lost coverage after the pandemic pause ended, with many errors attributed to procedural issues but some linked to system malfunctions. Deloitte denies the claims, while the state says it restored care for at least 90,000 people. The FTC has not yet responded to the complaint.
- Company involved
- Texas Health and Human Services Commission
- AI system involved
- TIERS
6 source articles · read the reporting →
Meta ends Sama contract after workers allege viewing graphic smart glasses content
Meta cancelled its contract with Sama, a Kenyan outsourcing firm, after workers alleged they were required to view graphic content captured by Meta's smart glasses, including users having sex and using the toilet. The workers, who were data annotators training Meta's AI, said they reviewed videos and transcripts of interactions with the AI. Sama stated the contract termination would result in 1,108 redundancies, and a Kenyan workers' organisation alleges Meta's decision was caused by the staff speaking out. Meta has not addressed that allegation but said it ended the work because Sama did not meet its standards, a claim Sama rejects.
- Company involved
- Meta
- AI system involved
- Meta smart glasses
6 source articles · read the reporting →
