New York City's McKinsey-led jail violence program manipulated data, violence increased
New York City paid McKinsey & Company $27.5 million to reduce violence at Rikers Island jail complex. McKinsey designed a predictive algorithm called the Housing Unit Balancer and Restart housing units, but jail officials and McKinsey consultants stacked the units with compliant inmates to artificially lower violence numbers. Violence actually increased by nearly 50% during the project. The city eventually decided to close Rikers.
- Company involved
- New York City Department of Correction
- AI system involved
- Housing Unit Balancer (HUB)
10 source articles · read the reporting →
LA's VI-SPDAT housing scoring system gives lower priority to Black and Latino people
The Los Angeles Homeless Services Authority uses the VI-SPDAT scoring system to prioritise unhoused people for subsidised permanent housing. An investigation by The Markup found that Black and Latino people consistently receive lower vulnerability scores than White people, leading to lower priority for housing. The agency has acknowledged the racial disparities and is working on a new tool, but continues to use the current system.
- Company involved
- Los Angeles Homeless Services Authority
- AI system involved
- VI-SPDAT
10 source articles · read the reporting →
Serbia's Social Card law could harm marginalized groups
Serbia's Social Card law, which entered into force on 1 March 2022, establishes a centralized government database to assess eligibility for social security support. Amnesty International and seven other rights organizations submitted a legal opinion to the Constitutional Court, alleging that the automated system is an intrusive surveillance system that could discriminate against Roma communities and people with disabilities. The system processes 130 categories of personal data and lacks transparency, potentially leading to errors and denial of benefits.
- Company involved
- Serbian government
- AI system involved
- Social Card system
10 source articles · read the reporting →
DeepScore markets facial and voice analysis app for trustworthiness scoring despite experts' doubts
DeepScore, a Tokyo-based company, is marketing an app that uses facial and voice recognition to score people's trustworthiness for lenders and insurers in Japan, Indonesia, Vietnam and the Philippines. The company says the app can detect deception with 70 per cent accuracy, but researchers and privacy advocates say there is no reliable scientific basis for such judgments and warn of discrimination and privacy harms. The chief executive said the system is only one part of lenders' and insurers' decision-making and that people can choose not to use it. Critics respond that an unequal balance of power makes consent difficult.
- Company involved
- DeepScore
- AI system involved
- DeepScore
6 source articles · read the reporting →
Amazon offers commitments to address EU antitrust concerns over seller data and Buy Box bias
The European Commission announced that Amazon has offered commitments to address competition concerns regarding its use of non-public data from independent sellers and alleged bias in its Buy Box and Prime programmes. The Commission preliminarily found that Amazon's automated systems may distort competition by favouring its own retail business and sellers using its logistics. Amazon commits to refrain from using seller data for retail decisions and to ensure equal treatment in Buy Box and Prime. The commitments are subject to a market test before becoming legally binding.
- Company involved
- Amazon
- AI system involved
- Buy Box and Prime algorithms
10 source articles · read the reporting →
PwC develops facial recognition tool to monitor employees working from home
Accounting giant PwC has developed a facial recognition tool that logs when employees are absent from their computer screens while working from home. The tool, intended for financial institutions, requires workers to provide written reasons for any absences, including toilet breaks. Commentators have criticised the tool as a huge invasion of privacy, with concerns about damage to trust and increased stress. PwC stated that the technology is designed to help regulated institutions meet compliance obligations and that voluntary consent of traders is essential.
- Company involved
- PwC
8 source articles · read the reporting →
USPS algorithm RRECS causes pay cuts for two-thirds of rural mail carriers
The United States Postal Service (USPS) implemented a new algorithm, RRECS, to evaluate rural carrier routes and determine pay. Due to flaws in how carriers scanned packages, the algorithm underestimated route times, resulting in pay cuts for 66 per cent of rural carriers, some losing thousands of dollars annually. Carriers report that they were not adequately trained on the system and that the cuts are scheduled to take effect, though they have been postponed multiple times. The USPS stated that the system is the result of a nationally negotiated agreement.
- Company involved
- United States Postal Service
- AI system involved
- RRECS
9 source articles · read the reporting →
University of Chicago report finds US healthcare algorithms rife with bias
A University of Chicago report alleges that algorithms used by hospitals, insurers and other businesses across the US are biased along racial and economic lines. The algorithms help triage patients, predict diabetes and flag those needing extra care, but the report says flawed products were introduced with little oversight. It says inequitable treatment has continued for more than a decade.
7 source articles · read the reporting →
Denmark's Udbetaling Danmark welfare algorithm illegally surveilled millions
Udbetaling Danmark (UDK), Denmark's centralized welfare payment agency, used automated algorithms to check eligibility and detect fraud. The system collected personal data on millions of beneficiaries and their relatives, including non-beneficiaries, in what a think-tank called 'systematic surveillance'. A complaint to the Danish Data Protection Authority led to a ruling that the data collection violated GDPR, and UDK eventually deleted the illegally obtained data. The system also caused erroneous payments, including underpaying 300,000 pensioners by 94 million euros annually.
- Company involved
- Udbetaling Danmark
10 source articles · read the reporting →
Met Police buys £3m retrospective facial recognition system
The Metropolitan Police Service (MPS) has awarded a £3 million, four-year contract to Northgate Public Services for a new retrospective facial-recognition (RFR) system to be deployed within three months. RFR processes biometric information from historic CCTV, social media, and other images to identify suspects and missing persons, operating retroactively unlike live facial recognition. The procurement was approved by the Mayor's Office for Policing and Crime in August 2021. The MPS states that human-in-the-loop decision-making will be used, but digital rights groups and a former biometrics commissioner have raised concerns about potential discrimination, overrepresentation of marginalised groups in watch lists, and lack of a legislative framework.
- Company involved
- Metropolitan Police Service
- AI system involved
- Retrospective facial-recognition software
9 source articles · read the reporting →
ScaleFactor reportedly failed to deliver promised automated bookkeeping software
ScaleFactor, an Austin-based startup, is reported to have failed to deliver the automated, real-time bookkeeping tools it promised customers, instead relying on human bookkeepers and a Filipino contract accounting firm. The company told Forbes in June that it was shutting down, initially blaming the pandemic, but Forbes later reported that its problems predated Covid-19. Investors reportedly came to see the company as more of a services business than a software platform, and pulled funding after a pivot to a marketplace model. No legal or regulatory action is reported.
- Company involved
- ScaleFactor
10 source articles · read the reporting →
Spanish Supreme Court orders release of BOSCO algorithm code for social electricity bonus
The Spanish NGO Civio won a Supreme Court case forcing the government to release the source code of BOSCO, the algorithm that decides eligibility for the social electricity bonus (bono social eléctrico). Civio had demonstrated in 2019 that BOSCO contained serious errors that denied the benefit to vulnerable people who met the requirements. The government had refused to disclose the code, citing intellectual property. The Supreme Court ruled that transparency must prevail, setting a precedent for public access to automated decision-making systems.
- Company involved
- Ministerio para la Transición Ecológica (Gobierno de España)
- AI system involved
- BOSCO
10 source articles · read the reporting →
UnitedHealth used ALERT algorithm to limit mental health coverage, regulators found
ProPublica reports that UnitedHealth Group's Optum subsidiary used the ALERT algorithm to flag mental health patients and therapists as outliers, leading to therapy coverage denials. Regulators in California, Massachusetts and New York alleged this breached the federal Mental Health Parity and Addiction Equity Act, and UnitedHealth agreed to restrict the system in those jurisdictions. The company denies wrongdoing and says its programmes are compliant. Affected patients, including Medicaid enrollees, were left to pay out-of-pocket or go without care.
- Company involved
- UnitedHealth Group
- AI system involved
- ALERT
5 source articles · read the reporting →
California EDD's automated fraud detection wrongly suspended 600,000 legitimate unemployment claims
In January 2021, the California Employment Development Department used Thompson Reuters automated batch review software to flag 1.1 million unemployment claims as potentially fraudulent. EDD stopped payments on those claims without prior notice. Later, over 600,000 were confirmed as legitimate after claimants used ID.me to verify their identity. The incident highlights the trade-off between fraud prevention and timely benefit access.
- Company involved
- California Employment Development Department (EDD)
- AI system involved
- Thompson Reuters Automated Batch Review
7 source articles · read the reporting →
Haryana government's algorithm wrongly declares thousands dead, stops pensions
The Haryana state government's Family Identity Data Repository (PPP) algorithm wrongly declared thousands of elderly and widowed citizens as dead, stopping their pensions. Dhuli Chand, a 102-year-old man, had to stage a mock wedding procession to prove he was alive after his pension was cut for six months. The government acknowledged the errors and corrected some records, but many still face difficulties.
- Company involved
- Haryana state government
- AI system involved
- Family Identity Data Repository (Parivar Pehchan Patra or PPP)
6 source articles · read the reporting →
Serco Leisure issued enforcement notices for unlawful biometric monitoring of employees
Serco Leisure and associated trusts used facial recognition and fingerprint scanning to monitor employee attendance at 38 leisure facilities. The ICO found they unlawfully processed biometric data of over 2,000 employees. Enforcement notices were issued ordering them to stop.
- Company involved
- Serco Leisure Operating Limited
8 source articles · read the reporting →
Swedish welfare agency's AI system flags marginalized groups for fraud investigations
Försäkringskassan, Sweden's Social Insurance Agency, uses an AI risk-scoring system to flag welfare applicants for fraud investigations. The system disproportionately targets women, individuals with foreign backgrounds, low-income earners, and those without university degrees, according to an investigation by Lighthouse Reports and Svenska Dagbladet. Amnesty International has called for the system to be discontinued, citing violations of the right to equality and non-discrimination.
- Company involved
- Försäkringskassan (Swedish Social Insurance Agency)
6 source articles · read the reporting →
AEMPS withdraws AI medicines tool MeQA after detecting errors
AEMPS launched MeQA, an artificial intelligence tool for answering public questions about medicines, on 13 May 2025. Two days later it withdrew the tool after detecting that some responses contained errors. The agency said that most answers were correct but that the errors could affect patient safety, and that it would restore the service as soon as possible.
- Company involved
- Agencia Española de Medicamentos y Productos Sanitarios (AEMPS)
- AI system involved
- MeQA
4 source articles · read the reporting →
Lahore safe city project stalls as facial recognition cameras fail
The Lahore safe city project in Punjab, Pakistan, has faced major challenges since 2016, with nearly 1,000 facial recognition cameras out of order. The Punjab Safe Cities Authority (PSCA) blames the contractor for failing to meet contractual clauses. Alternative measures, including reliance on privately-installed cameras, have been taken.
- Company involved
- Punjab Safe Cities Authority
- AI system involved
- Lahore safe city project
10 source articles · read the reporting →
Baltimore schools monitor student laptops for suicide signs using GoGuardian Beacon
Baltimore City Public Schools uses GoGuardian Beacon software to monitor student laptops for signs of suicide. Since March 2021, the system has flagged 786 alerts, with nine students taken to emergency rooms. Privacy advocates warn the monitoring could lead to disciplinary actions, outing of LGBTQ students, and disproportionately affect disadvantaged students. School officials defend the practice as a safeguard.
- Company involved
- Baltimore City Public Schools
- AI system involved
- GoGuardian Beacon
10 source articles · read the reporting →
DWP algorithm mistakenly flags 200,000 Housing Benefit claimants for fraud review
The UK Department for Work and Pensions (DWP) uses an algorithm to flag Housing Benefit claimants for possible fraud or error. According to a Big Brother Watch investigation, the algorithm has mistakenly flagged 200,000 innocent people, subjecting them to intrusive reviews. Only one in three flagged cases actually had errors, compared to two in three during the pilot. The DWP has spent £4.4 million on these pointless checks.
- Company involved
- Department for Work and Pensions (DWP)
6 source articles · read the reporting →
Parking Enforcement Services wrongly fines parents due to faulty licence plate cameras
Dozens of parents at a Christchurch childcare centre were wrongly issued $85 parking fines by Parking Enforcement Services after its licence plate recognition cameras failed to accurately capture multiple short visits. The company acknowledged some misreads and waived fines on appeal, but parents described the process as stressful and time-consuming. An additional camera was installed to improve accuracy.
- Company involved
- Parking Enforcement Services
1 source article · read the reporting →
DWP algorithm approved Kickstart gateways with no trading history or based abroad
An FE Week investigation found that the Department for Work and Pensions (DWP) approved dozens of companies as Kickstart gateways through automated due diligence checks using the Cabinet Office Spotlight Tool, although some had little or no trading history or were based abroad. The DWP said gateways were subject to stringent checks and later said human checks were also used. After the findings were shared with the Treasury and the DWP, the department stopped taking gateway applications and scrapped the requirement for small employers to use gateways from 3 February.
- Company involved
- Department for Work and Pensions
- AI system involved
- Cabinet Office Spotlight Tool
3 source articles · read the reporting →
GMB members report declining earnings since Uber introduced dynamic pricing
GMB Union has welcomed a report into Uber's dynamic pricing that appears to show it benefits Uber more than drivers. GMB members have reported declining earnings since the introduction of dynamic pricing. The union has called for greater transparency around earnings in the gig economy and will review the report before consulting members on next steps.
- Company involved
- Uber
- AI system involved
- dynamic pricing
8 source articles · read the reporting →
