Dutch probation service's OXREC algorithm flawed, leading to incorrect recidivism risk assessments
The Dutch Inspectorate of Justice and Security (Inspectie JenV) published a report finding that the probation service's (Reclassering) OXREC algorithm contains serious flaws, including swapped formulas and incorrect numbers, causing about a quarter of risk assessments to be wrong. The algorithm, used since 2018 for about 44,000 cases per year, also uses variables that can lead to discrimination, such as neighborhood score and income. The Inspectorate recommended immediate correction or temporary suspension. The probation service announced it would temporarily stop using OXREC.
- Company involved
- Reclassering Nederland
- AI system involved
- OXREC
4 source articles · read the reporting →
ChatGPT Health fails to direct 52% of medical emergencies to emergency care in study
A study published in Nature Medicine found that OpenAI's ChatGPT Health tool under-triaged 52% of true medical emergencies, directing users to non-urgent care instead of emergency departments. The AI also misclassified 35% of non-urgent cases. Researchers at Mount Sinai conducted 960 tests across 60 clinical scenarios, noting the tool's susceptibility to anchoring bias when symptoms were minimized. The study highlights potential safety concerns as millions use AI for health guidance.
- Company involved
- OpenAI
- AI system involved
- ChatGPT Health
4 source articles · read the reporting →
Princeton Review charges higher SAT prep fees to Asian Americans
A ProPublica study found that Princeton Review's geographically-determined pricing system charges Asian American students almost twice as likely as other ethnicities to pay the highest prices for online SAT tutoring. The system sets prices based on location, with higher prices in areas like New York City where Asian Americans are concentrated. Princeton Review stated that prices reflect local costs and competition, but the disparity raises concerns about racial discrimination in automated pricing.
- Company involved
- Princeton Review
7 source articles · read the reporting →
New Records Show Medicare WISeR AI Prior Authorization Model Causing Inappropriate Denials of Care - Medicare Rights Center
The system denied or delayed prior authorization for medical procedures, affecting Medicare beneficiaries in six states.
- Company involved
- Centers for Medicare & Medicaid Services
- AI system involved
- WISeR
1 source article · read the reporting →
UK prison risk algorithm may embed racial bias, experts warn
The Ministry of Justice developed a digital tool to categorize prisoners, which experts and advocacy groups warn could automate and embed racism. The tool uses intelligence data that may reflect existing bias, potentially leading to disproportionate categorization of BAME prisoners. The MoJ claims no evidence of discrimination, but critics question the small trial sample and lack of transparency.
- Company involved
- Ministry of Justice
- AI system involved
- Digital prisoner categorisation tool
5 source articles · read the reporting →
Citizens Advice finds ethnicity penalty in car insurance pricing
Citizens Advice conducted exploratory research into car insurance pricing and found that people of colour may be paying £250 more per year than White people. The research suggests that areas with large communities of colour may be identified as more risky by algorithms, even when objective risk factors are controlled. Citizens Advice has called on the Financial Conduct Authority to investigate the issue.
9 source articles · read the reporting →
TransUnion AI tenant screening denied 75-year-old man apartment due to mistaken littering conviction
Chris Robinson, then 75, applied for a senior living apartment in California. The property manager used an AI screening program from TransUnion, which assigned him a low score based on a mistaken conviction for littering that belonged to a different person with the same name in Texas. Robinson lost the apartment and his application fee. A federal class-action lawsuit against TransUnion moved toward a $11.5 million settlement in 2023.
10 source articles · read the reporting →
DWP suspends benefits for Bulgarian nationals using risk review system
The Department for Work and Pensions (DWP) used a 'risk review' unit to suspend universal credit benefits for dozens of Bulgarian nationals, many of whom are single mothers working part-time. The suspensions were made without explanation, leaving claimants without income, forcing some to use food banks and face eviction. An MP raised concerns that the system unfairly targets claimants based on their nationality, but the DWP denied the allegations, stating the risk review team investigates potential criminal activity and is not linked to nationality.
- Company involved
- Department for Work and Pensions
6 source articles · read the reporting →
MAA, JBG to pay DC $9.3M total to settle RealPage case - Multifamily Dive
The system set rent prices for tenants, causing them to pay higher rents.
- Company involved
- MAA (Mid-America Apartments) and JBG Smith
- AI system involved
- RealPage Revenue Management Software
1 source article · read the reporting →
Security Health Plan used AI to cut off nursing home care for 85-year-old woman
Frances Walter, an 85-year-old woman with a shattered shoulder, had her nursing home care payment cut off by Security Health Plan after an algorithm predicted she would recover in 16.6 days. The algorithm, nH Predict from NaviHealth, did not account for her severe pain and allergy to pain medicine. She was forced to spend her life savings and enroll in Medicaid while fighting the denial. A federal judge later ruled the denial was speculative and she was owed thousands of dollars.
- Company involved
- Security Health Plan
- AI system involved
- nH Predict
1 source article · read the reporting →
PredictiveHire builds AI to predict job hopping from interviews
PredictiveHire, an AI hiring firm, developed a machine-learning model that analyses candidates' open-ended interview responses to predict their likelihood of 'job hopping'. The company used data from 45,899 applicants to build the 'flight risk' assessment, which it advertises as coming soon. Scholars warn that such tools can suppress wages by screening out workers who might seek better pay or conditions, continuing a historical trend of using personality tests to identify potential labour organisers.
- Company involved
- PredictiveHire
- AI system involved
- Phai
1 source article · read the reporting →
Researchers find bias in chest X-ray AI classifiers against women, Hispanic and Medicaid patients
A study by the University of Toronto, Vector Institute, and MIT found that AI classifiers trained on public chest X-ray datasets exhibited racial, gender, and socioeconomic bias. Female patients, Hispanic patients, and those with Medicaid insurance were disproportionately affected, with the classifiers often providing incorrect diagnoses for Medicaid patients. The researchers attributed the bias to imbalanced training data and called for rigorous fairness analyses before clinical deployment.
2 source articles · read the reporting →
Spanish VioGén Algorithm Deems Woman Low Risk Before Fatal Domestic Violence
In January 2022, Lobna Hemid reported her husband's abuse to Spanish police. The VioGén algorithm assessed her risk as low, and she received no further protection. Seven weeks later, her husband fatally stabbed her. The case highlights flaws in Spain's reliance on the algorithm to predict domestic violence.
- Company involved
- Interior Ministry of Spain
- AI system involved
- VioGén
2 source articles · read the reporting →
NHS faces legal action over Palantir data contract extension
The NHS is being taken to court by campaign group Open Democracy over its contract with data firm Palantir. The legal action alleges that the extension of Palantir's involvement in analysing NHS patient data for pandemic response and beyond lacked a proper Data Protection Impact Assessment. The contract, initially an emergency response, was extended for two years at a cost of £23.5m. The case is pending.
- Company involved
- NHS
- AI system involved
- Palantir data analysis platform
10 source articles · read the reporting →
Insurer drops California homeowner after aerial photo analysis
Cindy Picos was dropped by her home insurer after the company used aerial photos to assess her roof. The insurer refused to let her see the photos, and she believes her roof is in fine shape. The incident highlights the growing use of drones and aerial imagery by insurers to inspect properties without in-person visits.
2 source articles · read the reporting →
Facebook's automated ad system rejects adaptive fashion ads
Small adaptive fashion companies have had their advertisements repeatedly rejected by Facebook's automated advertising system, which misidentifies their products as medical devices. The system's inability to understand context leads to discrimination against people with disabilities. Facebook has acknowledged the issue and restored some ads, but the problem persists, highlighting biases in machine learning.
- Company involved
- Facebook
- AI system involved
- Facebook automated advertising review system
2 source articles · read the reporting →
Racial bias in lung test software led to underdiagnosis of Black men, study suggests
A study published in JAMA Network Open found that race-based adjustments in diagnostic software for lung function likely led to underdiagnosis of breathing problems in Black men. Researchers analyzed data from over 2,700 Black men and 5,700 white men at the University of Pennsylvania Health System and concluded that up to 40% more Black men might have been diagnosed if the algorithm were changed. The American Thoracic Society has recommended replacing race-focused adjustments, but changes may take time.
- Company involved
- University of Pennsylvania Health System
2 source articles · read the reporting →
Brookdale Senior Living algorithm blamed for understaffing at assisted-living facilities
Managers at Brookdale Senior Living, the largest assisted-living chain in the US, allege that an algorithm called 'Service Alignment' set staffing levels so low that facilities were dangerously short-handed. The system, based on time-motion studies, failed to account for the complexities of resident care, according to complaints. Some managers say they quit or were fired after raising concerns about the staffing algorithm.
- Company involved
- Brookdale Senior Living
- AI system involved
- Service Alignment
1 source article · read the reporting →
Penn Medicine study: kidney algorithm reduced Black transplant access
For years, a clinical algorithm used for the national kidney transplant waitlist made Black patients appear healthier than they were, making it harder for them to secure transplant spots. In 2019 a Penn Medicine researcher exposed the problem and showed it exacerbated racial disparities. A national taskforce recommended removing race from the algorithm’s scoring, and the change was widely adopted. Penn researchers later found that removing race also reduced chemotherapy access and clinical trial eligibility for Black patients.
- AI system involved
- kidney function algorithm
1 source article · read the reporting →
U.S. hospitals' race-based eGFR calculation delayed Black kidney patients' transplants
For years, U.S. transplant hospitals used a race-based version of the eGFR equation to estimate Black patients' kidney function. The adjustment inflated Black patients' readings, delaying their referral to the kidney transplant waitlist and leaving some to die or deteriorate on dialysis. After years of advocacy, the United Network for Organ Sharing prohibited the race-based calculation in 2022 and required transplant programmes to revise affected patients' waiting times.
- Company involved
- U.S. transplant hospitals
- AI system involved
- eGFR (estimated glomerular filtration rate)
2 source articles · read the reporting →
Facebook exempted Trump and other high-profile users from content enforcement rules.
An internal Facebook program called XCheck granted special treatment to millions of high-profile users, including former President Donald Trump and Senator Elizabeth Warren, exempting them from content enforcement rules. The program, initially for quality control, was expanded to whitelist users deemed newsworthy or PR-risky, with the system failing to enforce rules against them. Internal documents revealed that the program made mistakes, often wrongly taking action against high-profile users. Facebook has attempted to phase out the practice but has struggled.
- Company involved
- Facebook
- AI system involved
- XCheck
10 source articles · read the reporting →
VA's Suicide Prevention Algorithm Favours Men, Ignores Sexual Trauma
The Department of Veterans Affairs' REACH VET algorithm, designed to identify veterans at high risk of suicide, reportedly prioritises white men and excludes factors like military sexual trauma that disproportionately affect women. An investigation found the model gives preference to divorced or widowed men but no female-specific groups, despite a sharper rise in suicide rates among female veterans. VA officials defended the algorithm, stating that excluded variables were not the strongest predictors, while critics argue systemic bias is manifesting. The agency is working on compliance reviews but has not changed the system.
- Company involved
- Department of Veterans Affairs
- AI system involved
- REACH VET
2 source articles · read the reporting →
Hospital Denies Pain Medication to Patient Based on Narx Score
Elizabeth Amirault, a 34-year-old chronic pain patient, was denied narcotic pain medication during a hospital visit in Fort Wayne, Indiana, in August 2022. A nurse practitioner told her that her Narx Score, generated by Bamboo Health's NarxCare platform, was too high to prescribe opioids. The algorithm uses prescription drug monitoring data to assess risk of misuse, but critics say it can harm patients by cutting off needed care. Amirault believes her score was unfairly high due to her multiple surgeries and doctors.
- AI system involved
- NarxCare
3 source articles · read the reporting →
State Farm accused of algorithmic bias against Black homeowners
A class action lawsuit filed in Chicago alleges State Farm's automated claims processing system discriminates against Black policyholders. The suit claims Black customers are more likely to face additional paperwork and longer waits for claim approval than white customers, citing data from 800 homeowners. State Farm has stated it takes the filing seriously and is committed to fair treatment, while the plaintiffs seek to enjoin the current system and require audits.
- Company involved
- State Farm
- AI system involved
- automated claims processing system
2 source articles · read the reporting →
