University of Chicago report finds US healthcare algorithms rife with bias
A University of Chicago report alleges that algorithms used by hospitals, insurers and other businesses across the US are biased along racial and economic lines. The algorithms help triage patients, predict diabetes and flag those needing extra care, but the report says flawed products were introduced with little oversight. It says inequitable treatment has continued for more than a decade.
7 source articles · read the reporting →
Study finds credit score algorithms less accurate for minorities
A study of 50 million US consumers found that credit scoring algorithms used by mortgage lenders are less accurate for minority and low-income applicants due to sparse credit data, leading to higher rejection rates. The inaccuracy is due to noise in the data, not bias, so fairer algorithms cannot fix it. The study suggests that adjusting for bias had no effect, and that addressing the inaccuracy could reduce disparities by 50%.
- Company involved
- Mortgage lenders (unnamed)
- AI system involved
- Credit scoring algorithms
6 source articles · read the reporting →
Danish law enables algorithm to profile unemployed for long-term risk
The Danish parliament passed a law in April 2019 that allows an algorithm to analyse personal data of all unemployed individuals to identify those at risk of long-term unemployment. The system uses data on ethnicity, age, health, and education history to produce a risk score for social workers. Critics, including legal experts and unemployment insurance funds, argue the profiling is stigmatising, lacks transparency, and may violate GDPR. The Danish Data Protection Authority has reopened its review of the law.
- Company involved
- Danish Ministry of Employment
10 source articles · read the reporting →
UnitedHealthcare sued for using AI to deny Medicare Advantage patients care
A class action complaint alleges that UnitedHealthcare and its subsidiary naviHealth deployed an artificial intelligence model to override treating physicians' determinations of medically necessary care for elderly Medicare Advantage patients. The complaint claims the AI model has a known 90% error rate and wrongfully denied patients care. The lawsuit, filed in the U.S. District Court for the District of Minnesota, seeks damages on behalf of the estates of two deceased patients and all others similarly situated.
- Company involved
- UnitedHealthcare, Inc.
10 source articles · read the reporting →
Google Health's diabetic retinopathy AI faced real-world issues in Thailand clinics
Google Health deployed a deep-learning system to screen for diabetic retinopathy in 11 clinics across Thailand. The AI was highly accurate in lab tests but rejected over a fifth of eye scans in real-world conditions due to poor lighting and slow internet. Patients whose scans were rejected had to visit specialists at other clinics, causing inconvenience and frustration for nurses. Google Health is now working with local staff to improve the system's workflow.
- Company involved
- Google Health
10 source articles · read the reporting →
Stable Diffusion amplifies racial and gender stereotypes in generated images
An analysis by Bloomberg of over 5,000 images generated by Stability AI's Stable Diffusion found that the text-to-image model amplifies racial and gender stereotypes. The model overrepresented lighter-skinned men in high-paying jobs and darker-skinned people in low-paying jobs, and underrepresented women in positions of power. Stability AI acknowledged the inherent biases in its models and stated it is working on mitigation.
- Company involved
- Stability AI
- AI system involved
- Stable Diffusion
8 source articles · read the reporting →
UnitedHealth used ALERT algorithm to limit mental health coverage, regulators found
ProPublica reports that UnitedHealth Group's Optum subsidiary used the ALERT algorithm to flag mental health patients and therapists as outliers, leading to therapy coverage denials. Regulators in California, Massachusetts and New York alleged this breached the federal Mental Health Parity and Addiction Equity Act, and UnitedHealth agreed to restrict the system in those jurisdictions. The company denies wrongdoing and says its programmes are compliant. Affected patients, including Medicaid enrollees, were left to pay out-of-pocket or go without care.
- Company involved
- UnitedHealth Group
- AI system involved
- ALERT
5 source articles · read the reporting →
NHS liver transplant algorithm causes longer waits for younger patients
An NHS computer algorithm that prioritises liver transplant patients most likely to die soon has led to younger patients waiting 156 days longer on average than those over 60. The algorithm, launched in 2018, gives a score based on 21 recipient and seven donor parameters. Patients like Sarah Meredith, 31, have been waiting two years for a transplant. NHS Blood and Transplant defends the system as saving lives but acknowledges the stress on younger patients.
- Company involved
- NHS Blood and Transplant
10 source articles · read the reporting →
NHS QCovid algorithm incorrectly labels young patients as high risk, causing vaccine call-up
The article reports that scientists are questioning the reliability of the QCovid risk prediction algorithm used by the NHS to prioritise COVID-19 vaccinations. Due to missing data on weight or ethnicity, the algorithm automatically assigns a high BMI and high-risk ethnicity, leading to young, healthy patients being incorrectly labelled as high risk and called for early vaccination. GPs have identified hundreds of such cases, causing anxiety and potentially delaying vaccinations for more vulnerable individuals. NHS Digital acknowledged the issue and said clinicians can remove patients from the shielding list.
- Company involved
- NHS
- AI system involved
- QCovid
10 source articles · read the reporting →
EviCore denied heart catheterization for patient using algorithm
In fall 2021, Little John Cupp's doctor requested a left heart catheterization exam. EviCore, a company hired by UnitedHealthcare, denied the request twice using an algorithm called 'the dial' that adjusts thresholds for review. The algorithm flagged the request for review, and EviCore's doctors determined it was not medically necessary. Cupp did not receive the procedure and his symptoms continued.
- Company involved
- EviCore (a Cigna company)
- AI system involved
- the dial
6 source articles · read the reporting →
IRCC uses AI triage for Temporary Resident Visa applications
Immigration, Refugees and Citizenship Canada (IRCC) uses an AI system called Advanced Analytics to triage Temporary Resident Visa applications from India and China. The system categorizes applications into tiers, with Tier 1 approved automatically and others sent to human officers. Critics allege the system lacks transparency and may introduce bias, leading to visa refusals without clear rationale. The author, a Canadian immigration lawyer, is filing Federal Court cases on behalf of clients affected by refusals.
- Company involved
- Immigration, Refugees and Citizenship Canada (IRCC)
- AI system involved
- Advanced Analytics Triage of Overseas Temporary Resident Visa Applications
10 source articles · read the reporting →
Humana sued for using AI to deny seniors rehabilitation care
Health insurer Humana is accused in a class-action lawsuit of using an AI algorithm to systematically deny rehabilitation care to Medicare Advantage patients, despite recommendations from their doctors. The lawsuit, filed on December 12, 2023, alleges that the AI tool restricted medically necessary care. This is the second major health insurer to face legal action over its use of AI to deny care.
- Company involved
- Humana
9 source articles · read the reporting →
Judge rules lawsuit over UnitedHealth’s AI care denials can move forward - STAT
The system denied necessary medical care to Medicare Advantage beneficiaries.
- Company involved
- UnitedHealth Group
1 source article · read the reporting →
Apple Watch algorithm changes raise research data concerns
Harvard biostatistician JP Onnela found that heart rate variability data exported from an Apple Watch at two different times showed inconsistencies for the same time period, suggesting Apple's algorithms change without warning. The discovery led Onnela's team to abandon plans to use Apple Watches in research studies. Apple said changes to its algorithms only apply to data going forward and did not explain the discrepancy.
- Company involved
- Apple
- AI system involved
- Apple Watch
6 source articles · read the reporting →
EvenUp AI errors in personal injury demand letters lead to scrutiny
EvenUp, a legal tech startup valued at $1 billion, uses AI to draft personal injury demand letters. Former employees revealed that the AI system frequently makes errors, including missing injuries and fabricating medical conditions. The company defends its hybrid approach with human oversight, but critics allege overpromised AI capabilities.
- Company involved
- EvenUp
6 source articles · read the reporting →
Black man alleges racial bias in kidney transplant algorithm
An African American man has filed a lawsuit in federal court alleging that the algorithm used to determine kidney transplant priority is racially biased. The plaintiff, a Los Angeles barber, claims the algorithm puts Black patients lower on the waitlist, denying them timely access to organs. He seeks damages and a higher position on the list.
7 source articles · read the reporting →
UNOS liver allocation policy reduces transplants in poorer states
The United Network for Organ Sharing (UNOS) implemented a new liver allocation policy in 2020 called the acuity circles system, which prioritizes the sickest patients regardless of location. An analysis by The Markup and The Washington Post found that the policy led to sharp declines in liver transplants in several Southern and Midwestern states and Puerto Rico, while increasing transplants in New York and California. Patients in affected states allege the system disadvantages poorer regions, and hospitals have sued to overturn the policy.
- Company involved
- United Network for Organ Sharing (UNOS)
- AI system involved
- Acuity circles liver allocation policy
10 source articles · read the reporting →
NHS plans AI to listen to appointments and generate notes
The UK's National Health Service announced plans to use AI to automatically generate notes from patient appointments. Health Secretary Victoria Atkins said the scheme would reduce admin time. Privacy campaigners raised concerns about data security and accuracy, citing an incident where AI misheard the chief medical officer's name. The Department of Health and Social Care stated that patient confidentiality remains a top priority.
- Company involved
- National Health Service (NHS)
3 source articles · read the reporting →
Swedish welfare agency's AI system flags marginalized groups for fraud investigations
Försäkringskassan, Sweden's Social Insurance Agency, uses an AI risk-scoring system to flag welfare applicants for fraud investigations. The system disproportionately targets women, individuals with foreign backgrounds, low-income earners, and those without university degrees, according to an investigation by Lighthouse Reports and Svenska Dagbladet. Amnesty International has called for the system to be discontinued, citing violations of the right to equality and non-discrimination.
- Company involved
- Försäkringskassan (Swedish Social Insurance Agency)
6 source articles · read the reporting →
DWP algorithm mistakenly flags 200,000 Housing Benefit claimants for fraud review
The UK Department for Work and Pensions (DWP) uses an algorithm to flag Housing Benefit claimants for possible fraud or error. According to a Big Brother Watch investigation, the algorithm has mistakenly flagged 200,000 innocent people, subjecting them to intrusive reviews. Only one in three flagged cases actually had errors, compared to two in three during the pilot. The DWP has spent £4.4 million on these pointless checks.
- Company involved
- Department for Work and Pensions (DWP)
6 source articles · read the reporting →
Virginia courts' use of algorithms raises fairness concerns
The Washington Post considers the use of risk-assessment algorithms in Virginia's courts, which were introduced to make judicial decisions fairer. The analysis finds that the outcomes have been far more complicated than expected, raising concerns about the system's fairness. The algorithms affect potentially many criminal defendants across the state.
- Company involved
- Virginia court system
7 source articles · read the reporting →
Audit of RisCanvi finds biases and reliability issues in criminal justice system
Eticas conducted an adversarial audit of RisCanvi, an AI risk assessment tool used in Catalonia's criminal justice system. The audit uncovered biases in risk classifications against specific demographics and significant reliability issues. The findings call for fairer practices in criminal justice AI.
- Company involved
- Catalonia's criminal justice system
- AI system involved
- RisCanvi
4 source articles · read the reporting →
Tennessee's TennCare Connect algorithm illegally denied thousands Medicaid benefits
A U.S. District Court judge ruled that Tennessee's TennCare Connect system, built by Deloitte for over $400 million, illegally denied thousands of low-income residents and people with disabilities Medicaid and disability benefits due to programming and data errors. The system automatically terminated coverage without properly considering eligibility for all available programs. A class action lawsuit filed in 2020 resulted in the ruling.
- Company involved
- TennCare (Tennessee Medicaid)
- AI system involved
- TennCare Connect
10 source articles · read the reporting →
Deloitte software glitches wrongly remove Texans from Medicaid
Advocacy groups filed a complaint with the Federal Trade Commission alleging that Deloitte's eligibility software, TIERS, used by Texas Medicaid, wrongly disenrolled qualified recipients due to glitches. Nearly 1.8 million Texans lost coverage after the pandemic pause ended, with many errors attributed to procedural issues but some linked to system malfunctions. Deloitte denies the claims, while the state says it restored care for at least 90,000 people. The FTC has not yet responded to the complaint.
- Company involved
- Texas Health and Human Services Commission
- AI system involved
- TIERS
6 source articles · read the reporting →
