Idaho Medicaid used secret formula to cut assistance for disabled adults
The Idaho Department of Health and Welfare used a secret formula to drastically cut Medicaid assistance for adults with developmental disabilities, claiming the formula was a trade secret. The ACLU of Idaho filed a class action lawsuit, K.W. v. Armstrong, in 2012, alleging the cuts violated due process. A federal court struck down the formula in 2016 and ordered a new system, but as of 2024, a fair system has not yet been implemented.
- Company involved
- Idaho Department of Health and Welfare
10 source articles · read the reporting →
EEOC sues iTutorGroup for automated rejection of older tutor applicants
The US Equal Employment Opportunity Commission (EEOC) alleges that tutoring provider iTutorGroup programmed its recruitment software automatically to reject female applicants aged 55 or older and male applicants aged 60 or older. More than 200 qualified US-based applicants were denied work in 2020. The EEOC has filed a lawsuit in the Eastern District of New York seeking back pay and damages.
- Company involved
- iTutorGroup
10 source articles · read the reporting →
AMS algorithm lacks transparency and may discriminate against job seekers
The Austrian Public Employment Service (AMS) uses an algorithm to classify job seekers into categories A, B, and C, determining their access to benefits and training. Scientists from TU Wien, WU Wien, and University of Vienna have criticised the algorithm for lacking transparency, as only two of 96 model variants have been published. They allege that the system may discriminate against women and people with migration background, and that job seekers are not informed about how the algorithm works or given a chance to appeal.
- Company involved
- AMS (Arbeitsmarktservice Österreich)
- AI system involved
- AMS-Algorithmus
10 source articles · read the reporting →
DWP's Universal Credit algorithm causes underpayment and hardship for claimants
The Department for Work and Pensions (DWP) uses an automated means-testing algorithm in its Universal Credit system that miscalculates earnings due to a design flaw, leading to underpayments and fluctuating benefit amounts. Claimants such as Janet R. received payments hundreds of pounds short, forcing them into debt and foodbank use. A Court of Appeal ruling in June 2020 ordered the DWP to fix the flaw, but the department had not yet fully implemented the remedy at the time of writing.
- Company involved
- Department for Work and Pensions
- AI system involved
- Universal Credit
10 source articles · read the reporting →
Titus Henderson files civil rights lawsuit against COMPAS
Titus Henderson, a prisoner, filed a civil rights lawsuit against COMPAS and other defendants in the U.S. District Court for the Western District of Wisconsin on July 16, 2018. The lawsuit alleges discrimination related to the use of the COMPAS risk assessment tool. The case is ongoing.
- AI system involved
- COMPAS
6 source articles · read the reporting →
Middle schooler beats Edgenuity grading algorithm to get perfect score
A seventh-grade student in the Los Angeles Unified School District received a failing grade on a history assignment graded by Edgenuity's automated scoring algorithm. With help from his mother, a history professor, he reverse-engineered the algorithm by writing a paragraph with relevant keywords and a jumble of words, earning a perfect score. The incident highlights concerns about the accuracy and fairness of automated grading systems in education.
- Company involved
- Los Angeles Unified School District
- AI system involved
- Edgenuity
10 source articles · read the reporting →
LA's VI-SPDAT housing scoring system gives lower priority to Black and Latino people
The Los Angeles Homeless Services Authority uses the VI-SPDAT scoring system to prioritise unhoused people for subsidised permanent housing. An investigation by The Markup found that Black and Latino people consistently receive lower vulnerability scores than White people, leading to lower priority for housing. The agency has acknowledged the racial disparities and is working on a new tool, but continues to use the current system.
- Company involved
- Los Angeles Homeless Services Authority
- AI system involved
- VI-SPDAT
10 source articles · read the reporting →
Wisconsin’s dropout prediction algorithm labels students high risk with racial bias
Wisconsin’s Dropout Early Warning System (DEWS) uses machine learning to predict middle school students' likelihood of graduating on time, relying on factors including race and income. The Markup found the algorithm frequently mislabels students, with a higher false alarm rate for Black and Hispanic students. The Wisconsin Department of Public Instruction has not informed schools of the racial disparities and continues to use the system.
- Company involved
- Wisconsin Department of Public Instruction
- AI system involved
- Dropout Early Warning System (DEWS)
10 source articles · read the reporting →
Serbia's Social Card law could harm marginalized groups
Serbia's Social Card law, which entered into force on 1 March 2022, establishes a centralized government database to assess eligibility for social security support. Amnesty International and seven other rights organizations submitted a legal opinion to the Constitutional Court, alleging that the automated system is an intrusive surveillance system that could discriminate against Roma communities and people with disabilities. The system processes 130 categories of personal data and lacks transparency, potentially leading to errors and denial of benefits.
- Company involved
- Serbian government
- AI system involved
- Social Card system
10 source articles · read the reporting →
Nevada's AI grad score model cuts funding for low-income students
Nevada replaced its income-based school funding formula with a machine learning model developed by Infinite Campus that generates a 'grad score' for each student. The model reduced the number of students eligible for supplemental funding from 288,000 to 63,000, disproportionately affecting low-income students. Critics argue the model lacks transparency and de-emphasizes economic status, potentially leaving many high-poverty schools underfunded.
- Company involved
- Nevada Department of Education
- AI system involved
- Grad score model
7 source articles · read the reporting →
CBSE introduces facial recognition system for students to access digital documents
The Central Board of Secondary Education (CBSE) has introduced a facial recognition system for Class 10 and 12 students to access their digital academic documents. A live image of the student is compared with the photograph on their CBSE admit card and, if the match succeeds, the certificate is emailed to them. The facility is available on Digi Locker for 2020 records and is expected to help foreign students and those unable to open a Digi Locker account.
- Company involved
- Central Board of Secondary Education (CBSE)
- AI system involved
- Facial Recognition System
10 source articles · read the reporting →
Dartmouth Medical School charges 17 students with cheating based on tracking system.
Dartmouth College's Geisel School of Medicine charged 17 students with cheating on remote exams during the pandemic. The charges were based on secret tracking of student activity on the learning management system. Critics say the system is prone to errors and inappropriate for monitoring students.
- Company involved
- Dartmouth College Geisel School of Medicine
10 source articles · read the reporting →
Denmark's Udbetaling Danmark welfare algorithm illegally surveilled millions
Udbetaling Danmark (UDK), Denmark's centralized welfare payment agency, used automated algorithms to check eligibility and detect fraud. The system collected personal data on millions of beneficiaries and their relatives, including non-beneficiaries, in what a think-tank called 'systematic surveillance'. A complaint to the Danish Data Protection Authority led to a ruling that the data collection violated GDPR, and UDK eventually deleted the illegally obtained data. The system also caused erroneous payments, including underpaying 300,000 pensioners by 94 million euros annually.
- Company involved
- Udbetaling Danmark
10 source articles · read the reporting →
DCPS used allegedly false test scores in teacher value-added evaluations
The District of Columbia Public Schools used student test scores from schools under investigation for cheating in value-added calculations for teacher evaluations. More than 200 teachers were terminated based on these evaluations. Teachers can appeal their ratings to the chancellor, but decisions will not be made before the next school year. The school system removes affected scores only when cheating is confirmed.
- Company involved
- District of Columbia Public Schools
- AI system involved
- value-added model
10 source articles · read the reporting →
UnitedHealth used ALERT algorithm to limit mental health coverage, regulators found
ProPublica reports that UnitedHealth Group's Optum subsidiary used the ALERT algorithm to flag mental health patients and therapists as outliers, leading to therapy coverage denials. Regulators in California, Massachusetts and New York alleged this breached the federal Mental Health Parity and Addiction Equity Act, and UnitedHealth agreed to restrict the system in those jurisdictions. The company denies wrongdoing and says its programmes are compliant. Affected patients, including Medicaid enrollees, were left to pay out-of-pocket or go without care.
- Company involved
- UnitedHealth Group
- AI system involved
- ALERT
5 source articles · read the reporting →
NHS liver transplant algorithm causes longer waits for younger patients
An NHS computer algorithm that prioritises liver transplant patients most likely to die soon has led to younger patients waiting 156 days longer on average than those over 60. The algorithm, launched in 2018, gives a score based on 21 recipient and seven donor parameters. Patients like Sarah Meredith, 31, have been waiting two years for a transplant. NHS Blood and Transplant defends the system as saving lives but acknowledges the stress on younger patients.
- Company involved
- NHS Blood and Transplant
10 source articles · read the reporting →
EviCore denied heart catheterization for patient using algorithm
In fall 2021, Little John Cupp's doctor requested a left heart catheterization exam. EviCore, a company hired by UnitedHealthcare, denied the request twice using an algorithm called 'the dial' that adjusts thresholds for review. The algorithm flagged the request for review, and EviCore's doctors determined it was not medically necessary. Cupp did not receive the procedure and his symptoms continued.
- Company involved
- EviCore (a Cigna company)
- AI system involved
- the dial
6 source articles · read the reporting →
IRCC uses AI triage for Temporary Resident Visa applications
Immigration, Refugees and Citizenship Canada (IRCC) uses an AI system called Advanced Analytics to triage Temporary Resident Visa applications from India and China. The system categorizes applications into tiers, with Tier 1 approved automatically and others sent to human officers. Critics allege the system lacks transparency and may introduce bias, leading to visa refusals without clear rationale. The author, a Canadian immigration lawyer, is filing Federal Court cases on behalf of clients affected by refusals.
- Company involved
- Immigration, Refugees and Citizenship Canada (IRCC)
- AI system involved
- Advanced Analytics Triage of Overseas Temporary Resident Visa Applications
10 source articles · read the reporting →
California EDD's automated fraud detection wrongly suspended 600,000 legitimate unemployment claims
In January 2021, the California Employment Development Department used Thompson Reuters automated batch review software to flag 1.1 million unemployment claims as potentially fraudulent. EDD stopped payments on those claims without prior notice. Later, over 600,000 were confirmed as legitimate after claimants used ID.me to verify their identity. The incident highlights the trade-off between fraud prevention and timely benefit access.
- Company involved
- California Employment Development Department (EDD)
- AI system involved
- Thompson Reuters Automated Batch Review
7 source articles · read the reporting →
UNOS liver allocation policy reduces transplants in poorer states
The United Network for Organ Sharing (UNOS) implemented a new liver allocation policy in 2020 called the acuity circles system, which prioritizes the sickest patients regardless of location. An analysis by The Markup and The Washington Post found that the policy led to sharp declines in liver transplants in several Southern and Midwestern states and Puerto Rico, while increasing transplants in New York and California. Patients in affected states allege the system disadvantages poorer regions, and hospitals have sued to overturn the policy.
- Company involved
- United Network for Organ Sharing (UNOS)
- AI system involved
- Acuity circles liver allocation policy
10 source articles · read the reporting →
Swedish welfare agency's AI system flags marginalized groups for fraud investigations
Försäkringskassan, Sweden's Social Insurance Agency, uses an AI risk-scoring system to flag welfare applicants for fraud investigations. The system disproportionately targets women, individuals with foreign backgrounds, low-income earners, and those without university degrees, according to an investigation by Lighthouse Reports and Svenska Dagbladet. Amnesty International has called for the system to be discontinued, citing violations of the right to equality and non-discrimination.
- Company involved
- Försäkringskassan (Swedish Social Insurance Agency)
6 source articles · read the reporting →
Texas uses AI to grade student STAAR test answers
The Texas Education Agency will use an automated scoring engine to grade written answers on the 2023 STAAR tests, replacing thousands of human graders. The system uses natural language processing and will initially score all responses, with a quarter rescored by humans. Educators have expressed concerns about the system's fairness and the potential for errors, especially for creative or non-standard answers.
- Company involved
- Texas Education Agency
- AI system involved
- automated scoring engine
10 source articles · read the reporting →
Woman denied state welfare card groceries after face scan fails
A grandmother rode her motorcycle several kilometres to a shop in Sapphaya district, Chai Nat province, to use her state welfare card to buy groceries for her family. After selecting items, she failed repeated facial recognition scans because her ID card photo was years old and no longer matched her appearance. She left the shop empty-handed and in tears.
- Company involved
- Government of Thailand (state welfare card system)
2 source articles · read the reporting →
DWP algorithm mistakenly flags 200,000 Housing Benefit claimants for fraud review
The UK Department for Work and Pensions (DWP) uses an algorithm to flag Housing Benefit claimants for possible fraud or error. According to a Big Brother Watch investigation, the algorithm has mistakenly flagged 200,000 innocent people, subjecting them to intrusive reviews. Only one in three flagged cases actually had errors, compared to two in three during the pilot. The DWP has spent £4.4 million on these pointless checks.
- Company involved
- Department for Work and Pensions (DWP)
6 source articles · read the reporting →
