CNAF algorithm flags single mother Juliette for welfare fraud investigation
In 2022, Juliette, a single mother on welfare in France, was flagged by CNAF's secretive fraud detection algorithm. A fraud investigator later determined she owed thousands of euros, which were deducted from her monthly payments. The algorithm, which scores half of France's population, is accused of discriminating against vulnerable people by using factors like single parenthood and low income.
- Company involved
- CNAF
10 source articles · read the reporting →
Poland's Ministry of Labor profiles unemployed with discriminatory algorithm
In May 2015, Poland's Ministry of Labor and Social Policy introduced an algorithmic system to profile unemployed people into three categories based on their readiness to work, location, and disabilities. The system is accused of being discriminatory and lacking transparency, as it uses default criteria that may lead to social exclusion. A report by the Panoptykon Foundation in October 2015 highlighted these issues and recommended reforms, including granting the right to appeal.
- Company involved
- Ministry of Labor and Social Policy
10 source articles · read the reporting →
Oregon child welfare agency stops using algorithm to screen families for investigations
The Oregon Department of Human Services announced it will stop using its Safety at Screening Tool algorithm, which helped hotline workers decide which families to investigate for child abuse and neglect. The decision followed concerns about racial bias and disparities, as the algorithm was inspired by a Pennsylvania tool that had flagged a disproportionate number of Black children. The agency will replace the algorithm with a new Structured Decision Making model.
- Company involved
- Oregon Department of Human Services
- AI system involved
- Safety at Screening Tool
10 source articles · read the reporting →
Instacart's rating and fraud detection system unfairly penalizes shoppers
Instacart shoppers allege that the company's automated rating and fraud detection system unfairly penalizes them for issues outside their control, such as customer fraud. The system assigns low ratings and blames shoppers for missing items, leading to reduced pay and deactivation. The Gig Workers Collective has sent an open letter to CEO Fidji Simo demanding changes including higher base pay, item commission, and occupational death benefits.
- Company involved
- Instacart
10 source articles · read the reporting →
WeChat Pay Facial Recognition Bypassed by Scammers Using Animated GIFs
In July 2020, a woman in Hubei, China, reported that 20,000 yuan ($3,000) was stolen from her WeChat Pay account. Scammers had tricked her 8-year-old son into revealing her password, then used animated GIFs of her face to bypass the facial recognition security. Police arrested three suspects, and the case remains under investigation. The incident highlights vulnerabilities in facial recognition systems that can be fooled by simple animated images.
- Company involved
- WeChat
- AI system involved
- WeChat Pay facial recognition
1 source article · read the reporting →
Tenant screening software denies housing to disabled Latino man in Connecticut
In 2016, Carmen Arroyo requested to move her disabled son Mikhail into a larger apartment in the same complex. The landlord, WinnResidential, used CoreLogic's CrimSafe tenant screening software, which flagged a dropped retail theft charge against Mikhail as disqualifying. The landlord rejected the application without explanation. Arroyo sued CoreLogic and WinnResidential under the Fair Credit Reporting Act and the Fair Housing Act, alleging that the software disproportionately excludes people of colour. The case was scheduled for trial in August 2021.
- Company involved
- WinnResidential
- AI system involved
- CrimSafe
10 source articles · read the reporting →
Dutch Tax Authority discriminated against dual nationality applicants for child care benefits using automated risk system.
The Dutch Data Protection Authority (AP) found that the Benefits Department of the Tax and Customs Administration (Belastingdienst) unlawfully processed the dual nationality of 1.4 million applicants for child care benefits. The data was used as an indicator in an automated risk-scoring system that flagged applications as high-risk, which was unnecessary and discriminatory. The AP concluded that the processing violated the GDPR and was discriminatory. The AP completed its fact-finding and will consider imposing a sanction.
- Company involved
- Belastingdienst
10 source articles · read the reporting →
Evolv AI scanner fails to detect knife, student stabbed at Utica school
In October 2022, a student at Proctor High School in Utica, New York, was stabbed with a knife that the Evolv Express AI weapons scanner failed to detect. The school district had installed the system in 13 schools. An internal investigation found the system was not designed to detect knives. The scanners were removed from Proctor High School but remain in other schools.
- Company involved
- Utica City School District
- AI system involved
- Evolv Express
10 source articles · read the reporting →
Just Eat UK couriers deactivated by AI for minor overpayments
Just Eat UK couriers were deactivated from the platform after an AI system flagged them for alleged overpayments as small as £1.35. The system used GPS data to detect drivers moving away from restaurants while waiting for orders. Drivers claim they were asked to move by restaurant staff and that the deactivation was done with little human oversight. A report by Worker Info Exchange details eleven cases, and the company says each case is reviewed by a human before deactivation.
- Company involved
- Just Eat
10 source articles · read the reporting →
Nigeria collects fingerprints and photos of children for school feeding program
The Nigerian government has begun collecting fingerprints and photos of children as young as five for the National Home Grown School Feeding Programme. The biometric enrollment aims to confirm eligibility and increase transparency, but the exercise has started before the country's data protection law comes into effect. The motives for collection are unclear, and there have been technical issues with the enrollment tablets. The program already feeds nine million children daily.
- Company involved
- Nigeria's National Home Grown School Feeding Programme (NHGSFP)
- AI system involved
- HID Global DP4500 biometric scanner and Plovtech enrollment app
5 source articles · read the reporting →
Polish school fined for processing students' fingerprints without legal basis
Primary School No. 2 in Gdansk used a biometric fingerprint reader to identify students for meal payment verification. The system processed the biometric data of 680 children without a valid legal basis, as the school could have used alternative identification methods. The Polish Personal Data Protection Office (UODO) imposed a fine of PLN 20,000 ($5,300) and ordered the erasure of the data. The authority also noted that students without biometric identification were required to wait at the end of the queue, constituting unequal treatment.
- Company involved
- Primary School No. 2 in Gdansk
6 source articles · read the reporting →
Telangana algorithm denied food to thousands of poor families
The Telangana government's algorithmic system Samagra Vedika wrongfully denied subsidised food to thousands of poor families by incorrectly tagging them as ineligible. The system, developed by Posidex Technologies, misidentified individuals across databases, leading to false positives. Affected families, including 67-year-old widow Bismillah Bee, were forced to prove their eligibility, often unsuccessfully. A Supreme Court case ordered re-verification, but many remain excluded.
- Company involved
- Government of Telangana
- AI system involved
- Samagra Vedika
8 source articles · read the reporting →
Illinois DCFS ends unreliable predictive analytics program for child abuse risk
The Illinois Department of Children and Family Services ended its Rapid Safety Feedback program, which used data mining to predict child abuse risk, after the agency's director called the technology unreliable. The system, developed by Eckerd Connects and Mindshare Technology, assigned risk scores to children but failed to flag several high-profile child deaths. The program was also criticised for overwhelming caseworkers with alerts and for potential bias against poor children of colour. DCFS decided not to renew the contract.
- Company involved
- Illinois Department of Children and Family Services
- AI system involved
- Rapid Safety Feedback
10 source articles · read the reporting →
SEC Investigating Evolv Over AI Weapons Detection System
The U.S. Securities and Exchange Commission is investigating Evolv Technology, the maker of an AI-based weapons detection system used in schools and other venues. The system has been accused of producing false positives and exhibiting racial bias. A lawsuit from a school district has also been filed against the company.
- Company involved
- Evolv Technology
- AI system involved
- Evolv AI weapons detection system
5 source articles · read the reporting →
DOJ examines Allegheny County child welfare AI screening tool for disability bias
The Justice Department is scrutinizing the Allegheny Family Screening Tool, an AI system used by Allegheny County, Pennsylvania, to assess risk for families reported for child welfare concerns. A father with an intellectual disability, represented by attorney Robin Frank, filed a complaint alleging the algorithm discriminates against families with disabilities by using disability-related data points. The county denies awareness of concerns and says the tool helps predict risk, but critics argue it hardens bias. The DOJ's Civil Rights Division is investigating.
- Company involved
- Allegheny County Department of Human Services
- AI system involved
- Allegheny Family Screening Tool
5 source articles · read the reporting →
Paraná schools use European facial recognition to mark attendance, risking welfare for false absences
Since 2023, the Paraná state education department in Brazil has deployed facial recognition software from Slovak company Innovatrics in over 1,700 schools to automate attendance for nearly one million children. The system has made errors, marking present students as absent. Because attendance records generated by the algorithm affect eligibility for the Bolsa Família welfare program, false absences could threaten families' benefits. In April 2025, the state's public prosecutor filed a legal challenge alleging the system violates Brazilian law; the case is pending.
- Company involved
- Paraná state education department (Secretaria da Educação do Paraná)
- AI system involved
- LRCO Paraná
8 source articles · read the reporting →
