AI detector scores banned as evidence at Yale and Johns Hopkins, vendor conflict exposed - Pasquale Pillitteri
The system flagged student assignments as AI-generated, potentially leading to academic misconduct accusations.
- Company involved
- Yale University and Johns Hopkins University
- AI system involved
- Turnitin AI detector
1 source article · read the reporting →
Study Finds COMPAS Sentencing Algorithm No More Accurate Than Random People, Shows Racial Bias
A study by Dartmouth College researchers found that the COMPAS recidivism risk assessment algorithm, used by judges in several US states, is no more accurate than random people recruited online. The algorithm, developed by Equivalent, exhibited racial bias by classifying black defendants as higher risk and white defendants as lower risk, leading to harsher rehabilitation recommendations for black defendants. The bias likely stems from skewed arrest rate data. The findings build on a 2016 ProPublica investigation.
- Company involved
- Broward County courts
- AI system involved
- COMPAS
10 source articles · read the reporting →
Class action claims Checkr misreported criminal records in background checks
Checkr's system misreported criminal records belonging to someone else on Natasha Davis's consumer report, impeding her ability to gain employment.
- Company involved
- Checkr Inc.
1 source article · read the reporting →
SafeRent Settles Class Action Over Tenant Screening for Voucher Holders
Mary Louis and Monica Douglas filed a class action lawsuit in 2022 alleging that SafeRent Solutions' SafeRent Score product discriminated against rental applicants in Massachusetts who held public housing vouchers, violating fair housing and consumer protection laws. SafeRent denied wrongdoing. The court approved a settlement in November 2024, and payments were distributed to eligible class members in 2025 and 2026.
- Company involved
- SafeRent Solutions, LLC
- AI system involved
- SafeRent Score
5 source articles · read the reporting →
Chicago Police's Secretive Strategic Subject List Disproportionately Scores Black Men
The Chicago Police Department has used the Strategic Subject List since 2012, an algorithm that assigns risk scores to individuals based on arrest records. The list was kept secret until a court forced its release, revealing that 56% of black men aged 20-29 have a score, and only 3.6% of those on the list were involved in violence. Despite claims it is not used for enforcement, documents show it influences police deployment and arrests, leading to allegations of racial bias and lack of transparency.
- Company involved
- Chicago Police Department
- AI system involved
- Strategic Subject List
9 source articles · read the reporting →
University of Texas at Austin Stops Using Biased Ph.D. Admissions Algorithm
The University of Texas at Austin's computer science department used a machine-learning system called GRADE from 2013 to 2020 to score Ph.D. applicants. Critics argued the system perpetuated bias by training on past admissions decisions and using features like institution prestige and biased language in recommendation letters. The department discontinued the system in 2020, citing maintenance difficulties, but the controversy highlighted concerns about algorithmic fairness in admissions. The system was never disclosed to applicants, and its impact on underrepresented groups remains unknown.
- Company involved
- University of Texas at Austin
- AI system involved
- GRADE
2 source articles · read the reporting →
California State Bar Flags Over 3,000 Online Bar Exam Takers for Possible Cheating
In October 2020, the California State Bar administered its first online bar exam using ExamSoft's AI-enabled proctoring software. The software flagged 3,190 out of 9,301 test takers for potential rule violations such as gazing off-screen or having food. The state bar is reviewing each case, which could require affected individuals to retake the exam, delaying their ability to practice law. Test takers have expressed distress and hired attorneys to contest the allegations.
- Company involved
- California State Bar
- AI system involved
- ExamSoft
2 source articles · read the reporting →
Schufa's Black-Box Scoring Unfairly Penalises Consumers with Positive Credit Data
An investigation by SPIEGEL and BR Data reveals that Schufa's credit scoring algorithm often assigns poor risk scores to consumers with only positive credit information. One consumer, Sven Drewert, was denied a credit card limit increase despite having no negative entries. The algorithm uses limited data, and its secret formula can lead to arbitrary categorisations, affecting access to loans, phone contracts, and housing. The system's opacity and potential biases raise concerns about fairness and accountability.
- Company involved
- Schufa Holding AG
- AI system involved
- Schufa Score
2 source articles · read the reporting →
Ofqual algorithm downgrades 39% of A-level results, sparking crisis
In August 2020, the UK exam regulator Ofqual used an algorithm to determine A-level grades after exams were cancelled due to COVID-19. The algorithm downgraded approximately 39% of results, disproportionately affecting disadvantaged students from state schools. Following protests and accusations of discrimination, the government reversed the decision and used teacher predictions instead. The incident caused university admissions chaos and prompted a review by the Office for Statistics Regulation.
- Company involved
- Ofqual
8 source articles · read the reporting →
DOJ's Pattern Algorithm Shows Racial Disparities in Early Release Decisions
The U.S. Department of Justice's Pattern risk assessment algorithm, used to determine federal prisoners' eligibility for early release under the First Step Act, was found to produce racial disparities. A December 2021 DOJ report revealed that the tool overpredicted recidivism risk for Black, Hispanic, and Asian inmates, with only 7% of Black prisoners classified as minimum risk compared to 21% of white prisoners. Civil rights groups have called for the suspension of the tool, while the DOJ is working on an overhaul and reevaluating 14,000 prisoners who were misclassified.
- Company involved
- U.S. Department of Justice
- AI system involved
- Pattern
1 source article · read the reporting →
International Baccalaureate's Secret Algorithm Downgrades Students, Jeopardising College Admissions
In July 2020, the International Baccalaureate (IB) program used a statistical model to predict final grades for over 170,000 students after cancelling exams due to Covid-19. Many students, including Anahita Nagpal, received lower-than-expected scores, causing them to lose college places and scholarships. Students, parents, and teachers questioned the fairness and transparency of the algorithm, with over 15,000 signing a petition demanding a fairer approach. IB defended the model but offered an appeals process, while some students planned to retake exams at their own expense.
- Company involved
- International Baccalaureate
1 source article · read the reporting →
Arity collected drivers' data via apps for insurance scores
Popular smartphone apps including Life360, MyRadar and GasBuddy reportedly shared users' location and motion data with Arity, an Allstate-owned company. Arity used the data to calculate driving scores that could be sold to car insurers to set rates. Users were said not to be clearly informed that their data would be used for insurance pricing. Life360 and Arity stated that users had to opt in and that no personally identifiable driving data was shared without consent.
- Company involved
- Arity
- AI system involved
- Arity IQ network
2 source articles · read the reporting →
Universities' use of race in student risk algorithms harms Black and Latinx students
The Markup found that several universities, including UMass Amherst, UW–Milwaukee, and Texas A&M, use EAB's Navigate software which incorporates race as a predictor of student dropout risk. The algorithms label Black and Latinx students as high risk at disproportionately high rates compared to White students. Students, professors, and experts worry this is pushing minority students out of math and science majors through biased advising.
- Company involved
- University of Massachusetts Amherst,University of Wisconsin–Milwaukee,Texas A&M University,Texas Tech University,South Dakota State University
- AI system involved
- Navigate
1 source article · read the reporting →
Australian terrorism prediction tool Vera-2R flagged autism as criminality risk factor
An independent report found that the Australian government's Vera-2R terrorism risk assessment tool included autism spectrum disorders as a risk factor without empirical evidence. The tool was used to assess terrorist offenders, influencing post-sentence orders including ongoing detention. Despite the critical report being received in May 2020, the federal and NSW governments continued to use the tool. The report concluded the tool was 'extremely poor' at predicting risk.
- Company involved
- Department of Home Affairs
- AI system involved
- Vera-2R
1 source article · read the reporting →
New York City Department of Education Releases Flawed Teacher Ratings Publicly
In February 2012, the New York City Department of Education publicly released individual Teacher Data Reports rating over 12,000 teachers based on value-added analysis of student test scores. The ratings, originally intended for internal use, were disclosed after a court ruled in favor of media organizations under the Freedom of Information Act. The teachers' union and education experts criticised the data as unreliable due to large margins of error and failure to account for demographic factors, warning that the release would unfairly shame or praise educators. The Department defended the ratings as a useful perspective on teacher effectiveness.
- Company involved
- New York City Department of Education
- AI system involved
- Teacher Data Reports
6 source articles · read the reporting →
Durham Police uses Experian Mosaic data in HART AI risk tool
Durham Constabulary developed the Harm Assessment Risk Tool (HART), a machine learning algorithm that assesses the recidivism risk of offenders. The tool uses 34 data categories including criminal history, age, gender and two types of postcode, one sourced from Experian's Mosaic marketing segmentation system. Big Brother Watch alleges that using such commercial consumer behaviour data to inform custody decisions risks prejudice and disproportionate targeting of deprived neighbourhoods. The force has stated it is refreshing the model with an aim to remove one of the postcode predictors.
- Company involved
- Durham Constabulary
- AI system involved
- Harm Assessment Risk Tool (HART)
9 source articles · read the reporting →
France: CNAF's discriminatory risk-scoring algorithm must be stopped
Amnesty International and coalition partners filed a complaint with the Council of State against CNAF's risk-scoring algorithm used to detect benefit overpayments. The algorithm assigns risk scores based on criteria that discriminate against vulnerable groups, including those with disabilities, single parents, and low-income households. The complaint alleges the system violates human rights to equality and privacy. The EU AI Act's social scoring ban may apply, but its definition remains unclear.
- Company involved
- CNAF
2 source articles · read the reporting →
AMS algorithm lacks transparency and may discriminate against job seekers
The Austrian Public Employment Service (AMS) uses an algorithm to classify job seekers into categories A, B, and C, determining their access to benefits and training. Scientists from TU Wien, WU Wien, and University of Vienna have criticised the algorithm for lacking transparency, as only two of 96 model variants have been published. They allege that the system may discriminate against women and people with migration background, and that job seekers are not informed about how the algorithm works or given a chance to appeal.
- Company involved
- AMS (Arbeitsmarktservice Österreich)
- AI system involved
- AMS-Algorithmus
10 source articles · read the reporting →
Educational Testing Service's E-rater algorithm biases essay scores against minority students
The Educational Testing Service's E-rater algorithm, used to grade essays on the GRE and other standardized tests, has been found to systematically give higher scores to students from mainland China and lower scores to African American students compared to human graders. The bias stems from the algorithm's reliance on surface-level metrics like vocabulary and sentence length, which disadvantage certain groups. Despite studies dating back to 1999, the bias persists, and in many states, only a small percentage of essays are reviewed by humans.
- Company involved
- Educational Testing Service
- AI system involved
- E-rater
10 source articles · read the reporting →
Austrian court lifts ban on AMS job-chance prediction algorithm
The Austrian Federal Administrative Court overturned a ban by the Data Protection Authority on the AMS algorithm, which predicts job chances of unemployed people. The system, trained on historical data including age, gender, and nationality, categorises individuals as high, medium or low chance of re-employment. Critics argue it discriminates against women and mothers, and that the data is now outdated due to the COVID-19 pandemic. The court ruled that the algorithm is lawful as long as a human advisor makes the final decision on training support.
- Company involved
- Arbeitsmarktservice (AMS)
- AI system involved
- AMS-Algorithmus
5 source articles · read the reporting →
UK councils use Covid OneView AI to harvest personal data for risk scoring
UK local authorities are using a system called Covid OneView, developed by data analytics firm Xantura, to harvest millions of personal details from council records. The system uses predictive analytics and AI to assign risk scores to households and individuals, aiming to identify those vulnerable to Covid or likely to break lockdown rules. Privacy campaigners and MPs have criticised the lack of transparency and the extent of data collection, which includes sensitive information such as debt levels, living arrangements, and even notes on unfaithful sex. Xantura and Barking and Dagenham Council have defended the system as compliant with data protection rules and focused on providing support.
- Company involved
- UK local authorities
- AI system involved
- Covid OneView
6 source articles · read the reporting →
Suzhou clarifies civility scoring system after public backlash
The Suzhou municipal government announced a civility code system that assigns scores to residents based on their behavior, including traffic violations and volunteer work. The system, currently in a trial phase, aims to generate a personal portrait for each resident. Following public backlash and concerns about privacy and potential abuse, an official stated that participation will not be mandatory and that a bad score will not affect citizens. The system is still under development.
- Company involved
- Suzhou municipal government
- AI system involved
- Civility code
10 source articles · read the reporting →
Hangzhou plans permanent health codes with scoring, sparking privacy backlash
Hangzhou health authorities announced plans to launch a permanent health-tracking QR code system that would assign residents a color and numerical score based on medical records and lifestyle choices. The proposal, an expansion of the existing COVID-19 health code system, has sparked online backlash over privacy concerns and fears of discrimination. Critics argue that the system could lead to public exposure of personal health data and potential misuse by employers. The authorities aim to complete the project by May or June 2020.
- Company involved
- Hangzhou Health Authorities
- AI system involved
- Health Code
10 source articles · read the reporting →
SQA exam algorithm disproportionately downgraded poorer pupils, report finds
The Scottish Qualifications Authority (SQA) used an algorithm to moderate 2020 exam results after COVID-19 cancelled exams. The algorithm disproportionately downgraded students from disadvantaged backgrounds, leading to a public outcry and a government U-turn. An independent report criticised the SQA and Scottish Government for failing to properly assess the equality impacts and for not making the algorithm available for analysis. The SQA has expressed no regret over the moderation approach.
- Company involved
- Scottish Qualifications Authority (SQA)
10 source articles · read the reporting →