Where automated decisions went wrong

Fired Teacher Says AI Tainted Review And Arbitration - law360.com

The AI-generated performance evaluation contributed to the teacher's termination.

Company involved

Concord Public Schools

1 source article · read the reporting →

15-Year-Old Test Exposes Flaw: ChatGPT for Teens Fails to Block Homework Cheating, Repeated Requests Bypass Restrictions

15歲使用者實測揭漏洞：青少年版ChatGPT難擋宿題代寫，反覆要求即破解 - BigGo 財經

A 15-year-old tester found that OpenAI's ChatGPT for Teens, launched in August, initially refused to write essays but generated full examples after repeated requests. It also immediately solved SAT-level math problems. Parental controls require account linking and are off by default, while experts warn the memory feature could lead to emotional attachment.

Company involved

OpenAI

AI system involved

ChatGPT for Teens

1 source article · read the reporting →

Study Finds COMPAS Sentencing Algorithm No More Accurate Than Random People, Shows Racial Bias

A study by Dartmouth College researchers found that the COMPAS recidivism risk assessment algorithm, used by judges in several US states, is no more accurate than random people recruited online. The algorithm, developed by Equivalent, exhibited racial bias by classifying black defendants as higher risk and white defendants as lower risk, leading to harsher rehabilitation recommendations for black defendants. The bias likely stems from skewed arrest rate data. The findings build on a 2016 ProPublica investigation.

Company involved

Broward County courts

AI system involved

COMPAS

10 source articles · read the reporting →

New York teacher challenges state evaluation system over flawed rating

Sheri Lederman, a 4th-grade teacher in the Great Neck district, filed a lawsuit against the New York state education department after being rated 'ineffective' on the portion of her evaluation tied to students' test performance, a year after receiving a 'highly effective' score. The lawsuit alleges that the teacher-evaluation system is statistically flawed and lacks a proper appeals process. The education department declined to comment on the litigation.

Company involved

New York state education department

10 source articles · read the reporting →

Meazure Learning Agrees $1.35m California Bar Exam Class-Action Settlement - Lawyer Monthly

The exam software experienced technical failures that prevented candidates from logging in or completing the California bar exam.

Company involved

Meazure Learning

AI system involved

ProctorU

1 source article · read the reporting →

University of Illinois Ends Proctorio Contract Over Accessibility Concerns

The University of Illinois at Urbana-Champaign announced it will not renew its emergency contract with Proctorio, an online proctoring software, citing significant accessibility, privacy, and equity concerns. Students reported that the AI system unfairly flagged individuals with disabilities, nonwhite students, and those wearing headscarves as suspicious during exams. The decision follows student petitions and campaigns against the software, which they described as invasive spyware. The university is exploring alternative assessment methods.

Company involved

University of Illinois at Urbana-Champaign

AI system involved

Proctorio

6 source articles · read the reporting →

Sydney students resit NAPLAN after predictive text glitch

Year 5 students at Waverley College and Kambala in Sydney were required to resit their NAPLAN writing exams after a technical glitch enabled Apple's predictive text and spellcheck features during the test. The issue was discovered by a teacher and students, and the schools reported it to NESA and ACARA. The students were given a new prompt and completed the test the following Monday. An ACARA spokesman stated the problem was not widespread, with over 880,000 tests submitted successfully.

Company involved

ACARA

AI system involved

NAPLAN

2 source articles · read the reporting →

Moltbook Database Hacked, Exposing Thousands of Emails and Private Messages

Security researchers at Wiz hacked Moltbook's database in under three minutes due to a backend misconfiguration, gaining access to 35,000 email addresses, thousands of private direct messages, and 1.5 million API tokens. The vulnerability could have allowed attackers to impersonate AI agents and manipulate content. Wiz disclosed the issue to Moltbook, which secured the database within hours, and all accessed data was deleted.

Company involved

Moltbook

AI system involved

Moltbook

1 source article · read the reporting →

Legal challenge against Met Police's discriminatory Gangs Matrix

The Metropolitan Police's Gangs Violence Matrix is a secretive database that disproportionately affects Black individuals. Liberty and the Civil Liberties Trust launched a judicial review on behalf of Awate Suleiman and UNJUST, alleging the Matrix discriminates on racial grounds and breaches human rights and data protection laws. The case resulted in the Met agreeing to overhaul the Matrix.

Company involved

Metropolitan Police

AI system involved

Gangs Violence Matrix

9 source articles · read the reporting →

AI detector falsely accuses student of cheating at Central Methodist University

Moira Olmsted, a student at Central Methodist University, submitted a written assignment and received a zero grade after her professor said an AI detection tool flagged her work as AI-generated. Olmsted alleges the accusation was false and that her writing had been flagged at least once before. The incident highlights the risk of AI detectors falsely accusing students of cheating.

Company involved

Central Methodist University

1 source article · read the reporting →

Universities' use of race in student risk algorithms harms Black and Latinx students

The Markup found that several universities, including UMass Amherst, UW–Milwaukee, and Texas A&M, use EAB's Navigate software which incorporates race as a predictor of student dropout risk. The algorithms label Black and Latinx students as high risk at disproportionately high rates compared to White students. Students, professors, and experts worry this is pushing minority students out of math and science majors through biased advising.

Company involved

University of Massachusetts Amherst,University of Wisconsin–Milwaukee,Texas A&M University,Texas Tech University,South Dakota State University

AI system involved

Navigate

1 source article · read the reporting →

LAUSD shelves AI chatbot after vendor AllHere collapses

Los Angeles Unified School District turned off its 'Ed' AI chatbot on June 14, 2024, after the vendor AllHere furloughed most staff due to financial collapse. The chatbot, which cost $3 million, was designed to provide students and parents with academic guidance and school information. A former AllHere employee alleged that student data was improperly shared with third parties and processed overseas, raising privacy concerns. The district stated it will ensure privacy protections and plans to eventually restore the chatbot.

Company involved

Los Angeles Unified School District

AI system involved

Ed

5 source articles · read the reporting →

Parents sue Hingham Public Schools over AI project D grade

The parents of a Hingham High School senior allege the school unfairly punished him for using AI on a social studies project, giving him a D and Saturday detention despite no handbook rule against AI. They say the grade kept him out of the National Honor Society and threatens his applications to top-tier colleges. The family sued Hingham High School administration and the school district in Plymouth County District Court, seeking a grade change and NHS admission. Hingham Public Schools has declined to comment, citing ongoing litigation.

Company involved

Hingham Public Schools

2 source articles · read the reporting →

Microsoft Copilot Audit Log Flaw Left Customers Unaware

A vulnerability in Microsoft 365 Copilot allowed users to access files without the access being recorded in audit logs, potentially enabling malicious insiders to exfiltrate data undetected. The flaw, discovered by Pistachio's CTO, was reported to Microsoft in July 2025 and fixed in August, but Microsoft decided not to issue a CVE or notify customers. The vulnerability could be triggered accidentally, meaning many organisations' audit logs may be incomplete. Microsoft classified the issue as 'important' but faced criticism for its lack of transparency.

Company involved

Microsoft

AI system involved

M365 Copilot

1 source article · read the reporting →

New York City Department of Education Releases Flawed Teacher Ratings Publicly

In February 2012, the New York City Department of Education publicly released individual Teacher Data Reports rating over 12,000 teachers based on value-added analysis of student test scores. The ratings, originally intended for internal use, were disclosed after a court ruled in favor of media organizations under the Freedom of Information Act. The teachers' union and education experts criticised the data as unreliable due to large margins of error and failure to account for demographic factors, warning that the release would unfairly shame or praise educators. The Department defended the ratings as a useful perspective on teacher effectiveness.

Company involved

New York City Department of Education

AI system involved

Teacher Data Reports

6 source articles · read the reporting →

Houston teachers sue over secret EVAAS evaluation algorithm

Houston Independent School District used the SAS Institute's EVAAS system, a proprietary algorithm, to evaluate teacher performance based on student test scores. The Houston Federation of Teachers and several teachers sued in 2014, alleging the system violated their Fourteenth Amendment procedural due process rights because the algorithm is a trade secret and teachers cannot verify the accuracy of their scores. A federal magistrate judge refused to dismiss the procedural due process claims in May 2017, ruling that teachers must have an opportunity to test the accuracy of the scores that could cost them their jobs.

Company involved

Houston Independent School District

AI system involved

Educational Value-Added Assessment System (EVAAS)

1 source article · read the reporting →

Teachers find no value in SAS EVAAS system in Southwest School District

A study examined the SAS Education Value-Added Assessment System (EVAAS®) used by the Southwest School District (SSD) to evaluate teacher effectiveness for high-stakes consequences. Teachers reported that the system produced inconsistent results, was biased by student factors, and reduced morale and collaboration. The study found that teachers did not use the data formatively as intended, and unintended consequences included heightened pressure and teaching to the test.

Company involved

Southwest School District (SSD)

AI system involved

SAS Education Value-Added Assessment System (EVAAS®)

10 source articles · read the reporting →

Texas A&M Professor Uses ChatGPT to Falsely Accuse Students of Cheating

Jared Mumm, an instructor at Texas A&M University–Commerce, used ChatGPT to check if students' writing was AI-generated, leading him to accuse them of using the chatbot. He informed students they would receive an incomplete grade and those deemed guilty would get a zero, causing distress and temporarily withholding one student's diploma. The university investigated and stated that no students failed or were barred from graduating, and the professor is working individually with students to resolve the issue.

Company involved

Texas A&M University–Commerce

AI system involved

ChatGPT

5 source articles · read the reporting →

Australian education ministers scrap NAPLAN robot marking

The Education Council decided not to use automated essay scoring for NAPLAN writing tests, dropping a proposal by ACARA to have computers mark English tasks. Teachers' unions had campaigned against the plan, and an academic warned that the software would reward verbose gibberish and could not assess creativity. The decision was made in December 2017 and announced in January 2018.

Company involved

Australian Curriculum, Assessment and Reporting Authority (ACARA)

10 source articles · read the reporting →

Hangzhou No. 11 Middle School installs facial recognition system to monitor students

Hangzhou No. 11 Middle School has deployed a smart classroom behaviour management system, developed with Hikvision, that uses facial recognition to take attendance and analyse students' expressions and behaviour. The system alerts teachers when students appear inattentive. The school says it records only behavioural state information, not video, and that it will be installed in all classes. Some people have raised privacy concerns.

Company involved

杭州第十一中学

AI system involved

智慧课堂行为管理系统

10 source articles · read the reporting →

341 Malicious ClawHub Skills Found Stealing OpenClaw User Data

Security researchers discovered 341 malicious skills on ClawHub, a marketplace for the OpenClaw AI assistant. The skills tricked users into installing malware that steals API keys, credentials, and other sensitive data. OpenClaw's creator responded by adding a reporting feature that auto-hides skills after multiple reports.

Company involved

OpenClaw

AI system involved

OpenClaw

4 source articles · read the reporting →

Beijing school uses Hanwang emotion recognition system to monitor student behaviour

Niulanshan First Secondary School in Beijing has installed Hanwang Technology's Classroom Care System, which photographs students every second and analyses their facial expressions and behaviour. The system sends weekly reports to teachers and parents, and one student was described as low participation in English class despite being recorded as focused 94% of the time. The school has been highlighted in a report by Article 19, which says such emotion recognition tools rely on junk science and risk eroding privacy and human rights.

Company involved

Niulanshan First Secondary School

AI system involved

Classroom Care System

10 source articles · read the reporting →

NSW Education Standards Authority used AI-generated image in HSC English exam without disclosure

The NSW Education Standards Authority (NESA) used an AI-generated image as a stimulus in the 2024 HSC English exam without disclosing its origin. The image, created by Florian Schroeder using OpenAI's ChatGPT and Dall-E 2, was published on Medium in July 2023. Students suspected AI use due to irregularities in the image, and NESA initially declined to confirm. After the Sydney Morning Herald confirmed the image was AI-generated, NESA stated that students would be marked on their response to the question, not the image's origin.

Company involved

NSW Education Standards Authority

AI system involved

ChatGPT and Dall-E 2

6 source articles · read the reporting →

Titus Henderson files civil rights lawsuit against COMPAS

Titus Henderson, a prisoner, filed a civil rights lawsuit against COMPAS and other defendants in the U.S. District Court for the Western District of Wisconsin on July 16, 2018. The lawsuit alleges discrimination related to the use of the COMPAS risk assessment tool. The case is ongoing.

AI system involved

COMPAS

6 source articles · read the reporting →