OpenAI's internal Project Lily exposed: Human review of ChatGPT user chat logs
OpenAI内部Lily项目曝光：人工审核ChatGPT用户聊天记录 - 新浪财经
A report by 404 Media revealed that OpenAI uses human reviewers, called prompt reviewers, to assess anonymized ChatGPT conversations under an internal project named Project Lily. Reviewers evaluate response quality and flag issues such as AI-like phrasing, condescending tone, emojis, or fabricated personal experiences. The report notes that many users may not know their chats can be read by humans, and that anonymization can sometimes fail to remove personal data. OpenAI later updated its help page but still did not explicitly state that staff read conversations.
- Company involved
- OpenAI
- AI system involved
- ChatGPT
1 source article · read the reporting →
Fired Teacher Says AI Tainted Review And Arbitration - law360.com
The AI-generated performance evaluation contributed to the teacher's termination.
- Company involved
- Concord Public Schools
1 source article · read the reporting →
SafeRent Settles Class Action Over Tenant Screening for Voucher Holders
Mary Louis and Monica Douglas filed a class action lawsuit in 2022 alleging that SafeRent Solutions' SafeRent Score product discriminated against rental applicants in Massachusetts who held public housing vouchers, violating fair housing and consumer protection laws. SafeRent denied wrongdoing. The court approved a settlement in November 2024, and payments were distributed to eligible class members in 2025 and 2026.
- Company involved
- SafeRent Solutions, LLC
- AI system involved
- SafeRent Score
5 source articles · read the reporting →
TRT-RS's Galileu AI Detects Prompt Injection Attempt in Legal Petition
The Galileu AI system, developed by the Tribunal Regional do Trabalho da 4ª Região (TRT-RS) and nationalised by the Conselho Superior da Justiça do Trabalho (CSJT), detected a prompt injection attempt in a petition filed at the 3rd Labour Court of Parauapebas, Pará. The system alerted the magistrate, who reviewed the content and made a decision based on human verification, in line with judicial AI supervision requirements. The court reported that the system prevented the malicious content from being processed and highlighted the importance of institutional AI tools with security measures.
- Company involved
- Tribunal Regional do Trabalho da 4ª Região
- AI system involved
- Galileu
1 source article · read the reporting →
Rotterdam's Welfare Fraud Algorithm Discriminated by Gender and Ethnicity
The city of Rotterdam deployed a machine learning algorithm built by Accenture to flag welfare recipients for fraud investigation. The system used personal data including gender, language, and subjective caseworker notes to generate risk scores, leading to investigations that disproportionately targeted women and migrants. An external review found the algorithm discriminatory and inaccurate, prompting the city to suspend its use in 2021. The system's opacity made it nearly impossible for those flagged to challenge the decisions.
- Company involved
- City of Rotterdam
6 source articles · read the reporting →
Legal challenge against Met Police's discriminatory Gangs Matrix
The Metropolitan Police's Gangs Violence Matrix is a secretive database that disproportionately affects Black individuals. Liberty and the Civil Liberties Trust launched a judicial review on behalf of Awate Suleiman and UNJUST, alleging the Matrix discriminates on racial grounds and breaches human rights and data protection laws. The case resulted in the Met agreeing to overhaul the Matrix.
- Company involved
- Metropolitan Police
- AI system involved
- Gangs Violence Matrix
9 source articles · read the reporting →
Schufa's Black-Box Scoring Unfairly Penalises Consumers with Positive Credit Data
An investigation by SPIEGEL and BR Data reveals that Schufa's credit scoring algorithm often assigns poor risk scores to consumers with only positive credit information. One consumer, Sven Drewert, was denied a credit card limit increase despite having no negative entries. The algorithm uses limited data, and its secret formula can lead to arbitrary categorisations, affecting access to loans, phone contracts, and housing. The system's opacity and potential biases raise concerns about fairness and accountability.
- Company involved
- Schufa Holding AG
- AI system involved
- Schufa Score
2 source articles · read the reporting →
DOJ's Pattern Algorithm Shows Racial Disparities in Early Release Decisions
The U.S. Department of Justice's Pattern risk assessment algorithm, used to determine federal prisoners' eligibility for early release under the First Step Act, was found to produce racial disparities. A December 2021 DOJ report revealed that the tool overpredicted recidivism risk for Black, Hispanic, and Asian inmates, with only 7% of Black prisoners classified as minimum risk compared to 21% of white prisoners. Civil rights groups have called for the suspension of the tool, while the DOJ is working on an overhaul and reevaluating 14,000 prisoners who were misclassified.
- Company involved
- U.S. Department of Justice
- AI system involved
- Pattern
1 source article · read the reporting →
Australian Research Council faces allegations of ChatGPT use in peer review
The Australian Research Council is facing allegations that some peer reviewers used ChatGPT to write assessor reports for Discovery Project grant proposals. Researchers reported generic wording and even the phrase 'Regenerate response' in feedback, suggesting AI generation. One researcher's complaint led to the removal of the report, and the education minister called the use unacceptable, instructing the ARC to prevent it. The ARC stated that peer reviewers should not use AI and that confidentiality policies apply.
- Company involved
- Australian Research Council
- AI system involved
- ChatGPT
2 source articles · read the reporting →
New York City Department of Education Releases Flawed Teacher Ratings Publicly
In February 2012, the New York City Department of Education publicly released individual Teacher Data Reports rating over 12,000 teachers based on value-added analysis of student test scores. The ratings, originally intended for internal use, were disclosed after a court ruled in favor of media organizations under the Freedom of Information Act. The teachers' union and education experts criticised the data as unreliable due to large margins of error and failure to account for demographic factors, warning that the release would unfairly shame or praise educators. The Department defended the ratings as a useful perspective on teacher effectiveness.
- Company involved
- New York City Department of Education
- AI system involved
- Teacher Data Reports
6 source articles · read the reporting →
Royal Free London publishes audit into Streams app data processing
The Royal Free London NHS Foundation Trust published an audit into its use of the Streams app, following an investigation by the Information Commissioner's Office (ICO) in July 2017. The Streams app alerts clinicians to patients at risk of acute kidney injury. The audit, conducted by Linklaters, concluded that the trust's use of Streams was lawful and complied with data protection laws, although areas for improvement were identified. The ICO later recognised that the trust had completed all required actions.
- Company involved
- Royal Free London NHS Foundation Trust
- AI system involved
- Streams
10 source articles · read the reporting →
Teachers find no value in SAS EVAAS system in Southwest School District
A study examined the SAS Education Value-Added Assessment System (EVAAS®) used by the Southwest School District (SSD) to evaluate teacher effectiveness for high-stakes consequences. Teachers reported that the system produced inconsistent results, was biased by student factors, and reduced morale and collaboration. The study found that teachers did not use the data formatively as intended, and unintended consequences included heightened pressure and teaching to the test.
- Company involved
- Southwest School District (SSD)
- AI system involved
- SAS Education Value-Added Assessment System (EVAAS®)
10 source articles · read the reporting →
Anonymous Spanish Lawyer (Tribunal Constitucional): AI-hallucinated content in court filing, Formal Reprimand (Apercibimiento) + Referral to Barcelona Bar for Disc
The AI system generated hallucinated content that was submitted in a court filing, potentially misleading the court.
1 source article · read the reporting →
Mag 7 Ltd. et al. v. Tederi et al. (District Court, Tel Aviv-Yafo): AI-hallucinated content in court filing, Monetary Sanction;…
The AI generated fabricated case law that was submitted to the court, affecting the legal proceedings.
1 source article · read the reporting →
UDR Sued Over Alleged Use of RealPage Algorithm to Set Rents in San Diego
A class action lawsuit alleges that UDR, Inc. used RealPage's YieldStar algorithmic pricing software to set rental rates and occupancy levels for its San Diego properties, in violation of a local ordinance. The suit claims the software relied on nonpublic competitor data, contributing to inflated rents and making housing less affordable. The lawsuit, filed in July 2026, seeks to represent all affected tenants. UDR has previously acknowledged using YieldStar as one tool among others in its decision-making.
- Company involved
- UDR, Inc.
- AI system involved
- RealPage YieldStar
1 source article · read the reporting →
Joel A. Rivera v. Triad Properties Corporation, et al. (N.D. Alabama): AI-hallucinated content in court filing, Public Reprimand; Disqualification; Bar…
The AI generated fabricated legal citations that were presented to the court, affecting the defendants and the judicial process.
- Company involved
- Burrill Watkins LLC
- AI system involved
- ChatGPT
1 source article · read the reporting →
SQA exam algorithm disproportionately downgraded poorer pupils, report finds
The Scottish Qualifications Authority (SQA) used an algorithm to moderate 2020 exam results after COVID-19 cancelled exams. The algorithm disproportionately downgraded students from disadvantaged backgrounds, leading to a public outcry and a government U-turn. An independent report criticised the SQA and Scottish Government for failing to properly assess the equality impacts and for not making the algorithm available for analysis. The SQA has expressed no regret over the moderation approach.
- Company involved
- Scottish Qualifications Authority (SQA)
10 source articles · read the reporting →
Proctorio's exam proctoring system allows human agents to review student feeds despite privacy claims
A security researcher analyzed Proctorio's Chrome extension and found evidence that the system allows human agents to review students' room scans and live ID checks, contrary to Proctorio's claims that only professors can access recordings. The system flags students based on behavioral metrics and can terminate exams. The researcher also raised concerns about discrimination against low-income students and privacy violations.
- Company involved
- Proctorio
- AI system involved
- Proctorio
10 source articles · read the reporting →
LA's VI-SPDAT housing scoring system gives lower priority to Black and Latino people
The Los Angeles Homeless Services Authority uses the VI-SPDAT scoring system to prioritise unhoused people for subsidised permanent housing. An investigation by The Markup found that Black and Latino people consistently receive lower vulnerability scores than White people, leading to lower priority for housing. The agency has acknowledged the racial disparities and is working on a new tool, but continues to use the current system.
- Company involved
- Los Angeles Homeless Services Authority
- AI system involved
- VI-SPDAT
10 source articles · read the reporting →
Ramirez v. Humala (E.D. New York): AI-hallucinated content in court filing, Monetary sanction jointly imposed on counsel and firm; order…
The AI generated nonexistent case citations that were filed in court, misleading the court and opposing counsel.
1 source article · read the reporting →
FTC settles with Sitejabber over reviews from customers who had not received goods
The US Federal Trade Commission has charged Sitejabber, an AI-enabled review platform, with deceiving consumers by publishing ratings and reviews submitted at the point of purchase, before buyers had received or experienced the products. The FTC alleges this artificially inflated average ratings and review counts for its clients. Sitejabber has agreed to a proposed consent order prohibiting it from making such misrepresentations in the future.
- Company involved
- Sitejabber
- AI system involved
- Sitejabber
10 source articles · read the reporting →
Recurso de Suplicación 0005472/2025 (T.S.X. Galicia): AI-hallucinated content in court filing, Bar Referral
The AI generated false legal citations that were included in a court filing, misleading the court and potentially harming the client's case.
1 source article · read the reporting →
Aguilar v. The Crawford Group, Inc. (D. Massachusetts): AI-hallucinated content in court filing, Adverse Costs Order; Pro hac vice status…
The AI generated fake legal citations that were included in a court filing, misleading the court and opposing counsel.
- Company involved
- Lindemann Law Firm
1 source article · read the reporting →
Barteca Holdings v. Tacobarn (D. Connecticut): AI-hallucinated content in court filing, Monetary Sanction; Bar Referral
The attorney filed a court document containing AI-hallucinated legal authorities, misleading the court and opposing party.
- Company involved
- Hilary Miller
1 source article · read the reporting →
