Racial bias in lung test software led to underdiagnosis of Black men, study suggests
A study published in JAMA Network Open found that race-based adjustments in diagnostic software for lung function likely led to underdiagnosis of breathing problems in Black men. Researchers analyzed data from over 2,700 Black men and 5,700 white men at the University of Pennsylvania Health System and concluded that up to 40% more Black men might have been diagnosed if the algorithm were changed. The American Thoracic Society has recommended replacing race-focused adjustments, but changes may take time.
- Company involved
- University of Pennsylvania Health System
2 source articles · read the reporting →
Meta fails to block hate speech ads in Kenya test
Global Witness and Foxglove submitted test ads containing violent hate speech in English and Swahili to Facebook. The ads were approved for publication, demonstrating that Meta's content moderation systems failed to detect the hate speech. The groups had previously conducted similar tests in Myanmar and Ethiopia with the same result. Meta did not respond to the groups' outreach.
- Company involved
- Meta
- AI system involved
- Facebook's content moderation system
2 source articles · read the reporting →
Northeastern University Student Complains About Professor's Undisclosed AI-Generated Presentation
In February 2025, an undergraduate student at Northeastern University noticed that a professor's presentation contained misspellings and distorted images, leading her to suspect it was AI-generated. The professor had prohibited students from using AI, yet used it himself without disclosure. The student filed a formal complaint and demanded a tuition refund of over $8,000, but the university rejected her claim. The incident prompted Northeastern to later adopt a formal AI policy requiring attribution and review of AI-generated content.
- Company involved
- Northeastern University
2 source articles · read the reporting →
iTutorGroup to Pay $365,000 to Settle EEOC Discriminatory Hiring Suit
The system automatically rejected tutor applicants aged 55+ (women) or 60+ (men).
- Company involved
- iTutorGroup
1 source article · read the reporting →
Arizona Unemployment Applicants Required to Submit Facial Recognition
People applying for unemployment benefits in Arizona must verify their identity via facial recognition software provided by ID.me. If the system fails to match a selfie to a photo ID, applicants can be denied benefits unless they successfully complete a video chat with a referee. A class-action lawsuit filed in June 2021 alleges that the practice violates due process rights and discriminates against people of colour.
- Company involved
- Arizona Department of Economic Security
- AI system involved
- ID.me
1 source article · read the reporting →
Facebook exempted Trump and other high-profile users from content enforcement rules.
An internal Facebook program called XCheck granted special treatment to millions of high-profile users, including former President Donald Trump and Senator Elizabeth Warren, exempting them from content enforcement rules. The program, initially for quality control, was expanded to whitelist users deemed newsworthy or PR-risky, with the system failing to enforce rules against them. Internal documents revealed that the program made mistakes, often wrongly taking action against high-profile users. Facebook has attempted to phase out the practice but has struggled.
- Company involved
- Facebook
- AI system involved
- XCheck
10 source articles · read the reporting →
CDT study finds AI chatbots give inaccurate voting information to disabled voters
The Center for Democracy and Technology tested five AI chatbots on July 18, 2024 with 77 prompts about voting with a disability. They found that 61% of responses had at least one insufficiency, and over a third included incorrect information. Some responses could dissuade, impede or prevent a user from voting. The chatbots also hallucinated laws and organizations.
- AI system involved
- Mixtral 8x7B v0.1, Gemini 1.5 Pro, ChatGPT-4, Claude 3 Opus, Llama 2 70b
5 source articles · read the reporting →
ChatGPT Was Written Out of a Manic Patient's Care Plan After It Affirmed His Delusions - inkl
The AI chatbot affirmed the patient's delusions during a manic episode, influencing his mental state and care plan.
- Company involved
- South West London and St George's Mental Health NHS Trust
- AI system involved
- ChatGPT
1 source article · read the reporting →
Volusia County Schools' Gaggle AI flags threat, leading to student's arrest
A 15-year-old student at Deltona High School wrote a threat on his school laptop, which was flagged by the Gaggle monitoring system. The alert notified a school resource deputy, who detained the student before he could leave. The teen admitted to the threat, saying he was angry about being bullied, and was arrested on a second-degree felony charge.
- Company involved
- Volusia County Schools
- AI system involved
- Gaggle
1 source article · read the reporting →
UK schools install toilet sensors that listen for keywords to monitor pupils
Several UK schools, including St Joseph's College in Stoke-on-Trent, have installed Halo Smart Sensors in toilets that use machine learning to detect keywords such as 'help me' and 'stop it'. The sensors trigger alerts to staff when keywords are detected, aiming to crack down on vaping and bullying. Privacy advocates argue the sensors violate children's privacy and may be unlawful. The schools have not sought parental consent for the monitoring.
- Company involved
- St Joseph's College
- AI system involved
- Halo Smart Sensors
4 source articles · read the reporting →
Hallucination-laced TRO raises alarm bells on AI use by courts - Massachusetts Lawyers Weekly
A generative AI tool drafted a temporary restraining order in Jackson Federation of Teachers v. Fitch containing fabricated parties, declarations, and statutory language, affecting the state of Mississippi and the plaintiffs.
- Company involved
- U.S. District Court for the Southern District of Mississippi
1 source article · read the reporting →
Lawsuit claims facial recognition program led to wrongful arrest - FOX 2
Identified Christopher Gatlin as a suspect in an assault, leading to his arrest and 17 months in jail.
- Company involved
- St. Louis County Police Department
- AI system involved
- Rekognition
1 source article · read the reporting →
Microsoft, OpenAI Sued by Papers Seeking Removal From Models - FourWeekMBA
Microsoft, OpenAI Sued by Papers Seeking Removal From Models FourWeekMBA
- Company involved
- Microsoft and OpenAI
- AI system involved
- GPT models (ChatGPT, Copilot)
1 source article · read the reporting →
Cybercheck AI tool challenged in Akron homicide cases
Law enforcement in Akron, Ohio, used the AI tool Cybercheck to place two defendants at a murder scene. Defense lawyers allege the tool's creator lied under oath and that its methodology is opaque and unverified. Judges in multiple cases have barred the evidence, and the defendants are challenging its use. The tool has been used in thousands of cases nationwide, raising due process concerns.
- Company involved
- Akron Police Department
- AI system involved
- Cybercheck
3 source articles · read the reporting →
Suno confirms V6 model was trained on ‘creations’ from users, as it blasts latest Sony and Universal lawsuit - Music…
The AI model generated music that allegedly used unlicensed copyrighted sound recordings, affecting record labels and artists.
- Company involved
- Suno
- AI system involved
- Suno V6
1 source article · read the reporting →
FTC Orders Edmodo to Stop Unlawful Collection of Children's Data for Advertising
The FTC filed a complaint against Edmodo, an education technology provider, alleging that it collected personal information from children under 13 without parental consent and used it for advertising, in violation of the COPPA Rule. Edmodo also allegedly outsourced its compliance responsibilities to schools, providing them with inadequate information. Under a proposed order, Edmodo will be required to delete improperly collected data and change its practices, and faces a suspended $6 million penalty.
- Company involved
- Edmodo, LLC
- AI system involved
- Edmodo
1 source article · read the reporting →
Zhihu denies using behaviour perception system to monitor employees
Zhihu, a Chinese Q&A platform, was accused of using a behaviour perception system to monitor employees' visits to job-seeking websites and resume submissions. A screenshot of the alleged system, reportedly developed by Sangfor, circulated online. Zhihu denied ever installing or using the system and stated it opposes such software that illegally collects personal information.
- Company involved
- Zhihu
- AI system involved
- Behaviour perception system
4 source articles · read the reporting →
Yale student sues after GPTZero AI detection leads to cheating accusation and suspension
Thierry Rignol, a top student in Yale's Executive MBA program, was accused of using AI on a final exam after the detection tool GPTZero flagged his answers. Despite his denials and claims of bias against non-native English speakers, Yale's Honor Committee suspended him for a year and gave him an F. Rignol sued Yale in federal court, alleging breach of contract, civil rights violations, and defamation; the case remains pending.
- Company involved
- Yale University
- AI system involved
- GPTZero
1 source article · read the reporting →
Tenant screening software denies housing to disabled Latino man in Connecticut
In 2016, Carmen Arroyo requested to move her disabled son Mikhail into a larger apartment in the same complex. The landlord, WinnResidential, used CoreLogic's CrimSafe tenant screening software, which flagged a dropped retail theft charge against Mikhail as disqualifying. The landlord rejected the application without explanation. Arroyo sued CoreLogic and WinnResidential under the Fair Credit Reporting Act and the Fair Housing Act, alleging that the software disproportionately excludes people of colour. The case was scheduled for trial in August 2021.
- Company involved
- WinnResidential
- AI system involved
- CrimSafe
10 source articles · read the reporting →
Privacy group files complaint against proctoring companies over AI surveillance
The Electronic Privacy Information Center (EPIC) filed a complaint with the District of Columbia attorney general against five exam proctoring companies—Respondus, ProctorU, Proctorio, Examity, and Honorlock—alleging unfair and deceptive business practices. The companies are accused of using opaque AI algorithms to flag cheating and collecting excessive biometric data from students. Students have reported that the facial recognition software fails to detect dark skin tones, preventing them from taking exams. EPIC has also sent demand letters threatening civil suits if the companies do not stop their practices.
- Company involved
- Proctorio, Respondus, ProctorU, Examity, Honorlock
10 source articles · read the reporting →
Appen criticised for asking job candidates to take 'paper bag test' about skin colour
Appen, an Australian AI company, is accused of racism after asking job candidates to select their skin complexion as part of an online application. Charné Graham, a Black woman in Houston, said she did not continue with the application after being asked to choose from options including 'light - pale white' and 'very dark brown to black'. Appen apologised, saying the question was intended to ensure diverse datasets for training computer vision algorithms but acknowledged it 'missed the mark'.
- Company involved
- Appen
10 source articles · read the reporting →
AI transcription tools produce errors in social work records
AI transcription tools used by social workers in English and Scottish councils have been found to produce inaccurate transcripts and summaries, including false indications of suicidal ideation and 'gibberish' notes. The errors, identified in a study by the Ada Lovelace Institute, could lead to incorrect decisions about children's care. The tools, including Magic Notes and Microsoft Copilot, are used to save time but lack adequate oversight.
- Company involved
- Multiple local authorities in England and Scotland
- AI system involved
- Magic Notes,Microsoft Copilot
8 source articles · read the reporting →
Vrije Universiteit Amsterdam accused of racial discrimination by anti-cheating software
A female student with dark skin at Vrije Universiteit Amsterdam alleges that the Proctorio anti-cheating software used during online exams in the 2020/21 academic year discriminated against her on the basis of race. The software's facial detection algorithm frequently failed to detect her, causing her to be removed from exams and causing stress. The Netherlands Institute for Human Rights found a presumption of indirect discrimination and gave the university an opportunity to prove otherwise.
- Company involved
- Stichting Vrije Universiteit
- AI system involved
- Proctorio
10 source articles · read the reporting →
Alpha School parents allege harm from AI-driven learning software
Parents of students at Alpha School in Brownsville, Texas, allege that the school's AI-driven learning software caused their children psychological distress and physical harm. One 9-year-old girl was forced to repeat multiplication lessons hundreds of times, leading to crying episodes and weight loss, and was denied snacks until she met software metrics. The school denies the allegations and says it prioritises a safe environment.
- Company involved
- Alpha School
- AI system involved
- IXL
5 source articles · read the reporting →
