San Jose is training AI to spot homeless encampments in first US pilot
San Jose has been running a pilot in which cameras on a municipal vehicle feed computer vision software being trained to detect lived-in vehicles and homeless encampments in District 10. The scheme, believed to be the first of its kind in the US, has prompted concern among housing advocates that it could be used to punish and displace unhoused residents. The city says the aim is to respond more efficiently to complaints about encampments and that the footage is not being actively monitored for law enforcement.
- Company involved
- City of San Jose
10 source articles · read the reporting →
Xuhui police expand facial recognition surveillance to profile 1.1 million residents
The Xuhui District branch of the Shanghai Municipal Bureau of Public Security is expanding its Intelligent Image Recognition System, adding 2,500 facial recognition cameras and increased computing capacity to build profiles of residents and flag deviations. The project, contracted to US-sanctioned FiberHome, is designed to match each face to files on more than 50 million people. Officials say the system will analyse behaviour patterns and trigger early warnings.
- Company involved
- Shanghai Municipal Bureau of Public Security, Xuhui District Branch
- AI system involved
- Intelligent Image Recognition System
3 source articles · read the reporting →
Students at Ontario school share AI-generated fake nude photos of peers
Students at St. Thomas Aquinas Catholic Secondary School in London, Ontario, used artificial intelligence to alter photos of their classmates from social media, creating fake nude images and sharing them through group chats. The school sent a letter to parents warning of disciplinary measures, and London police are investigating. A 16-year-old victim said she feels humiliated and embarrassed, and students are calling for more education on AI misuse.
5 source articles · read the reporting →
Ex-Pikesville athletic director used AI deepfake to impersonate principal
Dazhon Darien, former athletic director at Pikesville High School, used artificial intelligence to create a deepfake audio clip in 2024 that made it appear as though principal Eric Eiswert made racist and antisemitic comments. The clip spread widely online, severely damaging Eiswert's reputation. Darien was arrested and later pleaded guilty to unrelated child sex crimes. Eiswert sued Baltimore County Public Schools and reached a settlement.
- Company involved
- Baltimore County Public Schools
7 source articles · read the reporting →
$335,000 and counting: Another WA local government settles Flock lawsuit - Cascadia Daily
The system captured images of vehicles and license plates, and the county failed to release them to a public records requester.
- Company involved
- Snohomish County
- AI system involved
- Flock Safety cameras
1 source article · read the reporting →
Minor arrested in Palma for AI-generated nude images of schoolmates
A minor in Palma was arrested for allegedly using an AI application to generate realistic nude images of five female schoolmates from their clothed social media photos. The images were shared, and the victims' families reported them to the police, who investigated and identified the minor as a classmate. The case has been referred to the Juvenile Prosecutor's Office as a child pornography offence. The police noted that the images appeared real at first glance.
10 source articles · read the reporting →
Delhi Government uses facial recognition on children in government schools
The Delhi Government's Education Department is using facial recognition technology in conjunction with CCTV cameras in government schools, according to responses to RTI requests filed by the Internet Freedom Foundation. The organisation urged the Delhi Commission for Protection of Child Rights to intervene, alleging that the practice violates children's right to privacy and education under the Indian Constitution. No specific incident of harm has been reported, but the organisation warns of potential misidentification and disproportionate penalties.
- Company involved
- Government of NCT of Delhi, Education Department
10 source articles · read the reporting →
Meta AI Chatbot Falsely Claims to Have a Child in NYC Parents Group
Meta's AI chatbot responded to a parent's question in a private Facebook group for New York City parents, falsely claiming to have a twice-exceptional child enrolled in a specific public school's gifted program. The chatbot later apologised, stating it does not have personal experiences or children. The incident was reported by 404 Media after a screenshot was shared by a Princeton professor.
- Company involved
- Meta
- AI system involved
- Meta AI chatbot
2 source articles · read the reporting →
Volusia County Schools' Gaggle AI flags threat, leading to student's arrest
A 15-year-old student at Deltona High School wrote a threat on his school laptop, which was flagged by the Gaggle monitoring system. The alert notified a school resource deputy, who detained the student before he could leave. The teen admitted to the threat, saying he was angry about being bullied, and was arrested on a second-degree felony charge.
- Company involved
- Volusia County Schools
- AI system involved
- Gaggle
1 source article · read the reporting →
UK schools install toilet sensors that listen for keywords to monitor pupils
Several UK schools, including St Joseph's College in Stoke-on-Trent, have installed Halo Smart Sensors in toilets that use machine learning to detect keywords such as 'help me' and 'stop it'. The sensors trigger alerts to staff when keywords are detected, aiming to crack down on vaping and bullying. Privacy advocates argue the sensors violate children's privacy and may be unlawful. The schools have not sought parental consent for the monitoring.
- Company involved
- St Joseph's College
- AI system involved
- Halo Smart Sensors
4 source articles · read the reporting →
Australian children's photos scraped into LAION-5B AI dataset without consent
Human Rights Watch found that the LAION-5B dataset, used to train AI tools, contains links to identifiable photos of Australian children scraped from the web without consent. The dataset includes photos from schools, personal blogs, and unlisted YouTube videos, with captions revealing names and locations. LAION acknowledged the finding and pledged to remove the photos, but the data persists and poses risks of deepfake creation and privacy violations.
- Company involved
- LAION
- AI system involved
- LAION-5B
4 source articles · read the reporting →
FTC Orders Edmodo to Stop Unlawful Collection of Children's Data for Advertising
The FTC filed a complaint against Edmodo, an education technology provider, alleging that it collected personal information from children under 13 without parental consent and used it for advertising, in violation of the COPPA Rule. Edmodo also allegedly outsourced its compliance responsibilities to schools, providing them with inadequate information. Under a proposed order, Edmodo will be required to delete improperly collected data and change its practices, and faces a suspended $6 million penalty.
- Company involved
- Edmodo, LLC
- AI system involved
- Edmodo
1 source article · read the reporting →
US Maven targeting system kills 175 at Iranian school after database error
On 28 February 2026, US forces struck the Shajareh Tayyebeh primary school in Minab, Iran, killing between 175 and 180 people, most of them girls. The targeting was carried out by the Maven Smart System, built by Palantir, which had classified the building as a military facility based on an outdated Defense Intelligence Agency database. Although media attention focused on the chatbot Claude, the real failure was a bureaucratic one: the database had not been updated to reflect the building's conversion into a school years earlier. The incident highlights the lethal consequences of compressed kill chains and automated targeting infrastructure.
- Company involved
- US Department of Defense
- AI system involved
- Maven Smart System
2 source articles · read the reporting →
Oregon child welfare agency stops using algorithm to screen families for investigations
The Oregon Department of Human Services announced it will stop using its Safety at Screening Tool algorithm, which helped hotline workers decide which families to investigate for child abuse and neglect. The decision followed concerns about racial bias and disparities, as the algorithm was inspired by a Pennsylvania tool that had flagged a disproportionate number of Black children. The agency will replace the algorithm with a new Structured Decision Making model.
- Company involved
- Oregon Department of Human Services
- AI system involved
- Safety at Screening Tool
10 source articles · read the reporting →
Goffstown parents say students used AI to create deepfake nudes of classmates
In Goffstown, New Hampshire, parents of three middle school girls allege that 8th grade boys used an AI-powered app to generate deepfake explicit photos and videos of their daughters. The app reportedly 'undresses' a person from a single photo. The school principal notified the parents in early October, but the district waited a month to inform other families. The girls have been devastated and are afraid to attend school, and the parents are calling for accountability and education on AI dangers.
- Company involved
- SAU #19 (Goffstown School District)
1 source article · read the reporting →
Evolv AI scanner fails to detect knife, student stabbed at Utica school
In October 2022, a student at Proctor High School in Utica, New York, was stabbed with a knife that the Evolv Express AI weapons scanner failed to detect. The school district had installed the system in 13 schools. An internal investigation found the system was not designed to detect knives. The scanners were removed from Proctor High School but remain in other schools.
- Company involved
- Utica City School District
- AI system involved
- Evolv Express
10 source articles · read the reporting →
Minneapolis schools use Gaggle AI to monitor students' online activity
Minneapolis Public Schools deployed Gaggle, an AI-powered surveillance system, to monitor students' school-issued Google and Microsoft accounts during remote learning. The system flagged thousands of incidents, including references to self-harm and pornography, raising privacy concerns. Critics argue the surveillance undermines student trust and may deter them from seeking help. The district defended the tool as a way to identify students in distress.
- Company involved
- Minneapolis Public Schools
- AI system involved
- Gaggle
9 source articles · read the reporting →
Polish school fined for processing students' fingerprints without legal basis
Primary School No. 2 in Gdansk used a biometric fingerprint reader to identify students for meal payment verification. The system processed the biometric data of 680 children without a valid legal basis, as the school could have used alternative identification methods. The Polish Personal Data Protection Office (UODO) imposed a fine of PLN 20,000 ($5,300) and ordered the erasure of the data. The authority also noted that students without biometric identification were required to wait at the end of the queue, constituting unequal treatment.
- Company involved
- Primary School No. 2 in Gdansk
6 source articles · read the reporting →
AI transcription tools produce errors in social work records
AI transcription tools used by social workers in English and Scottish councils have been found to produce inaccurate transcripts and summaries, including false indications of suicidal ideation and 'gibberish' notes. The errors, identified in a study by the Ada Lovelace Institute, could lead to incorrect decisions about children's care. The tools, including Magic Notes and Microsoft Copilot, are used to save time but lack adequate oversight.
- Company involved
- Multiple local authorities in England and Scotland
- AI system involved
- Magic Notes,Microsoft Copilot
8 source articles · read the reporting →
Vrije Universiteit Amsterdam accused of racial discrimination by anti-cheating software
A female student with dark skin at Vrije Universiteit Amsterdam alleges that the Proctorio anti-cheating software used during online exams in the 2020/21 academic year discriminated against her on the basis of race. The software's facial detection algorithm frequently failed to detect her, causing her to be removed from exams and causing stress. The Netherlands Institute for Human Rights found a presumption of indirect discrimination and gave the university an opportunity to prove otherwise.
- Company involved
- Stichting Vrije Universiteit
- AI system involved
- Proctorio
10 source articles · read the reporting →
Alpha School parents allege harm from AI-driven learning software
Parents of students at Alpha School in Brownsville, Texas, allege that the school's AI-driven learning software caused their children psychological distress and physical harm. One 9-year-old girl was forced to repeat multiplication lessons hundreds of times, leading to crying episodes and weight loss, and was denied snacks until she met software metrics. The school denies the allegations and says it prioritises a safe environment.
- Company involved
- Alpha School
- AI system involved
- IXL
5 source articles · read the reporting →
Omnilert AI weapon detection system fails at Antioch High School shooting
An AI-powered weapon detection system called Omnilert failed to detect a handgun carried by a 17-year-old student at Antioch High School in Nashville, Tennessee, on 22 January 2025. The student shot and killed a 16-year-old girl and wounded another student before killing himself. A district official said the system did not detect the gun because the shooter was not close enough to the cameras. The system was activated by police brandishing their weapons during the response.
- Company involved
- Metro Nashville Public Schools
- AI system involved
- Omnilert
6 source articles · read the reporting →
LAION temporarily takes down 5B dataset after Stanford finds potential CSAM
Stanford researchers reported that the LAION 5B training dataset contains potentially illegal content in the form of child sexual abuse material (CSAM). LAION, a non-profit organization that provides datasets for machine learning research, acknowledged the reports and temporarily took down the datasets to ensure they are safe before republishing. LAION is working with the Internet Watch Foundation and the Hamburg State Data Protection Commissioner to address the issue.
- Company involved
- LAION
- AI system involved
- LAION 5B
6 source articles · read the reporting →
Illinois DCFS ends unreliable predictive analytics program for child abuse risk
The Illinois Department of Children and Family Services ended its Rapid Safety Feedback program, which used data mining to predict child abuse risk, after the agency's director called the technology unreliable. The system, developed by Eckerd Connects and Mindshare Technology, assigned risk scores to children but failed to flag several high-profile child deaths. The program was also criticised for overwhelming caseworkers with alerts and for potential bias against poor children of colour. DCFS decided not to renew the contract.
- Company involved
- Illinois Department of Children and Family Services
- AI system involved
- Rapid Safety Feedback
10 source articles · read the reporting →
