Niulanshan First Secondary School Classroom Care System opacity
The movements and behaviour of students at Niulanshan First Secondary School in Beijing are being constantly monitored and scored by a system about which they have been no given information nor to which they have provided their consent. Launched in 2017, Hanwang Education's Class Care System (CCS) uses facial recognition, emotion recognition and deep learning algorithms to identify students' faces and analyse and classify their behaviour into one of five categories: listening, answering questions, writing, interacting with other students, or sleeping. Each student is automatically scored weekly and their information shared via a mobile app with their teachers and parents. Hanwang argues the scores are important for identifying which students require support. But c ritics point out that the system is not only highly intrusive, it can also be inaccurate and unfair. As Rest of World describes , in one example 'a student who had answered just a single question in his English class was called out for low participation — despite the app recording him as 'focused' 94% of the time.' Research studies regularly conclude that emotion recognition technologies are based on the fundamentally flawed premise that algorithms can analyse a person’s facial expressions and accurately infer their inner state or mood. System 🤖 Class Care System Operator: Niulanshan First Secondary School Developer: Hanwang Technology Country: China Sector: Education Purpose: Assess and rank student behaviour Technology: Facial recognition; Emotion recognition; Deep learning; Neural network; Machine learning Issue: Accuracy /reliability; Privacy; Surveillance
- Date it happened
- 2018-09-01
- Organisation involved
- Niulanshan First Secondary School
- Product, system or model
- Hanwang CCS Classroom Care System
This incident was imported from AIAAIC and is used under CC BY-SA 4.0. Our additions to it — the structured fields, the translation, the checks against other reports — are published under the same licence.
This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.