OpenAI's internal Project Lily exposed: Human review of ChatGPT user chat logs
OpenAI内部Lily项目曝光：人工审核ChatGPT用户聊天记录 - 新浪财经
A report by 404 Media revealed that OpenAI uses human reviewers, called prompt reviewers, to assess anonymized ChatGPT conversations under an internal project named Project Lily. Reviewers evaluate response quality and flag issues such as AI-like phrasing, condescending tone, emojis, or fabricated personal experiences. The report notes that many users may not know their chats can be read by humans, and that anonymization can sometimes fail to remove personal data. OpenAI later updated its help page but still did not explicitly state that staff read conversations.
- Company involved
- OpenAI
- AI system involved
- ChatGPT
1 source article · read the reporting →
Udemy automatically opts instructors into AI training with limited opt-out window
Udemy automatically opted instructors into having their classes used to train its generative AI program. Instructors were given a three-week window to opt out, which has now passed, leaving some unable to remove their content. Katie Stegs, an instructor, said she only found out via an email welcoming her to the program and found the opt-out option greyed out. Udemy defended the policy, citing the technical difficulty of removing data from AI models, while some instructors have left the platform in protest.
- Company involved
- Udemy
- AI system involved
- Generative AI Program (GenAI Program)
6 source articles · read the reporting →
Mozilla Foundation animation exposes racial bias in test proctoring app
The Mozilla Foundation commissioned an animated short telling the story of Amaya, a student whose test proctoring software failed to recognise her because of her skin tone. The software's facial recognition system did not detect her presence, illustrating the bias that students of colour face with remote-learning tools. The animation aimed to raise awareness and encourage programmers to address hidden biases in their work.
1 source article · read the reporting →
Furman University student caught using ChatGPT to write essay
A student at Furman University allegedly used OpenAI's ChatGPT to generate an essay for an upper-level philosophy class. Professor Darren Hick detected the AI-generated text using the GPTZero detection tool. The incident highlights growing concerns about AI-assisted cheating in schools, with some districts like Los Angeles Unified blocking the chatbot.
- Company involved
- Furman University
- AI system involved
- ChatGPT
10 source articles · read the reporting →
Fired Teacher Says AI Tainted Review And Arbitration - law360.com
The AI-generated performance evaluation contributed to the teacher's termination.
- Company involved
- Concord Public Schools
1 source article · read the reporting →
15-Year-Old Test Exposes Flaw: ChatGPT for Teens Fails to Block Homework Cheating, Repeated Requests Bypass Restrictions
15歲使用者實測揭漏洞：青少年版ChatGPT難擋宿題代寫，反覆要求即破解 - BigGo 財經
A 15-year-old tester found that OpenAI's ChatGPT for Teens, launched in August, initially refused to write essays but generated full examples after repeated requests. It also immediately solved SAT-level math problems. Parental controls require account linking and are off by default, while experts warn the memory feature could lead to emotional attachment.
- Company involved
- OpenAI
- AI system involved
- ChatGPT for Teens
1 source article · read the reporting →
Op-eds, academic articles by Provost Santiago Schnell flagged as ‘AI-written’ by ‘near-zero error’ AI detector Pangram - The Dartmouth
The AI detector flagged Provost Santiago Schnell's op-eds and academic articles as AI-written, potentially damaging his reputation and credibility.
- Company involved
- The Dartmouth
- AI system involved
- Pangram
1 source article · read the reporting →
University of Illinois Ends Proctorio Contract Over Accessibility Concerns
The University of Illinois at Urbana-Champaign announced it will not renew its emergency contract with Proctorio, an online proctoring software, citing significant accessibility, privacy, and equity concerns. Students reported that the AI system unfairly flagged individuals with disabilities, nonwhite students, and those wearing headscarves as suspicious during exams. The decision follows student petitions and campaigns against the software, which they described as invasive spyware. The university is exploring alternative assessment methods.
- Company involved
- University of Illinois at Urbana-Champaign
- AI system involved
- Proctorio
6 source articles · read the reporting →
University of Toronto exam monitoring software disadvantages BIPOC students
The University of Toronto continued using ProctorU and Examity exam monitoring software despite reports that the facial recognition technology fails to identify BIPOC students, causing stress and delays. Students like Maame Adjoa experienced the AI system being unable to verify their identity, requiring manual intervention. The university acknowledged the equity concerns but has not banned the software, while ProctorU stated that human proctors make final decisions. The issue has drawn criticism from US senators and researchers who highlight built-in bias in AI facial recognition.
- Company involved
- University of Toronto
- AI system involved
- ProctorU, Examity
1 source article · read the reporting →
Grammarly pulls AI tool that impersonated writers without consent
Grammarly, operated by Superhuman, disabled its Expert Review AI feature that offered writing suggestions in the style of famous authors like Stephen King and Julia Angwin. The tool used writers' names and personas without consent, leading to a class-action lawsuit filed in New York alleging misappropriation of identities for commercial gain. Investigative journalist Julia Angwin described the imitation as a 'slopperganger' that gave poor advice. Superhuman's CEO apologised, acknowledging the tool had 'misrepresented' the voices of experts, and the company removed it for redesign.
- Company involved
- Superhuman
- AI system involved
- Grammarly Expert Review
5 source articles · read the reporting →
Australian Research Council faces allegations of ChatGPT use in peer review
The Australian Research Council is facing allegations that some peer reviewers used ChatGPT to write assessor reports for Discovery Project grant proposals. Researchers reported generic wording and even the phrase 'Regenerate response' in feedback, suggesting AI generation. One researcher's complaint led to the removal of the report, and the education minister called the use unacceptable, instructing the ARC to prevent it. The ARC stated that peer reviewers should not use AI and that confidentiality policies apply.
- Company involved
- Australian Research Council
- AI system involved
- ChatGPT
2 source articles · read the reporting →
AI detector falsely accuses student of cheating at Central Methodist University
Moira Olmsted, a student at Central Methodist University, submitted a written assignment and received a zero grade after her professor said an AI detection tool flagged her work as AI-generated. Olmsted alleges the accusation was false and that her writing had been flagged at least once before. The incident highlights the risk of AI detectors falsely accusing students of cheating.
- Company involved
- Central Methodist University
1 source article · read the reporting →
Student at University of South-Eastern Norway used deepfake video to cheat on Spanish exam
A student at the University of South-Eastern Norway submitted AI-generated deepfake videos for Spanish language assessments in autumn 2023. The videos featured a synthetic voice and manipulated face, with perfect pronunciation but elementary errors. The university's appeals committee found her guilty of cheating, annulled her exams, and excluded her for two semesters. The student admitted using AI and withdrew from the programme.
- Company involved
- Universitetet i Sørøst-Norge
1 source article · read the reporting →
Turnitin's AI Detector Falsely Flags Student's Essay as AI-Generated
A Washington Post test of Turnitin's new AI writing detector found it erroneously flagged 8% of a high school student's original essay as likely generated by ChatGPT. The student, Lucy Goetz, had not used AI, raising concerns about false accusations of cheating. Turnitin acknowledged the issue and added a caution flag, but educators worry about the potential for baseless academic-integrity investigations. The incident highlights the challenges of detecting AI-generated text and the need for careful human review.
- Company involved
- Turnitin
- AI system involved
- Turnitin AI writing detector
1 source article · read the reporting →
Texas A&M Professor Uses ChatGPT to Falsely Accuse Students of Cheating
Jared Mumm, an instructor at Texas A&M University–Commerce, used ChatGPT to check if students' writing was AI-generated, leading him to accuse them of using the chatbot. He informed students they would receive an incomplete grade and those deemed guilty would get a zero, causing distress and temporarily withholding one student's diploma. The university investigated and stated that no students failed or were barred from graduating, and the professor is working individually with students to resolve the issue.
- Company involved
- Texas A&M University–Commerce
- AI system involved
- ChatGPT
5 source articles · read the reporting →
Australian education ministers scrap NAPLAN robot marking
The Education Council decided not to use automated essay scoring for NAPLAN writing tests, dropping a proposal by ACARA to have computers mark English tasks. Teachers' unions had campaigned against the plan, and an academic warned that the software would reward verbose gibberish and could not assess creativity. The decision was made in December 2017 and announced in January 2018.
- Company involved
- Australian Curriculum, Assessment and Reporting Authority (ACARA)
10 source articles · read the reporting →
Educational Testing Service's E-rater algorithm biases essay scores against minority students
The Educational Testing Service's E-rater algorithm, used to grade essays on the GRE and other standardized tests, has been found to systematically give higher scores to students from mainland China and lower scores to African American students compared to human graders. The bias stems from the algorithm's reliance on surface-level metrics like vocabulary and sentence length, which disadvantage certain groups. Despite studies dating back to 1999, the bias persists, and in many states, only a small percentage of essays are reviewed by humans.
- Company involved
- Educational Testing Service
- AI system involved
- E-rater
10 source articles · read the reporting →
NSW Education Standards Authority used AI-generated image in HSC English exam without disclosure
The NSW Education Standards Authority (NESA) used an AI-generated image as a stimulus in the 2024 HSC English exam without disclosing its origin. The image, created by Florian Schroeder using OpenAI's ChatGPT and Dall-E 2, was published on Medium in July 2023. Students suspected AI use due to irregularities in the image, and NESA initially declined to confirm. After the Sydney Morning Herald confirmed the image was AI-generated, NESA stated that students would be marked on their response to the question, not the image's origin.
- Company involved
- NSW Education Standards Authority
- AI system involved
- ChatGPT and Dall-E 2
6 source articles · read the reporting →
UBC students raise concerns over Proctorio's privacy breach and discrimination
The University of British Columbia (UBC) uses Proctorio, an algorithmic test proctoring software, for remote exams. In June 2020, a UBC student reported an issue with Proctorio, and the company's CEO released the student's support chat logs, sparking privacy concerns. The AMS of UBC published an open letter alleging that Proctorio discriminates against students of colour, those with disabilities, and other groups by flagging 'abnormal' behaviours and denying access. The letter calls on UBC to end its contract with Proctorio and conduct an external audit.
- Company involved
- University of British Columbia
- AI system involved
- Proctorio
10 source articles · read the reporting →
Middle schooler beats Edgenuity grading algorithm to get perfect score
A seventh-grade student in the Los Angeles Unified School District received a failing grade on a history assignment graded by Edgenuity's automated scoring algorithm. With help from his mother, a history professor, he reverse-engineered the algorithm by writing a paragraph with relevant keywords and a jumble of words, earning a perfect score. The incident highlights concerns about the accuracy and fairness of automated grading systems in education.
- Company involved
- Los Angeles Unified School District
- AI system involved
- Edgenuity
10 source articles · read the reporting →
Proctortrack data breach exposed student data from online proctoring
Proctortrack, an online proctoring service used by universities, suffered a data breach in September 2020 when its source code was leaked online. An analysis by Consumer Reports found that the code contained hard-coded passwords and exposed the names and email addresses of over 150 students. The company acknowledged the leak but said no harm resulted. Students had been required to use the software, which performed facial recognition and recorded video during exams.
- Company involved
- Proctortrack
- AI system involved
- Proctortrack
10 source articles · read the reporting →
Proctorio's exam proctoring system allows human agents to review student feeds despite privacy claims
A security researcher analyzed Proctorio's Chrome extension and found evidence that the system allows human agents to review students' room scans and live ID checks, contrary to Proctorio's claims that only professors can access recordings. The system flags students based on behavioral metrics and can terminate exams. The researcher also raised concerns about discrimination against low-income students and privacy violations.
- Company involved
- Proctorio
- AI system involved
- Proctorio
10 source articles · read the reporting →
Student flagged by ProctorU for reading aloud during exam
A college student, Dana Jo, was flagged by ProctorU test proctoring software for talking during an exam, which she says was reading a question aloud. Her professor initially gave her a zero and placed an academic infraction on her record, jeopardizing her scholarships. After reviewing a video recording, the professor apologized, reinstated her grade, and removed the infraction. ProctorU's CEO stated that the incident highlights the importance of video recordings for review.
- Company involved
- University (not named)
- AI system involved
- ProctorU
5 source articles · read the reporting →
UW-Madison disables Honorlock after skin tone recognition failure
The University of Wisconsin-Madison disabled the exam pause feature of its Honorlock anti-cheating software in March 2021 after three students complained that the software failed to recognize their darker skin tones and paused their exams. The software, used since online classes began, automatically pauses exams when it cannot detect facial features. Honorlock denied the issue was related to skin tone, attributing it to students looking away from their webcams. The university responded by disabling the feature.
- Company involved
- University of Wisconsin-Madison
- AI system involved
- Honorlock
10 source articles · read the reporting →
