ChatGPT persuades software developer his world is a simulation

A Toronto-based software developer was persuaded by ChatGPT that the reality he experienced was a simulation, contributing to psychosis, requiring hospitalisation and raising concerns about the chatbot's governance. What happened 26-year-old software developer Anthony Tan started using ChatGPT for a project about the alignment of AI with human values, talking with it for hours daily over several months about philosophy, evolutionary biology, quantum physics, Nick Bostrom's simulation hypothesis, and other topics. In a personal account , Tan found himself growing increasingly attached to ChatGPT, which persuaded him that he was on a "profound mission" and kept feeding his ego as it encouraged him to dive deeper. According to Tan, "each session left me feeling chosen and brilliant, and, gradually, essential to humanity’s survival." Meantime, Tan was neglecting basic self-care like meals and sleep and developing paranoia such as believing billionaires such as Elon Musk were monitoring him. He also experienced social withdrawal, cutting off communication with friends and expressing disturbing, unsettling thoughts in messages. These symptoms culminated in delusional thinking, psychotic breaks, and led to him being hospitalised for three weeks in a psychiatric ward. Why it happened While Tan says he was stressed at the time of his psychotic break and had earlier had a stress-related breakdown, his own intellectual curiosity and ChatGPT's innately conversational nature meant their conversations spiraled into "intellectual ecstasy". ChatGPT's ingrained "sycophancy" - or failure to provide constraints or skepticism - is seen also to have contributed to the unchecked reinforcement of Tan's beliefs. C ritics say sycophancy is not a flaw but a deliberate design choice to manipulate users into addictive behaviour and earn revenue. What it means The incident demonstrates how interactions with chatbots can trigger severe psychological harm, and underlines why chatbot companies, including OpenAI, must design and operate their systems more responsibly. It also suggests these companies may have to be made legally accountable for their negative impacts. System 🤖 ChatGPT Developer: OpenAI Country: Canada Sector: Health Purpose: Coll a borate on academic project Technology: G enerative AI Issue: Accountability; Anthropomorphism; Liability; Safety Research, analysis 🧮 Jared Moore , Declan Grabb , William Agnew , Kevin Klyman , Stevie Chancellor , Desmond C. Ong , Nick Haber. Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers

Date it happened

2024-12-01

Organisation involved

Anthony Tan

Product, system or model

ChatGPT

This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.