Koko AI mental health counselling 'experiment' fails to obtain user consent
Mental health non-profit Koko use of GPT-3 in a Discord-based 'experiment' to provide support to people seeking counseling was criticised for failing to obtain the informed consent of the 4,000 people using the system. Users send direct messages to the Discord 'Kokobot' that asks several multiple-choice questions, and then shares a person's concerns anonymously with someone else on the server who can reply anonymously with a short message - either of their own, or one automatically generated by GPT-3. According to Koko CEO Rob Morris, 'Messages composed by AI (and supervised by humans) were rated significantly higher than those written by humans on their own (p < .001). Response times went down 50%, to well under a minute … [but] once people learned the messages were co-created by a machine, it didn’t work. Simulated empathy feels weird, empty.' During the backlash that ensued, critics asked whether an Instituti o nal Review Board (IRB) had approved the experiment. It is illegal to conduct research on human subjects without so-called 'informed consent' unless an IRB finds that consent can be waived in the US. In response, Morris said the experiment was exempt because participants opted in, their identities were anonymised, and an intermediary evaluated the responses before they were shared with people who sought help. Morris told Vice 'We pulled the feature anyway and I wanted to unravel the concern as a thought piece, to help reign in enthusiasm about gpt3 replacing therapists.' System 🤖 GPT-3 Operator: Koko Developer: Koko Country: USA Sector: Health Purpose: Provide mental health support Technology: Large language model (LLM); Machine learning Issue: Privacy; Transparency News, analysis, commentary 🗞️ https://www.vice.com/en/article/4ax9yw/startup-uses-ai-chatbot-to-provide-mental-health-counseling-and-then-realizes-it-feels-weird https://arstechnica.com/information-technology/2023/01/contoversy-erupts-over-non-consensual-ai-mental-health-experiment/ https://www.newscientist.com/article/2354077-mental-health-service-used-an-ai-chatbot-without-telling-people-first/ https://metro.co.uk/2023/01/10/mental-health-app-faces-backlash-for-testing-chatgpt-on-4000-users-18070559/ https://www.politico.com/newsletters/digital-future-daily/2023/01/10/tracking-the-ai-apocalypse-00077279 https://gizmodo.com/mental-health-therapy-app-ai-koko-chatgpt-rob-morris-1849965534 Related ChatGBT chatbot Crisis Text Line data sharing Page info Type: Incident Publish ed: January 2023
- Date it happened
- 2023-01-01
- Organisation involved
- Koko; Discord
- Product, system or model
- GPT-3
This incident was imported from AIAAIC and is used under CC BY-SA 4.0. Our additions to it — the structured fields, the translation, the checks against other reports — are published under the same licence.
This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.