Inaccuracies reveal Australian child protection worker used ChatGPT to draft court report

A child protection worker in Auatralia used ChatGPT to draft a critical report for a custody case, inadvertently disclosing sensitive information that could have endangered a child's safety and raising questions about over-reliance on generative AI for sensitive issues. What happened The incident, which was reported to the Victorian Office of the Information Commissioner (OVIC) in December 2023, revealed that the worker included personal details about the child and their family, which ChatGPT mishandled by downplaying risks associated with the child's living situation with parents charged with sexual offenses. The investigation found multiple signs of ChatGPT's involvement in the report, including inappropriate language and sentence structures inconsistent with child protection protocols. Notably, the report described a child's doll, allegedly used for inappropriate purposes by the father, as an "age-appropriate toy," which raised alarms about the potential misrepresentation of risks to the child. Why it happened The use of ChatGPT by Victoria's Department of Families, Fairness and Housing (DFFH) to draft child protection-related documents was system ic and not policied. OVIC found that the worker may have used ChatGPT in 100 cases in drafting child protection-related documents, and that nearly 900 employees - almost 13 percent of the workforce - had accessed ChatGPT across the department from July to December 2023 . What it means OVIC mandated that the DFFH block access to generative AI tools like ChatGPT , with t he department being given until November 5, 2024, to implement the restrictions and ensure compliance with privacy regulations. While the outcome of the child's case remained unchanged, th e incident highlights significant ethical and safety issues surrounding AI use in sensitive areas like child protection. OVIC Commissioner Sean Morrison emphasi s ed that AI systems can make serious errors that may lead to harmful consequences if relied upon without verification. System 🤖 ChatGPT Developer: OpenAI Country: Australia Sector: Govt - welfare Purpose: Generate text Technology: Generative AI Issue: Accountability; Privacy/surveilla nce; Safety Legal, regulatory 👩🏼‍⚖️ Office of the Victorian Information Commissioner . Investigation into the use of ChatGPT by a Child Protection worker

Date it happened

2023-12-01

Organisation involved

Department of Families, Fairness and Housing

Product, system or model

ChatGPT

This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.