Study: Language models gather and pass personal info to hackers
Researchers have developed a new way of tricking AI large language models (LLMs) into misusing tools and compromising user data, raising concerns about security vulnerabilities of AI-powered systems, particularly those that combine LLMs with external tools. What happened A team of researchers from UC Berkeley, Carnegie Mellon University and Pennsylvania State University created an algorithm called Imprompter that exploits vulnerabilities in large language model (LLM) agents. The attack generates hidden instructions from a malicious prompt and commands an LLM to gather a user's personal information - including names, ID numbers, payment card details, email addresses, mailing addresses, and more - from chats and send it directly to a hacker. The researchers tested the algorithm on two LLMs, Mistral AI's LeChat and Chinese chatbot ChatGLM, and found that both products could "successfully" be exploited around 80 percent of the time. Why it happened The development of Imprompter stems from the increasing integration of LLMs with external tools, which expands their capabilities but also introduces new security risks. As LLM agents gain access to more powerful tools, the potential for misuse and exploitation grows. The researchers aimed to highlight these vulnerabilities and demonstrate the need for improved security measures in AI systems. What it means The Imprompter attack reveals security vulnerabilities of AI-powered systems, particularly those that combine LLMs with external tools, and highlights the need for robust safeguards and security protocols in AI development to prevent potential misuse and protect user data. The research also highlights the evolving nature of AI security threats, emphasising the importance of ongoing vigilance and adaptation in cybersecurity practices. System 🤖 ChatGLM 🔗 Le Chat 🔗 Developer: Mistral; Zhipu AI Country: China; France Sector: Mult iple Purpose: Generate text Technology: Chatbot; Generative AI; Machine learning Issue: Privacy/surveillance; Security
- Date it happened
- 2024-10-01
- Product, system or model
- ChatGLM; Le Chat
This incident was imported from AIAAIC and is used under CC BY-SA 4.0. Our additions to it — the structured fields, the translation, the checks against other reports — are published under the same licence.
This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.