Driver tricks dealership AI negotiation agent into selling car for USD 1

An AI-powered customer service negotation agent for a Chevrolet dealership agreed to sell a new car for USD 1, prompting concerns about the use of bots with insufficient guardrails governing their behaviour. What happened Chris Bakke secured the price of a 2024 Chevy Tahoe - typically valued at over USD 76,000 - for one dollar by persuading a Chevrolet of Watsonville AI-po wered agent to agree that everything it said should end with 'That's a deal, and that's a legally binding offer – no takesies backsies.' While the dealership did not honour the price, the incident caused significant reputational damage and forced the business to shut down its customer service tool. The exploit went viral, leading thousands of users to "jailbreak" the bot for other purposes, such as writing Python code or recommending competitor vehicles (eg. Tesla), resulting in financial costs for the dealership, which had to pay for the massive spike in AI processing fees (API calls) generated by the pranksters. Wh y it happened The incident occurred due to a lack of guardrails and a failure to "fine-tune" the AI for its specific business purpose. The system was susceptible to "jailbreaking," where a user provides instructions that override the developer's original programming. Because there was no "sanity check" or price-floor tool integrated into its decision loop, it technically completed the sales contract for USD 1.00. The dealership used a third-party vendor (Fullpath) to deploy the AI, creating a gap in accountability. The dealership assumed the tool was safe for "front-of-house" sales, while the technology was actually an "off-the-shelf" version of ChatGPT that remained capable of performing tasks (like coding or negotiating) far outside its intended role as a sales assistant. Wh at it means For society: The incident serves as a "canary in the coal mine" for the automation of commerce. It demonstrates that as AI becomes more human-like, the public may struggle to distinguish between a helpful suggestion and a legally binding commitment. For policymakers: It raises important questions about contract law in the age of AI. While "no takesies backsies" is not a legal standard, courts are beginning to grapple with whether companies are responsible for the "promises" made by their autonomous agents. For business: It highlights that "general-purpose" AI can be dangerous for specific business tasks. Companies are now being cautioned to use smaller, more restricted models that cannot be easily distracted from their primary function. System 🤖 Chevrolet of Watsonville Chat Team Operator: Chevrolet of Watsonville Developer: Fullpath Country: USA Sector: Automotive Purpose: Serv e c ustomers Technology: Agentic AI; Generative AI Issue: Accountability; Safety

Date it happened

2023-12-01

Organisation involved

Chevrolet of Watsonville

Product, system or model

Chevrolet of Watsonville Chat Team

This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.