NYC AI chatbot tells businesses to break law
New York City’s Microsoft-powered AI chatbot was found to give wrong advice and inadvertently encouraged users to violate the law, according to reports. Hailed by NYC mayor Eric Adams as 'a once-in-a-generation opportunity to more effectively deliver for New Yorkers,' the MyCity Chatbot was intended to provid e information to reside nts about starting and operating businesses within the city . However, the bot was discovered to be providing 'dangerously' inaccurate and misleading information and advice, including: Wrongly asserting that landlords do not need to accept tenants with Section 8 vouchers or those receiving rental assistance. In reality, New York City law prohibits discrimination based on source of income. Incorrectly claiming that it is legal to lock out tenants and that there are no restrictions on rent amounts for residential tenants. Actually, tenant lockouts are not permissible after 30 days of residence, and rent-stabilised units have specific regulations, while landlords of other private units have more flexibility in setting rent prices. Failing to recognise a 2020 City Council law requiring businesses to accept cash to prevent discrimination against unbanked customers. The concerns prompted experts to complain that the chatbot offered misguided advice with significant potential legal, reputational and operational ramifications, and that it was taken down. Per The Markup , the bot was still live a week later and churning out much the same misguided advice and encouraging illegal behaviour. Meantime, a message flagged it as 'a beta product' that may provide “inaccurate or incomplete” responses to queries. Hallucination (artificial intelligence) In the field of artificial intelligence (AI), a hallucination or artificial hallucination (also called bullshitting, confabulation or delusion) is a response generated by AI that contains false or misleading information presented as fact . Source: Wikipedia System 🤖 MyCity Chatbot 🔗 Operator: City of New York Developer: Microsoft Country: USA Sector: Govt - business Purpose: Provide business support Technology: Chatbot Issue: Accuracy/re liability; Mis/disinformation; Safety
- Date it happened
- 2024-03-01
- Organisation involved
- City of New York
- Product, system or model
- MyCity Chatbot
This incident was imported from AIAAIC and is used under CC BY-SA 4.0. Our additions to it — the structured fields, the translation, the checks against other reports — are published under the same licence.
This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.