AI models found to generate inaccurate and untrue election info
Over half of answers to questions about 2024's US presidential election were inaccurate, and 40 percent untrue, according to a test of five AI models by experts. A bipartisan group of AI experts from civil society, academia, industry, and journalism were asked to rate responses to questions about the US election put to three closed (Anthropic's Claude, Google's Gemini and OpenAI's GPT-4 ) and two open AI (Meta's Llama 2 and Mistral's Mixtral) models for bias, accuracy, completeness and harmfulness. The report found that the models were prone to suggesting voters head to polling places that do not exist, or inventing illogical responses based on rehashed, outdated information. For example, four of the five chatbots tested wrongly asserted that voters in Nevada would be blocked from registering to vote weeks before Election Day. S ame-day voter registration has been allowed in the state since 2019 . Meta spokesman Daniel Roberts told the AP that the findings were 'meaningless' as they did not exactly mirror the experience a person typically would have with a chatbot. The finding raised concerns about the five systems' potential, and others like them, to mislead and disenfrancise voters, and reduce the quality of election-related information. System 🤖 GPT-4 Gemini Operator: Julia Angwin, Alondra Nelson, Rina Palta Developer: Alphabet/Google, Anthropic, Meta, Mistral; OpenAI Country: USA Sector: Alphabet/Google, Anthropic, Meta, Mistral; OpenAI Purpose: Generate text Technology: Chatbot; NLP/text analysis; Neural network; Deep learning; Machine learning; Reinforcement learning Issue: Accuracy/reliability; Mis/disinformation
- Date it happened
- 2024-02-01
- Organisation involved
- Julia Angwin, Alondra Nelson, Rina Palta
- Product, system or model
- Claude; Gemini; GPT-4; Llama 2; Mixtral
This incident was imported from AIAAIC and is used under CC BY-SA 4.0. Our additions to it — the structured fields, the translation, the checks against other reports — are published under the same licence.
This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.