ChatGPT's ability to generate accurate computer code plummets
ChatGPT has become less accurate at generating computer code and other tasks, according to Stanford University and UC Berkeley researchers. Using the March and June 2023 versions of OpenAI's GPT-3 .5 and GPT-4 large language models - which power ChatGPT - on tasks such as maths problem-solving, answering sensitive questions, code generation, and visual reasoning, t he researchers found GPT-4's ability to identify prime numbers declined significantly from an accuracy of 97.6 percent in March to 2.4 percent in June. However, the study's methodology and findings were said to be unconvincing by some researchers. Princeton computer science professor Arvind Narayanan argued that the researchers failed to distinguish between ChatGPT's capabilities, which were acquired through pre-training, and its behaviour, which arises through regular fine-tuning. System 🤖 ChatGPT Operator: Developer: OpenAI Country: Global Sector: Multiple Purpose: Generate computer code Technology: Chatbot; Generative AI; Machine learning Issue: Accuracy/reliability
- Date it happened
- 2023-01-01
- Organisation involved
- OpenAI
- Product, system or model
- ChatGPT; GPT-4; GPT-3.5
This incident was imported from AIAAIC and is used under CC BY-SA 4.0. Our additions to it — the structured fields, the translation, the checks against other reports — are published under the same licence.
This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.