Study: ChatGPT mostly gets programming questions wrong
ChatGPT wrongly answers over half of software engineering questions it receives, according to a research study. Purdue University researchers analysed how ChatGPT responded to 517 questions posed on Stack Overflow to assess the correctness, consistency, comprehensiveness, and conciseness of the chatbot's answers using linguistic and sentiment analysis, and by questioning a dozen volunteer participants. The analysis showed that 52 percent of ChatGPT answers are incorrect, and 77 percent are verbose. 'Nonetheless', the researchers said, 'ChatGPT answers are still preferred 39.34 percent of the time due to their comprehensiveness and well-articulated language style.' The paper raised questions about ChatGPT's ability to generate high quality engineering information. System 🤖 ChatGPT Operator: Stack Overflow; OpenAI Developer: OpenAI Country: USA Sector: Technology Purpose: Generate text Technology: Chatbot; Generative AI; Machine learning Issue: Accuracy/reliability
- Date it happened
- 2023-08-01
- Organisation involved
- Stack Overflow; OpenAI
- Product, system or model
- ChatGPT
This incident was imported from AIAAIC and is used under CC BY-SA 4.0. Our additions to it — the structured fields, the translation, the checks against other reports — are published under the same licence.
This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.