Study: Devin "AI software engineer" fails at most tasks
The world's "first AI software engineer" was discovered to perform poorly at most things it does, raising major questions about its reliability and the leadership and marketing of the company behind it. What happened Researchers from Answer.AI spent a month testing Devin and reported that out of 20 attempted tasks, 14 failed, 3 were inconclusive, and only 3 succeeded. The AI tool struggled to create new projects, perform research tasks and analyse existing code, and often produced overly complex, unusable solutions and spent days pursuing impossible tasks. It also failed to interpret user commands correctly and continued running, producing errors in the output. Specifically, when asked to deploy multiple applications to deployment platform Railway, Devin failed to recognise this wasn't possible and spent over a day attempting nonviable approaches. The tool also became trapped in endless cycles of trying to parse HTML when given web scraping tasks . Furthermore, security reviews resulted in numerous false positives and hallucinated issues. Why it happened The huge gap between Devin's promised and actual capabilities indicates inadequate testing before release. It also may reflect pressure from investors - which include Founders Fund and Khosla Ventures - for quick returns from Devin developer and owner Cognition AI. What it means Answer AI's study raises serious questions about Devin's reliability and effectiveness as a product. It also reflects poorly on Cognition AI's leadership, which is seen to have massively overhyped and misrepresented the tool's capabilities, thereby casting doubt on the company's integrity and approach to transparency, and damaging its credibility. More broadly, the findings highlight the current limitations of fully autonomous AI software and suggests AI tools may work better as assistants rather than as replacements. System 🤖 Devin 🔗 Developer: Cognition AI Country: Global Sector: Technology Purpose: Develop software Technology: Bot/intelligent agent Issue: Accuracy/reliability; Transparency
- Date it happened
- 2025-01-01
- Organisation involved
- Answer AI
- Product, system or model
- Devin
This incident was imported from AIAAIC and is used under CC BY-SA 4.0. Our additions to it — the structured fields, the translation, the checks against other reports — are published under the same licence.
This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.