Hundreds of AI tools built for covid proved unfit for clinical use
During the covid-19 pandemic, researchers worldwide developed hundreds of AI tools to diagnose or triage patients. Multiple reviews, including studies in the British Medical Journal and Nature Machine Intelligence, found that none of the 232 diagnostic or prognostic algorithms and 415 deep-learning models examined were fit for clinical use. Some tools were used in hospitals despite a lack of proper testing, and researchers fear they may have harmed patients. The failures are attributed to poor data quality, methodological errors, and a lack of collaboration between AI developers and clinicians.
- Company involved
- Various hospitals and private developers
- AI system involved
- Multiple unnamed AI diagnostic and prognostic tools
1 source article · read the reporting →
Cigna uses algorithm to deny medical claims without doctor review
Cigna, one of the largest health insurers, used an algorithm called PXDX to automatically deny medical claims. The system flagged mismatches between diagnoses and procedures, and company doctors signed off on denials in batches without reviewing patient files. One patient, Nick van Terheyden, was denied coverage for a $350 vitamin D test that his doctor had ordered. Former employees said the system saved Cigna billions of dollars but left patients with unexpected bills.
- Company involved
- Cigna
- AI system involved
- PXDX
10 source articles · read the reporting →
What a $150,000 scam reveals about AI and elder fraud - Rolling Out
AI-generated deepfakes and voice cloning convinced a 63-year-old woman to send $150,000 to a scammer posing as a romantic partner
1 source article · read the reporting →
Australian Telco Loses $11 Million After AI Bot Misfires
In early 2018, an Australian telecommunications company deployed an AI bot to handle network incidents, expecting to cut operational costs by 25%. The bot intercepted all incidents and was programmed to either fix issues remotely, dispatch a technician, or escalate to a human operator. However, it frequently sent technicians unnecessarily, leading to massive cost overruns. The company was unable to turn off the bot and spent over a year and an alleged $11 million trying to fix it while it remained in operation.
1 source article · read the reporting →
Resume prompt injection tricks AI hiring - moneywise.com
AI screening system determined which job applicants to advance to the next stage of recruitment.
1 source article · read the reporting →
Rotterdam's Welfare Fraud Algorithm Discriminated by Gender and Ethnicity
The city of Rotterdam deployed a machine learning algorithm built by Accenture to flag welfare recipients for fraud investigation. The system used personal data including gender, language, and subjective caseworker notes to generate risk scores, leading to investigations that disproportionately targeted women and migrants. An external review found the algorithm discriminatory and inaccurate, prompting the city to suspend its use in 2021. The system's opacity made it nearly impossible for those flagged to challenge the decisions.
- Company involved
- City of Rotterdam
6 source articles · read the reporting →
SEC Charges Delphia and Global Predictions for False AI Claims
The SEC charged Delphia (USA) Inc. and Global Predictions Inc. for making false and misleading statements about their use of artificial intelligence. Delphia claimed from 2019 to 2023 that it used AI and machine learning to predict investments, while Global Predictions falsely claimed in 2023 to be the 'first regulated AI financial advisor'. Both firms settled the charges without admitting or denying the findings, agreeing to pay a total of $400,000 in civil penalties and to cease and desist from further violations.
- Company involved
- Delphia (USA) Inc. and Global Predictions Inc.
1 source article · read the reporting →
Michigan's MiDAS system falsely accused 34,000 of unemployment fraud
Michigan's Unemployment Insurance Agency (UIA) deployed the Michigan Integrated Data Automated System (MiDAS) to automate fraud detection. The system algorithmically adjudicated over 40,000 fraud cases between October 2013 and September 2015, with 85 percent of those resulting in incorrect fraud determinations. The false accusations led to severe financial penalties, ruined credit, and homelessness for many victims. A class-action lawsuit was filed, and while the state has apologised and returned fines, it has fought against compensation claims, though legislators have promised to seek at least $30 million for those affected.
- Company involved
- Michigan Unemployment Insurance Agency
- AI system involved
- Michigan Integrated Data Automated System (MiDAS)
10 source articles · read the reporting →
AI deepfakes of doctors used to scam diabetes patients in Australia
Scammers used AI-generated deepfake videos and fake interviews impersonating diabetes expert Professor Jonathan Shaw and other health professionals to promote a dietary supplement called Glyco Balance. The fraudulent ads falsely claimed the supplement, sold by Vellec Group, was developed at the Baker Institute and that commonly prescribed diabetes medication is dangerous. One victim, Michael, purchased the supplement for $340 before realising it was a scam. The Therapeutic Goods Administration is assessing the situation, and Meta removed the fake video after the Baker Institute reported it.
- Company involved
- Vellec Group
- AI system involved
- Generative AI for deepfake video creation
1 source article · read the reporting →
Portland Water Bureau AI pilot offers discount to billionaire CEO
The Portland Water Bureau used a machine learning system in a randomised control trial to identify customers for its financial assistance programme. The system selected Columbia Sportswear CEO Tim Boyle, a billionaire and one of the city's largest residential water consumers, for a 40% water bill discount. Boyle declined the discount, stating it should go to someone who needs it. The bureau is testing whether the SERVUS algorithm can accurately identify customers' ability to pay.
- Company involved
- Portland Water Bureau
- AI system involved
- Smart Discount Program
1 source article · read the reporting →
AI chatbots found giving inaccurate financial advice to UK consumers
A Which? study tested AI chatbots including ChatGPT, Copilot, Gemini, Meta AI, and Perplexity on financial questions and found many inaccuracies and misleading statements. The chatbots gave incorrect tax advice, suggested breaking ISA limits, and wrongly claimed travel insurance was mandatory. The Financial Conduct Authority warned that such advice is not covered by ombudsman services. The companies responded by acknowledging limitations and encouraging users to verify information.
- Company involved
- Meta, OpenAI, Microsoft, Google, Perplexity
- AI system involved
- Meta AI, ChatGPT, Copilot, Gemini, Perplexity
1 source article · read the reporting →
AI-generated Centrelink phishing emails target 270,000 Australians
More than 270,000 fake emails impersonating Services Australia and Centrelink were detected over four months in a broad phishing campaign. Cybersecurity firm Mimecast reports that cybercriminals are using artificial intelligence to create highly convincing clones of legitimate government communications about benefits. The attack targets vulnerable people and can lead to identity theft, data theft, malware, or ransomware.
- Company involved
- Services Australia
4 source articles · read the reporting →
IBM Watson for Oncology recommended unsafe cancer treatments
Internal IBM documents reveal that its Watson for Oncology system often provided erroneous cancer treatment advice, including 'multiple examples of unsafe and incorrect treatment recommendations'. The problems were attributed to training on a small number of synthetic cases and the expertise of a few specialists rather than real patient data or established guidelines. The system was promoted to hospitals and physicians worldwide. It is unclear whether any patients were harmed as a result.
- Company involved
- IBM
- AI system involved
- Watson for Oncology
2 source articles · read the reporting →
CMS warns insurers against using AI to deny Medicare Advantage care
The Centers for Medicare & Medicaid Services (CMS) clarified that health insurers cannot use algorithms or AI to deny care to Medicare Advantage members. This follows lawsuits alleging UnitedHealth and Humana used an AI tool, nH Predict, to wrongfully deny post-acute care to elderly patients. The tool reportedly produced rigid estimates ignoring individual patient needs, leading to premature denials. CMS warned that non-compliance could result in penalties and increased audits.
- Company involved
- UnitedHealth, Humana
- AI system involved
- nH Predict
10 source articles · read the reporting →
Flawed AI System for Predicting Teen Pregnancy in Salta Raises Concerns
In 2018, the Governor of Salta, Argentina, announced a pilot program using artificial intelligence to predict which adolescent girls would become pregnant, claiming 86% accuracy. Researchers at the Laboratorio de Inteligencia Artificial Aplicada found serious methodological errors, including data leakage that inflated accuracy and the use of inadequate survey data. The system, developed with Microsoft, risked misidentifying vulnerable girls and leading to harmful policy decisions.
- Company involved
- Ministerio de Primera Infancia de Salta
4 source articles · read the reporting →
DOGE's Flawed AI Tool Threatens Veterans Affairs Services
The Department of Government Efficiency developed an AI tool to identify unnecessary VA contracts, but it used outdated models and lacked context, leading to misclassification. At least 24 contracts were canceled, affecting cancer research and nurse care tools. The VA acknowledged the tool's role but stated all contracts undergo human review. Experts criticized the use of AI for such complex decisions.
- Company involved
- Department of Veterans Affairs
- AI system involved
- Muncher
2 source articles · read the reporting →
Deloitte to refund Australian government after AI-generated report errors
Deloitte used generative AI (Azure OpenAI GPT-4o) to help produce an independent assurance review for Australia's Department of Employment and Workplace Relations. The report, published in July 2025, contained multiple errors including non-existent academic references and a fabricated court case. After the errors were flagged, Deloitte acknowledged the AI use and agreed to refund the final instalment of the A$439,000 contract. The report was corrected, but its substance and recommendations remained unchanged.
- Company involved
- Deloitte
- AI system involved
- Azure OpenAI GPT-4o
5 source articles · read the reporting →
Texas AG Settles with Pieces Technologies Over Deceptive Healthcare AI Claims
The Texas Attorney General investigated Pieces Technologies, a Dallas-based healthcare AI company, for making false and misleading statements about the accuracy of its generative AI product used in hospitals. The company claimed an error rate of less than 1 per 100,000, but the investigation found these metrics were likely inaccurate. As part of the settlement, Pieces agreed to accurately disclose its product's accuracy and ensure hospital staff understand the appropriate reliance on its AI. The case marks the first-of-its-kind healthcare generative AI investigation by the AG.
- Company involved
- Pieces Technologies
4 source articles · read the reporting →
KPMG drops AI report after false case studies exposed
KPMG removed a global report on AI after multiple case studies were found to be inaccurate and apparently generated from AI hallucinations. The report falsely claimed that UBS, NHS Greater Manchester, Swiss Federal Railways and Transport for London used AI agents in various ways. The companies denied the claims, and KPMG took the report down while reviewing its production.
- Company involved
- KPMG
3 source articles · read the reporting →
Walmart employees claim Everseen anti-theft AI is flawed
A group of Walmart employees, calling themselves the Concerned Home Office Associates, alleged that the Everseen AI system used in thousands of stores to prevent shoplifting frequently produces false positives, flagging innocent customer behaviour as theft. They created a video showing the system failing to stop actual theft and mistakenly alerting for a phone. Employees said the false positives frustrate customers and lead to unnecessary interventions, raising public health concerns during the COVID-19 pandemic. Walmart denied a widespread issue but said it had made improvements.
- Company involved
- Walmart
- AI system involved
- Everseen
9 source articles · read the reporting →
UIUC researchers use OpenAI API to automate phone scams for under a dollar
Researchers at the University of Illinois Urbana-Champaign used OpenAI's Realtime API to create AI agents that can autonomously execute phone scams. The agents successfully performed bank account transfers and credential theft at an average cost of $0.75 per scam. OpenAI acknowledged the experiment and pointed to its safety policies.
- Company involved
- University of Illinois Urbana-Champaign
- AI system involved
- GPT-4o Realtime API
6 source articles · read the reporting →
Epic sepsis prediction algorithm misses two-thirds of cases, study finds
A study of nearly 30,000 patients at University of Michigan hospitals found that Epic Systems' sepsis prediction algorithm missed two-thirds of sepsis cases and frequently issued false alarms. The algorithm, used by over 100 US health systems, is designed to alert staff to potential sepsis, but the study suggests it provides little value and may contribute to alert fatigue. Epic disputed the study's conclusions, claiming its system has helped save lives.
- Company involved
- Epic Systems
- AI system involved
- Epic sepsis prediction model
6 source articles · read the reporting →
Google AI Overviews provide inaccurate finance information in 43% of searches
A study by The College Investor found that Google's AI Overviews provided misleading or inaccurate information in 43% of 100 personal finance-related searches. The AI-generated answers included outdated tax rules and incorrect student loan repayment plan details. One user believed she could convert her 529 plan to a Roth IRA in California, which is not allowed. Google has not responded to requests for comment.
- Company involved
- Google
- AI system involved
- AI Overviews
3 source articles · read the reporting →
Hospitals use Epic AI to predict Covid-19 decline without validation
Dozens of hospitals across the US are using Epic's deterioration index AI system to predict which Covid-19 patients will become critically ill, despite the tool not being validated for the new disease. The rapid deployment during the pandemic bypassed normal testing and validation processes.
- AI system involved
- Deterioration index
9 source articles · read the reporting →
