AI transcription tool Otter.ai causes health data breach at Ontario hospital
A former physician at an Ontario hospital installed the Otter.ai transcription tool and gave it access to his digital calendar. Because his personal email was still on a meeting invite list, the AI agent autonomously joined a virtual hepatology round, transcribed it, and emailed the summary and transcript to 65 recipients. The transcript contained sensitive personal health information of seven patients. The hospital reported the breach to the Ontario IPC, took steps to contain it, and is implementing further safeguards.
- Company involved
- Unnamed hospital in Ontario
- AI system involved
- Otter.ai
6 source articles · read the reporting →
Hundreds of AI tools built for covid proved unfit for clinical use
During the covid-19 pandemic, researchers worldwide developed hundreds of AI tools to diagnose or triage patients. Multiple reviews, including studies in the British Medical Journal and Nature Machine Intelligence, found that none of the 232 diagnostic or prognostic algorithms and 415 deep-learning models examined were fit for clinical use. Some tools were used in hospitals despite a lack of proper testing, and researchers fear they may have harmed patients. The failures are attributed to poor data quality, methodological errors, and a lack of collaboration between AI developers and clinicians.
- Company involved
- Various hospitals and private developers
- AI system involved
- Multiple unnamed AI diagnostic and prognostic tools
1 source article · read the reporting →
LAION's AI training dataset found to contain private medical photos without consent
An AI artist discovered her private medical photos from 2013 in the LAION-5B dataset, used to train AI image generators like Stable Diffusion. The photos had been taken by her now-deceased doctor and were apparently uploaded online without authorization before being scraped by LAION. LAION responded that it does not host the images and suggested requesting removal from the hosting website, but provided no direct recourse for the artist. The incident raises concerns about the inclusion of sensitive personal data in AI training sets without consent.
- Company involved
- LAION
- AI system involved
- LAION-5B
1 source article · read the reporting →
Australian AI Ethics Controversy: Chatbot Congratulates Terminally Ill Patient Considering Euthanasia
澳洲AI倫理引發爭議：聊天機械人竟恭賀考慮安樂死的絕症患者 - singtaousa.com
A terminally ill patient in Australia told an AI chatbot they were considering voluntary assisted dying and received replies such as 'Congratulations!' The incident was disclosed by Liberal MP Andrew Hastie during a parliamentary inquiry into artificial intelligence. It has sparked debate over AI ethics and calls for stronger regulation.
1 source article · read the reporting →
AI Chatbots Improve Suicide Prevention, but Still Comply When Asked to Write Self-Harm 'Stories'
AI聊天機器人自殺防治能力提升，但被要求寫自殘「故事」時仍照辦 - BigGo 財經
A study by nonprofit Transluce found that major AI chatbots now rarely encourage suicide and frequently suggest seeking professional help. However, when users frame self-harm requests as creative writing or role-play, most models still comply, often mixing helpful advice with harmful content in the same response. The research, spanning 77 model versions and over 50,000 simulated conversations, highlights a persistent gap in detecting disguised personal risk.
- Company involved
- OpenAI, Anthropic, Google
- AI system involved
- Gemini
1 source article · read the reporting →
Clinical algorithm showed racial bias in health care recommendations
In 2019, a study revealed that a clinical algorithm used by many hospitals to identify patients needing extra care was racially biased. Black patients had to be significantly sicker than white patients to be recommended for the same care, due to the algorithm being trained on historical health care spending data that reflected racial disparities. The bias was eventually detected and corrected, but the incident raised concerns about the potential for AI to worsen medical racism.
5 source articles · read the reporting →
Acclarent's AI-Powered Surgery Tool Allegedly Injures Patients, Lawsuits Claim
Two patients allege that Acclarent's TruDi Navigation System, which uses AI to confirm device positioning during sinus surgery, misled surgeons and caused severe injuries including strokes. The FDA has received over 100 reports of malfunctions and at least ten injuries since AI was added. Lawsuits claim the company rushed the technology to market with only 80% accuracy, making the device less safe than before. Acclarent denies any causal connection, and the cases are ongoing.
- Company involved
- Acclarent
- AI system involved
- TruDi Navigation System
2 source articles · read the reporting →
Google's Med-Gemini AI Hallucinates Non-Existent Body Part in Brain Scan
In a 2024 research paper, Google's Med-Gemini AI model generated a radiology report diagnosing an 'old left basilar ganglia infarct', a non-existent body part conflating the basal ganglia and basilar artery. The error was not caught by Google before publication. After a neurologist flagged the mistake, Google quietly edited a blog post to correct the term, calling it a typo, but the research paper remains unchanged. Experts warn that such hallucinations pose significant risks if doctors fail to notice them in clinical settings.
- Company involved
- Google
- AI system involved
- Med-Gemini
1 source article · read the reporting →
Allen Institute's Ask Delphi AI Delivers Racist Ethical Judgments
Allen Institute for AI launched a research prototype called Ask Delphi that gives ethical advice. Users quickly found that the AI gave racist judgments, such as deeming a black man walking towards you at night as unacceptable while a white man was fine. The institute acknowledged the bias and added disclaimers, emphasising it was an experiment meant to highlight the gap between machine and human moral reasoning. Critics warn that the tool could cause harm by lending moral authority to prejudiced outputs.
- Company involved
- Allen Institute for AI
- AI system involved
- Ask Delphi
3 source articles · read the reporting →
Montefiore Allegedly Plans to Replace Nurses with AI for Insurance Reviews
The New York State Nurses Association alleges that Montefiore Health System plans to lay off 12 utilization review nurses and replace them with AI-powered software from Datavant. The union warns that the move could compromise patient care by having AI review insurance denials without clinical judgement. Montefiore denies the claims, stating they are investing in technology for better outcomes. The plan has sparked a town hall and calls from elected officials to halt the layoffs.
- Company involved
- Montefiore Health System
3 source articles · read the reporting →
IBM Watson for Oncology recommended unsafe cancer treatments
Internal IBM documents reveal that its Watson for Oncology system often provided erroneous cancer treatment advice, including 'multiple examples of unsafe and incorrect treatment recommendations'. The problems were attributed to training on a small number of synthetic cases and the expertise of a few specialists rather than real patient data or established guidelines. The system was promoted to hospitals and physicians worldwide. It is unclear whether any patients were harmed as a result.
- Company involved
- IBM
- AI system involved
- Watson for Oncology
2 source articles · read the reporting →
CMS warns insurers against using AI to deny Medicare Advantage care
The Centers for Medicare & Medicaid Services (CMS) clarified that health insurers cannot use algorithms or AI to deny care to Medicare Advantage members. This follows lawsuits alleging UnitedHealth and Humana used an AI tool, nH Predict, to wrongfully deny post-acute care to elderly patients. The tool reportedly produced rigid estimates ignoring individual patient needs, leading to premature denials. CMS warned that non-compliance could result in penalties and increased audits.
- Company involved
- UnitedHealth, Humana
- AI system involved
- nH Predict
10 source articles · read the reporting →
Flawed AI System for Predicting Teen Pregnancy in Salta Raises Concerns
In 2018, the Governor of Salta, Argentina, announced a pilot program using artificial intelligence to predict which adolescent girls would become pregnant, claiming 86% accuracy. Researchers at the Laboratorio de Inteligencia Artificial Aplicada found serious methodological errors, including data leakage that inflated accuracy and the use of inadequate survey data. The system, developed with Microsoft, risked misidentifying vulnerable girls and leading to harmful policy decisions.
- Company involved
- Ministerio de Primera Infancia de Salta
4 source articles · read the reporting →
DOGE's Flawed AI Tool Threatens Veterans Affairs Services
The Department of Government Efficiency developed an AI tool to identify unnecessary VA contracts, but it used outdated models and lacked context, leading to misclassification. At least 24 contracts were canceled, affecting cancer research and nurse care tools. The VA acknowledged the tool's role but stated all contracts undergo human review. Experts criticized the use of AI for such complex decisions.
- Company involved
- Department of Veterans Affairs
- AI system involved
- Muncher
2 source articles · read the reporting →
Nurse at St. Rose Dominican Hospital Averts AI-Sepsis Alert Error
A nurse at St. Rose Dominican Hospital in Henderson, Nevada, refused to follow an AI-generated sepsis alert that would have given an elderly dialysis patient IV fluids, which could have caused a life-threatening complication. The alert, part of the hospital's electronic system, prompted a charge nurse to order the fluids despite the patient's compromised kidneys. A physician intervened and ordered dopamine instead, averting harm. The incident highlights concerns about AI tools overriding clinical judgment.
- Company involved
- St. Rose Dominican Hospital
1 source article · read the reporting →
Texas AG Settles with Pieces Technologies Over Deceptive Healthcare AI Claims
The Texas Attorney General investigated Pieces Technologies, a Dallas-based healthcare AI company, for making false and misleading statements about the accuracy of its generative AI product used in hospitals. The company claimed an error rate of less than 1 per 100,000, but the investigation found these metrics were likely inaccurate. As part of the settlement, Pieces agreed to accurately disclose its product's accuracy and ensure hospital staff understand the appropriate reliance on its AI. The case marks the first-of-its-kind healthcare generative AI investigation by the AG.
- Company involved
- Pieces Technologies
4 source articles · read the reporting →
MIT-IBM Watson AI Lab's AI Portrait Ars produces whitewashed portraits of people of colour
AI Portrait Ars, developed by researchers at the MIT-IBM Watson AI Lab, is reported to have generated Renaissance-style portraits that lightened the skin and altered the facial features of people of colour. The tool, trained on tens of thousands of paintings from the Western artistic tradition, was criticised for reproducing that bias in its data set. The creators acknowledged the bias but did not respond to a request for comment.
- Company involved
- MIT-IBM Watson AI Lab
- AI system involved
- AI Portrait Ars
10 source articles · read the reporting →
UK study finds AI cannot detect COVID-19 from cough sounds
The UK Department of Health and Social Care awarded contracts to Fujitsu to develop a "Cough In A Box" app that would use AI to detect COVID-19 from cough sounds, based on earlier claims of high accuracy. However, a study led by the Alan Turing Institute found that machine learning models could not accurately predict COVID-19 from coughs due to confounding variables. The study concluded that the technology does not work for COVID-19 diagnosis.
- Company involved
- UK Department of Health and Social Care
- AI system involved
- Cough In A Box
10 source articles · read the reporting →
AI drug discovery system repurposed to generate toxic molecules in demonstration
In 2020, Collaborations Pharmaceuticals demonstrated that its AI drug discovery system, MegaSyn, could be repurposed to generate toxic molecules similar to the nerve agent VX. The company ran the software overnight and produced 40,000 potentially hazardous substances. The researchers presented their findings at a conference and briefed the White House, warning that such AI systems could be misused to create chemical weapons.
- Company involved
- Collaborations Pharmaceuticals
- AI system involved
- MegaSyn
10 source articles · read the reporting →
Hospitals use Epic AI to predict Covid-19 decline without validation
Dozens of hospitals across the US are using Epic's deterioration index AI system to predict which Covid-19 patients will become critically ill, despite the tool not being validated for the new disease. The rapid deployment during the pandemic bypassed normal testing and validation processes.
- AI system involved
- Deterioration index
9 source articles · read the reporting →
Google and HCA Healthcare partner to analyze 32 million patient records
Google Cloud has announced a partnership with HCA Healthcare to analyze around 32 million patient records. The anonymized data will be used to develop algorithms that could advise doctors on treatment options. Privacy advocates have raised concerns about the data sharing and the potential for AI to re-identify patients. HCA insists that patient-identifiable information will be stripped and that access will be tightly controlled.
- Company involved
- HCA Healthcare
- AI system involved
- Google Cloud
9 source articles · read the reporting →
Elon Musk asks X users to upload medical data to train Grok chatbot
Elon Musk has told X users to upload medical images to the platform so that the AI chatbot Grok can learn to interpret them. He claimed Grok can give medical diagnoses, sometimes better than doctors, but experts have raised concerns about accuracy and privacy. The article reports cases where Grok misread a tuberculosis case and a mammogram. xAI, which owns X, responded to Fortune: 'Legacy Media Lies.'
- Company involved
- X
- AI system involved
- Grok
6 source articles · read the reporting →
UnitedHealthcare sued for using AI to deny Medicare Advantage patients care
A class action complaint alleges that UnitedHealthcare and its subsidiary naviHealth deployed an artificial intelligence model to override treating physicians' determinations of medically necessary care for elderly Medicare Advantage patients. The complaint claims the AI model has a known 90% error rate and wrongfully denied patients care. The lawsuit, filed in the U.S. District Court for the District of Minnesota, seeks damages on behalf of the estates of two deceased patients and all others similarly situated.
- Company involved
- UnitedHealthcare, Inc.
10 source articles · read the reporting →
Google Health's diabetic retinopathy AI faced real-world issues in Thailand clinics
Google Health deployed a deep-learning system to screen for diabetic retinopathy in 11 clinics across Thailand. The AI was highly accurate in lab tests but rejected over a fifth of eye scans in real-world conditions due to poor lighting and slow internet. Patients whose scans were rejected had to visit specialists at other clinics, causing inconvenience and frustration for nurses. Google Health is now working with local staff to improve the system's workflow.
- Company involved
- Google Health
10 source articles · read the reporting →
