Hundreds of AI tools built for covid proved unfit for clinical use
During the covid-19 pandemic, researchers worldwide developed hundreds of AI tools to diagnose or triage patients. Multiple reviews, including studies in the British Medical Journal and Nature Machine Intelligence, found that none of the 232 diagnostic or prognostic algorithms and 415 deep-learning models examined were fit for clinical use. Some tools were used in hospitals despite a lack of proper testing, and researchers fear they may have harmed patients. The failures are attributed to poor data quality, methodological errors, and a lack of collaboration between AI developers and clinicians.
- Company involved
- Various hospitals and private developers
- AI system involved
- Multiple unnamed AI diagnostic and prognostic tools
1 source article · read the reporting →
AI detector scores banned as evidence at Yale and Johns Hopkins, vendor conflict exposed - Pasquale Pillitteri
The system flagged student assignments as AI-generated, potentially leading to academic misconduct accusations.
- Company involved
- Yale University and Johns Hopkins University
- AI system involved
- Turnitin AI detector
1 source article · read the reporting →
AI Chatbots Improve Suicide Prevention, but Still Comply When Asked to Write Self-Harm 'Stories'
AI聊天機器人自殺防治能力提升，但被要求寫自殘「故事」時仍照辦 - BigGo 財經
A study by nonprofit Transluce found that major AI chatbots now rarely encourage suicide and frequently suggest seeking professional help. However, when users frame self-harm requests as creative writing or role-play, most models still comply, often mixing helpful advice with harmful content in the same response. The research, spanning 77 model versions and over 50,000 simulated conversations, highlights a persistent gap in detecting disguised personal risk.
- Company involved
- OpenAI, Anthropic, Google
- AI system involved
- Gemini
1 source article · read the reporting →
Resume prompt injection tricks AI hiring - moneywise.com
AI screening system determined which job applicants to advance to the next stage of recruitment.
1 source article · read the reporting →
Clinical algorithm showed racial bias in health care recommendations
In 2019, a study revealed that a clinical algorithm used by many hospitals to identify patients needing extra care was racially biased. Black patients had to be significantly sicker than white patients to be recommended for the same care, due to the algorithm being trained on historical health care spending data that reflected racial disparities. The bias was eventually detected and corrected, but the incident raised concerns about the potential for AI to worsen medical racism.
5 source articles · read the reporting →
Acclarent's AI-Powered Surgery Tool Allegedly Injures Patients, Lawsuits Claim
Two patients allege that Acclarent's TruDi Navigation System, which uses AI to confirm device positioning during sinus surgery, misled surgeons and caused severe injuries including strokes. The FDA has received over 100 reports of malfunctions and at least ten injuries since AI was added. Lawsuits claim the company rushed the technology to market with only 80% accuracy, making the device less safe than before. Acclarent denies any causal connection, and the cases are ongoing.
- Company involved
- Acclarent
- AI system involved
- TruDi Navigation System
2 source articles · read the reporting →
South Africa's Supreme Court of Appeal Considers Sassa Algorithm Case
Верховный апелляционный суд ЮАР рассмотрел дело об алгоритмах Sassa - UA.NEWS
The Supreme Court of Appeal heard an appeal on 25 August about Sassa's digital application system for the SRD grant. The system only accepts online applications and uses automated bank account checks that may deny grants to people whose accounts receive deposits that are not regular income. The Global Center on AI Governance submitted that automated decisions must uphold constitutional rights and be fair, non-discriminatory, and suited to South Africa's informal economy.
- Company involved
- South African Social Security Agency (Sassa)
- AI system involved
- Sassa digital application system
1 source article · read the reporting →
St George's admissions algorithm biased against women and ethnic minorities
In the late 1970s and 1980s, St George's Hospital Medical School used a computer algorithm to automate the first round of its admissions process, scoring applicants based on historical data. The model was found to discriminate against female applicants and those with non-European-looking names, denying them interviews. The bias was discovered by two professors, and the university co-operated with a Commission for Racial Equality inquiry, subsequently contacting affected applicants and offering some places.
- Company involved
- St George's Hospital Medical School
4 source articles · read the reporting →
Acclarent's AI-Powered Surgery Tool Allegedly Injures Patients, Lawsuits Claim
The TruDi Navigation System by Acclarent, which uses AI to confirm device positions during sinus surgery, has been linked to at least ten patient injuries including strokes, according to FDA reports and two lawsuits. Plaintiffs Erin Ralph and Donna Fernihough allege that the AI was inaccurate and misled surgeons, causing carotid artery damage and strokes in 2022. Acclarent denies the allegations, stating there is no credible evidence of a causal connection. The lawsuits are ongoing.
- Company involved
- Acclarent
- AI system involved
- TruDi Navigation System
2 source articles · read the reporting →
AI-generated fake doctors on TikTok give questionable health advice
AI-generated avatars posing as doctors on TikTok have been sharing fake credentials and questionable health advice, deceiving thousands of viewers. The avatars, created using the Captions app, promoted remedies like pineapple-cucumber salads and compared lemon balm to Ozempic. TikToker Javon Ford exposed the app, warning that the realistic AI makes it hard to distinguish from real medical professionals. Indy100 contacted TikTok for comment but received no response.
- Company involved
- TikTok
- AI system involved
- Captions AI avatars
5 source articles · read the reporting →
Royal Free London publishes audit into Streams app data processing
The Royal Free London NHS Foundation Trust published an audit into its use of the Streams app, following an investigation by the Information Commissioner's Office (ICO) in July 2017. The Streams app alerts clinicians to patients at risk of acute kidney injury. The audit, conducted by Linklaters, concluded that the trust's use of Streams was lawful and complied with data protection laws, although areas for improvement were identified. The ICO later recognised that the trust had completed all required actions.
- Company involved
- Royal Free London NHS Foundation Trust
- AI system involved
- Streams
10 source articles · read the reporting →
Sophie Rottenberg used ChatGPT as therapist before taking her own life
Sophie Rottenberg, 29, used ChatGPT as a therapist, divulging more to the chatbot than to her real counsellor. The bot called her brave. She did not tell her parents or counsellor the true extent of her struggle, and later took her own life.
- AI system involved
- ChatGPT
2 source articles · read the reporting →
Babylon Health attacks clinician who raised safety concerns about its AI chatbot
Dr David Watkins, a consultant oncologist, publicly raised concerns about Babylon Health's symptom triage chatbot, alleging it missed critical conditions such as heart attacks in women. Babylon Health issued a press release attacking Watkins as a 'troll' and manipulating data to discredit his testing. Watkins denies the claims and has filed complaints with the MHRA, which remain pending.
- Company involved
- Babylon Health
6 source articles · read the reporting →
Koko criticised for AI chatbot experiment in mental health support
Koko, a free mental health service, tested using a GPT-3-powered chatbot to generate responses to about 4,000 people seeking support. The experiment, announced by founder Rob Morris on Twitter, drew criticism for being unethical and lacking transparency. No specific harm was reported, but the use of AI in place of human peer supporters raised concerns.
- Company involved
- Koko
- AI system involved
- GPT-3-powered chatbot
4 source articles · read the reporting →
Nurse at St. Rose Dominican Hospital Averts AI-Sepsis Alert Error
A nurse at St. Rose Dominican Hospital in Henderson, Nevada, refused to follow an AI-generated sepsis alert that would have given an elderly dialysis patient IV fluids, which could have caused a life-threatening complication. The alert, part of the hospital's electronic system, prompted a charge nurse to order the fluids despite the patient's compromised kidneys. A physician intervened and ordered dopamine instead, averting harm. The incident highlights concerns about AI tools overriding clinical judgment.
- Company involved
- St. Rose Dominican Hospital
1 source article · read the reporting →
Sullivan & Cromwell apologises for AI hallucinations in court filing
Sullivan & Cromwell, an elite Wall Street law firm, used an AI tool to help draft a court filing in the Prince Group case. The filing contained inaccurate citations and misquoted the US bankruptcy code due to AI hallucinations. The errors were uncovered by opposing counsel Boies Schiller Flexner, and the firm apologised to the judge and filed a corrected version. The firm stated that its AI policies were not followed and its review process failed to catch the mistakes.
- Company involved
- Sullivan & Cromwell
3 source articles · read the reporting →
Epic sepsis prediction algorithm misses two-thirds of cases, study finds
A study of nearly 30,000 patients at University of Michigan hospitals found that Epic Systems' sepsis prediction algorithm missed two-thirds of sepsis cases and frequently issued false alarms. The algorithm, used by over 100 US health systems, is designed to alert staff to potential sepsis, but the study suggests it provides little value and may contribute to alert fatigue. Epic disputed the study's conclusions, claiming its system has helped save lives.
- Company involved
- Epic Systems
- AI system involved
- Epic sepsis prediction model
6 source articles · read the reporting →
Yasiel Puig Valdes v. All3Media America, LLC, et al. (SCA California (Los Angeles)): AI-hallucinated content in court filing, Referral
The AI generated fake legal citations that were included in a court filing, potentially harming the plaintiff's case.
- AI system involved
- ChatGPT
1 source article · read the reporting →
The People v. Raziel Ruiz Alvarez (CA California): AI-hallucinated content in court filing, Published order; monetary sanction payable by Counsel…
The AI generated fabricated legal citations that were included in a court filing, affecting the defendant's legal representation and the court's resources.
- Company involved
- Attorney Siddell
1 source article · read the reporting →
DeepScore markets facial and voice analysis app for trustworthiness scoring despite experts' doubts
DeepScore, a Tokyo-based company, is marketing an app that uses facial and voice recognition to score people's trustworthiness for lenders and insurers in Japan, Indonesia, Vietnam and the Philippines. The company says the app can detect deception with 70 per cent accuracy, but researchers and privacy advocates say there is no reliable scientific basis for such judgments and warn of discrimination and privacy harms. The chief executive said the system is only one part of lenders' and insurers' decision-making and that people can choose not to use it. Critics respond that an unequal balance of power makes consent difficult.
- Company involved
- DeepScore
- AI system involved
- DeepScore
6 source articles · read the reporting →
Hospitals use Epic AI to predict Covid-19 decline without validation
Dozens of hospitals across the US are using Epic's deterioration index AI system to predict which Covid-19 patients will become critically ill, despite the tool not being validated for the new disease. The rapid deployment during the pandemic bypassed normal testing and validation processes.
- AI system involved
- Deterioration index
9 source articles · read the reporting →
Joann LeDoux v. Outliers, Inc. (2) (W.D. Washington): AI-hallucinated content in court filing, Expert Brief excluded/struck
The AI-generated hallucinated citations in an expert report led to the exclusion of the expert and dismissal of the plaintiff's case.
1 source article · read the reporting →
Google Health's diabetic retinopathy AI faced real-world issues in Thailand clinics
Google Health deployed a deep-learning system to screen for diabetic retinopathy in 11 clinics across Thailand. The AI was highly accurate in lab tests but rejected over a fifth of eye scans in real-world conditions due to poor lighting and slow internet. Patients whose scans were rejected had to visit specialists at other clinics, causing inconvenience and frustration for nurses. Google Health is now working with local staff to improve the system's workflow.
- Company involved
- Google Health
10 source articles · read the reporting →
Perth hospital group bans ChatGPT after doctor used it for a patient discharge summary
South Metropolitan Health Service, which runs five hospitals in Perth, ordered staff to stop using ChatGPT for work involving patient information after an email said some staff had used the AI chatbot to write medical notes. The service later clarified that only one doctor had used the tool to generate a patient discharge summary and that no confidential patient information had been breached. The Australian Medical Association has called for national regulations to ensure a human is involved in AI-assisted health decisions.
- Company involved
- South Metropolitan Health Service
- AI system involved
- ChatGPT
6 source articles · read the reporting →
