Epic sepsis prediction algorithm misses two-thirds of cases, study finds
A study of nearly 30,000 patients at University of Michigan hospitals found that Epic Systems' sepsis prediction algorithm missed two-thirds of sepsis cases and frequently issued false alarms. The algorithm, used by over 100 US health systems, is designed to alert staff to potential sepsis, but the study suggests it provides little value and may contribute to alert fatigue. Epic disputed the study's conclusions, claiming its system has helped save lives.
- Company involved
- Epic Systems
- AI system involved
- Epic sepsis prediction model
6 source articles · read the reporting →
NSW Education Standards Authority used AI-generated image in HSC English exam without disclosure
The NSW Education Standards Authority (NESA) used an AI-generated image as a stimulus in the 2024 HSC English exam without disclosing its origin. The image, created by Florian Schroeder using OpenAI's ChatGPT and Dall-E 2, was published on Medium in July 2023. Students suspected AI use due to irregularities in the image, and NESA initially declined to confirm. After the Sydney Morning Herald confirmed the image was AI-generated, NESA stated that students would be marked on their response to the question, not the image's origin.
- Company involved
- NSW Education Standards Authority
- AI system involved
- ChatGPT and Dall-E 2
6 source articles · read the reporting →
PimEyes face search engine found photos of journalists without consent
The New York Times tested PimEyes, a face search engine, by submitting photos of a dozen employees. The system returned photos of the journalists from various sources, including ones they had never seen, even when faces were obscured. The system also misidentified women journalists as possibly being on adult sites. German authorities are currently investigating PimEyes over privacy concerns.
- Company involved
- PimEyes
- AI system involved
- PimEyes
2 source articles · read the reporting →
AI drug discovery system repurposed to generate toxic molecules in demonstration
In 2020, Collaborations Pharmaceuticals demonstrated that its AI drug discovery system, MegaSyn, could be repurposed to generate toxic molecules similar to the nerve agent VX. The company ran the software overnight and produced 40,000 potentially hazardous substances. The researchers presented their findings at a conference and briefed the White House, warning that such AI systems could be misused to create chemical weapons.
- Company involved
- Collaborations Pharmaceuticals
- AI system involved
- MegaSyn
10 source articles · read the reporting →
Qoves facial assessment tool ranks journalist's attractiveness and suggests procedures
Tate Ryan-Mosley, a journalist at MIT Technology Review, used Qoves Studio's AI facial assessment tool, which automatically scored her face and listed perceived flaws such as smile lines and under-eye bags. The tool recommended skin-care products and paid surgical consultation reports. She also tried Megvii's Face++ beauty-scoring system, which gave her two percentage attractiveness scores. The article raises concerns that such algorithms are inaccurate, biased and not transparently used by social media platforms.
- Company involved
- Qoves Studio
- AI system involved
- Facial Assessment Tool
9 source articles · read the reporting →
Italian police facial recognition system SARI uses 16 million mugshots, raising legality questions
The Italian State Police have been using a facial recognition system called SARI since July 2018, based on a database of 16 million mugshots. The system has helped in arrests but has raised civil rights concerns because nine million of the mugshots belong to people stopped only once. A parliamentary question has been filed seeking more details, but has not yet been answered. The Italian Data Protection Authority has found the system compliant with the law.
- Company involved
- Italian State Police
- AI system involved
- SARI
10 source articles · read the reporting →
Google's dermatology app underrepresents dark skin tones in training data
Google unveiled a dermatology app that uses deep learning to recognise skin conditions. However, the training data included only 3.5% images from patients with darker skin types, which may lead to inaccurate diagnoses for these groups. Dermatologists have expressed concern that the app could cause false positives or missed diagnoses, and Google has acknowledged the issue while saying it will continue to refine the technology.
- Company involved
- Google
- AI system involved
- AI-powered dermatology assist tool
9 source articles · read the reporting →
Hospitals use Epic AI to predict Covid-19 decline without validation
Dozens of hospitals across the US are using Epic's deterioration index AI system to predict which Covid-19 patients will become critically ill, despite the tool not being validated for the new disease. The rapid deployment during the pandemic bypassed normal testing and validation processes.
- AI system involved
- Deterioration index
9 source articles · read the reporting →
Google and HCA Healthcare partner to analyze 32 million patient records
Google Cloud has announced a partnership with HCA Healthcare to analyze around 32 million patient records. The anonymized data will be used to develop algorithms that could advise doctors on treatment options. Privacy advocates have raised concerns about the data sharing and the potential for AI to re-identify patients. HCA insists that patient-identifiable information will be stripped and that access will be tightly controlled.
- Company involved
- HCA Healthcare
- AI system involved
- Google Cloud
9 source articles · read the reporting →
Elon Musk asks X users to upload medical data to train Grok chatbot
Elon Musk has told X users to upload medical images to the platform so that the AI chatbot Grok can learn to interpret them. He claimed Grok can give medical diagnoses, sometimes better than doctors, but experts have raised concerns about accuracy and privacy. The article reports cases where Grok misread a tuberculosis case and a mammogram. xAI, which owns X, responded to Fortune: 'Legacy Media Lies.'
- Company involved
- X
- AI system involved
- Grok
6 source articles · read the reporting →
Google Health's diabetic retinopathy AI faced real-world issues in Thailand clinics
Google Health deployed a deep-learning system to screen for diabetic retinopathy in 11 clinics across Thailand. The AI was highly accurate in lab tests but rejected over a fifth of eye scans in real-world conditions due to poor lighting and slow internet. Patients whose scans were rejected had to visit specialists at other clinics, causing inconvenience and frustration for nurses. Google Health is now working with local staff to improve the system's workflow.
- Company involved
- Google Health
10 source articles · read the reporting →
Perth hospital group bans ChatGPT after doctor used it for a patient discharge summary
South Metropolitan Health Service, which runs five hospitals in Perth, ordered staff to stop using ChatGPT for work involving patient information after an email said some staff had used the AI chatbot to write medical notes. The service later clarified that only one doctor had used the tool to generate a patient discharge summary and that no confidential patient information had been breached. The Australian Medical Association has called for national regulations to ensure a human is involved in AI-assisted health decisions.
- Company involved
- South Metropolitan Health Service
- AI system involved
- ChatGPT
6 source articles · read the reporting →
Four commercial large language models perpetuate race-based medical misconceptions
A study published in npj Digital Medicine tested four commercial large language models (Bard, ChatGPT, GPT-4, and Claude) for their tendency to propagate discredited race-based medical beliefs. When asked about kidney function, lung capacity, and skin thickness, the models sometimes endorsed debunked racial differences, particularly affecting Black patients. The study concludes that these biases pose a potential hazard and urges caution before using such models in clinical decision-making.
- Company involved
- Not named in article (refers to commercial LLMs generically as Google's Bard, OpenAI's ChatGPT and GPT-4, and Anthropic's Claude)
- AI system involved
- Bard, ChatGPT, GPT-4, Claude
6 source articles · read the reporting →
MrDeepFakes hosted non-consensual deepfake images of journalist Patrizia Schlosser
Patrizia Schlosser, a German investigative journalist, found more than 30 pornographic images of herself on MrDeepFakes, a website hosting non-consensual deepfake content. The images had been online for almost two years before being removed in January 2025. Bellingcat's investigation linked MrDeepFakes to apps including Deepswap and Candy.ai, which were advertised on the site. MrDeepFakes did not respond to requests for comment.
- Company involved
- MrDeepFakes
- AI system involved
- MrDeepFakes
1 source article · read the reporting →
ChatGPT generates error-filled cancer treatment plans, study finds
A study by researchers at Brigham and Women's Hospital found that ChatGPT, developed by OpenAI, generated cancer treatment plans with errors. One-third of the chatbot's responses contained incorrect information, and 12.5% were hallucinated. The study was published in JAMA Oncology and reported by Bloomberg.
- Company involved
- OpenAI
- AI system involved
- ChatGPT
10 source articles · read the reporting →
ChatGPT 3.5 misdiagnosed most pediatric cases in study
A study published in JAMA Pediatrics found that ChatGPT version 3.5 provided incorrect diagnoses for 83 out of 100 pediatric case challenges. The chatbot's errors included both completely wrong diagnoses and diagnoses that were too broad. The researchers noted that the chatbot failed to identify relationships such as that between autism and vitamin deficiencies, and suggested that more selective training is needed to improve accuracy.
- Company involved
- OpenAI
- AI system involved
- ChatGPT version 3.5
7 source articles · read the reporting →
PimEyes scraped Ancestry.com for photos of dead people without permission.
PimEyes, a facial recognition search engine, scraped images from Ancestry.com and other websites without authorization, including photos of deceased individuals. The company's algorithms indexed the images to create biometric faceprints, potentially allowing identification of living relatives. Cher Scarlett discovered the scraping after finding photos of her deceased sister on the platform. PimEyes has since blocked Ancestry's domain and is removing the indexes, but privacy concerns remain.
- Company involved
- PimEyes
- AI system involved
- PimEyes
7 source articles · read the reporting →
Frontiers retracts study with AI-generated rat penis images
A study published in the journal Frontiers in Cell Development and Biology was retracted after it was discovered that the authors used the AI tool Midjourney to generate nonsensical images, including a giant rat penis. The journal apologised and said a reviewer's concerns were ignored. The incident highlights issues with AI use in academic publishing.
- AI system involved
- Midjourney
8 source articles · read the reporting →
PimEyes faces fine proceedings over biometric facial recognition in Baden-Württemberg
PimEyes, a facial recognition search engine, is accused of scraping images from the internet and processing biometric data without a valid legal basis under the GDPR. The data protection authority of Baden-Württemberg (LfDI) opened fine proceedings after it found PimEyes's response to its questions inadequate. PimEyes argues that the images it processes are publicly available and not personal data. The LfDI says the processing endangers citizens' rights and freedoms and is not covered by the GDPR exceptions.
- Company involved
- PimEyes
- AI system involved
- PimEyes
9 source articles · read the reporting →
Council Rock student investigated for creating AI deepfakes of classmates
A middle school student in the Council Rock School District allegedly used AI to create pornographic deepfake images of up to 12 underage girls. The incident occurred in March and was reported to Newtown Township police, who are investigating. The student could be prosecuted under a new Pennsylvania law criminalising the distribution of such deepfakes. The school district has declined to comment further.
1 source article · read the reporting →
PimEyes facial recognition search engine identifies individuals from public photos
PimEyes, a Polish facial recognition search engine, allows users to upload a photo and find matching images from across the internet. The system scrapes billions of photos and creates biometric profiles without consent. A banker named Dylan (pseudonym) discovered that an old party photo of him was found by the system, potentially outing his private life. Privacy advocates and platforms like Instagram and YouTube are taking action against PimEyes, alleging violations of the GDPR.
- Company involved
- PimEyes
- AI system involved
- PimEyes
6 source articles · read the reporting →
ChatGPT Health fails to direct 52% of medical emergencies to emergency care in study
A study published in Nature Medicine found that OpenAI's ChatGPT Health tool under-triaged 52% of true medical emergencies, directing users to non-urgent care instead of emergency departments. The AI also misclassified 35% of non-urgent cases. Researchers at Mount Sinai conducted 960 tests across 60 clinical scenarios, noting the tool's susceptibility to anchoring bias when symptoms were minimized. The study highlights potential safety concerns as millions use AI for health guidance.
- Company involved
- OpenAI
- AI system involved
- ChatGPT Health
4 source articles · read the reporting →
Deepfake adverts using Barbara O'Neill's likeness promote fake prostate cures
AI-generated deepfake videos impersonating Australian naturopath Barbara O'Neill are circulating on Facebook, falsely endorsing a cure for prostate problems and erectile dysfunction. The videos, which have millions of views, are part of a scam to sell products or phish for personal information. Africa Check found the videos to be deepfakes, with audio out of sync and posted by accounts unaffiliated with O'Neill, who has previously warned about such impersonations.
2 source articles · read the reporting →
Security Health Plan used AI to cut off nursing home care for 85-year-old woman
Frances Walter, an 85-year-old woman with a shattered shoulder, had her nursing home care payment cut off by Security Health Plan after an algorithm predicted she would recover in 16.6 days. The algorithm, nH Predict from NaviHealth, did not account for her severe pain and allergy to pain medicine. She was forced to spend her life savings and enroll in Medicaid while fighting the denial. A federal judge later ruled the denial was speculative and she was owed thousands of dollars.
- Company involved
- Security Health Plan
- AI system involved
- nH Predict
1 source article · read the reporting →
