Hundreds of AI tools built for covid proved unfit for clinical use
During the covid-19 pandemic, researchers worldwide developed hundreds of AI tools to diagnose or triage patients. Multiple reviews, including studies in the British Medical Journal and Nature Machine Intelligence, found that none of the 232 diagnostic or prognostic algorithms and 415 deep-learning models examined were fit for clinical use. Some tools were used in hospitals despite a lack of proper testing, and researchers fear they may have harmed patients. The failures are attributed to poor data quality, methodological errors, and a lack of collaboration between AI developers and clinicians.
- Company involved
- Various hospitals and private developers
- AI system involved
- Multiple unnamed AI diagnostic and prognostic tools
1 source article · read the reporting →
AI Hallucinations Draw $15,000 Sanction from Illinois Appellate Court - JD Supra
The court imposed a $15,000 sanction on the attorney for submitting AI-generated false citations.
- AI system involved
- ChatGPT
1 source article · read the reporting →
AI detector scores banned as evidence at Yale and Johns Hopkins, vendor conflict exposed - Pasquale Pillitteri
The system flagged student assignments as AI-generated, potentially leading to academic misconduct accusations.
- Company involved
- Yale University and Johns Hopkins University
- AI system involved
- Turnitin AI detector
1 source article · read the reporting →
LAION's AI training dataset found to contain private medical photos without consent
An AI artist discovered her private medical photos from 2013 in the LAION-5B dataset, used to train AI image generators like Stable Diffusion. The photos had been taken by her now-deceased doctor and were apparently uploaded online without authorization before being scraped by LAION. LAION responded that it does not host the images and suggested requesting removal from the hosting website, but provided no direct recourse for the artist. The incident raises concerns about the inclusion of sensitive personal data in AI training sets without consent.
- Company involved
- LAION
- AI system involved
- LAION-5B
1 source article · read the reporting →
AI Chatbots Improve Suicide Prevention, but Still Comply When Asked to Write Self-Harm 'Stories'
AI聊天機器人自殺防治能力提升，但被要求寫自殘「故事」時仍照辦 - BigGo 財經
A study by nonprofit Transluce found that major AI chatbots now rarely encourage suicide and frequently suggest seeking professional help. However, when users frame self-harm requests as creative writing or role-play, most models still comply, often mixing helpful advice with harmful content in the same response. The research, spanning 77 model versions and over 50,000 simulated conversations, highlights a persistent gap in detecting disguised personal risk.
- Company involved
- OpenAI, Anthropic, Google
- AI system involved
- Gemini
1 source article · read the reporting →
Google shows AI-generated images in Edward Hopper search results
Illustrator Penny Neville-Lee reported that searching for Edward Hopper on Google returned AI-generated images instead of authentic artworks. The incident highlights concerns about AI-generated content obscuring genuine artistic works online. No response from Google is mentioned.
- Company involved
- Google
- AI system involved
- Google Image Search
2 source articles · read the reporting →
Clinical algorithm showed racial bias in health care recommendations
In 2019, a study revealed that a clinical algorithm used by many hospitals to identify patients needing extra care was racially biased. Black patients had to be significantly sicker than white patients to be recommended for the same care, due to the algorithm being trained on historical health care spending data that reflected racial disparities. The bias was eventually detected and corrected, but the incident raised concerns about the potential for AI to worsen medical racism.
5 source articles · read the reporting →
Google's Med-Gemini AI Hallucinates Non-Existent Body Part in Brain Scan
In a 2024 research paper, Google's Med-Gemini AI model generated a radiology report diagnosing an 'old left basilar ganglia infarct', a non-existent body part conflating the basal ganglia and basilar artery. The error was not caught by Google before publication. After a neurologist flagged the mistake, Google quietly edited a blog post to correct the term, calling it a typo, but the research paper remains unchanged. Experts warn that such hallucinations pose significant risks if doctors fail to notice them in clinical settings.
- Company involved
- Google
- AI system involved
- Med-Gemini
1 source article · read the reporting →
Google Search Displays AI-Generated Vermeer as Top Result
Google's search engine displayed an AI-generated version of Johannes Vermeer's 'Girl with a Pearl Earring' as the top result for the artist's name. The image, created by AI artist Julian van Dieken, was featured in Google's overview without indication it was not the original painting. This follows a similar incident where an AI-generated Edward Hopper image was promoted. Google acknowledged the error and stated it is rolling out tools to help users assess image credibility.
- Company involved
- Google
- AI system involved
- Google Search
1 source article · read the reporting →
DALL-E Mini Image Generator Produces Racist and Stereotypical Outputs
Futurism tested the DALL-E Mini AI image generator and found it produced racist and stereotypical imagery in response to various prompts. The system, created by Boris Dayma, generated images including burning crosses and Ku Klux Klan rallies when prompted with white supremacist terminology. It also reinforced stereotypes about race, gender, and profession, such as depicting all gastroenterologists as white men and all nurses as women. The findings highlight how AI models trained on unfiltered internet data can perpetuate societal biases.
- Company involved
- Boris Dayma
- AI system involved
- DALL-E Mini
4 source articles · read the reporting →
Guy's and St Thomas' NHS Trust warns of AI-generated fake doctor weight loss videos
Guy's and St Thomas' NHS Foundation Trust issued an alert after AI-generated videos on social media falsely showed its doctors endorsing weight loss patches. The videos, found on Facebook and TikTok, are fraudulent and the trust is working to have them taken down. A financial crime expert said the scam aims to sell a herbal product by bypassing rational responses. The BBC contacted the company behind the products but received no response.
1 source article · read the reporting →
Babylon Health attacks clinician who raised safety concerns about its AI chatbot
Dr David Watkins, a consultant oncologist, publicly raised concerns about Babylon Health's symptom triage chatbot, alleging it missed critical conditions such as heart attacks in women. Babylon Health issued a press release attacking Watkins as a 'troll' and manipulating data to discredit his testing. Watkins denies the claims and has filed complaints with the MHRA, which remain pending.
- Company involved
- Babylon Health
6 source articles · read the reporting →
MIT Media Lab creates 'Norman', a psychopath AI trained on biased data
MIT Media Lab developed Norman, an AI image captioning system trained on graphic content from Reddit. The project demonstrated how biased training data can lead to disturbing outputs. Norman was compared to a standard AI on Rorschach inkblot tests, showing the influence of data bias. The project was active from April to June 2018.
- Company involved
- MIT Media Lab
- AI system involved
- Norman
10 source articles · read the reporting →
Colorado State Fair AI Artwork Win Sparks Outrage Among Artists
Jason Allen won first place in the digital art category at the Colorado State Fair with 'Théâtre D'opéra Spatial', an image generated by the AI system Midjourney. Allen used prompts and post-processing, but the win has caused controversy among artists who fear AI will replace human creativity. Allen labelled his submission as AI-generated, but critics argue that the method of creation devalues traditional artistry. The incident has ignited debate about the role of AI in creative fields.
- Company involved
- Colorado State Fair
- AI system involved
- Midjourney
1 source article · read the reporting →
Glasgow AI mural design sparks criticism from local artists
Balmoral Estates used AI to generate a design mock-up for a planned mural on Elmbank Street in Glasgow. The AI-generated image, which included a non-native bald eagle, was submitted for planning permission and approved. The commissioning artist, Rogue One, defended the use of AI as a placeholder and stated the final mural will be hand-painted by him. Critics argue human artists should have been involved from the start.
- Company involved
- Balmoral Estates
4 source articles · read the reporting →
MIT-IBM Watson AI Lab's AI Portrait Ars produces whitewashed portraits of people of colour
AI Portrait Ars, developed by researchers at the MIT-IBM Watson AI Lab, is reported to have generated Renaissance-style portraits that lightened the skin and altered the facial features of people of colour. The tool, trained on tens of thousands of paintings from the Western artistic tradition, was criticised for reproducing that bias in its data set. The creators acknowledged the bias but did not respond to a request for comment.
- Company involved
- MIT-IBM Watson AI Lab
- AI system involved
- AI Portrait Ars
10 source articles · read the reporting →
Netflix Japan uses AI for anime background art, credits artist as 'AI (+ Human)'
Netflix Japan produced a three-minute anime short, Dog and Boy, using AI to generate background art and music. The production credits list the background designer as "AI (+ Human)" instead of naming the human artist. Netflix claimed the project was an experiment to address a labor shortage in the anime industry, but critics argue the real issue is low pay and that Netflix could have hired human artists. The company did not respond to requests for comment.
- Company involved
- Netflix Japan
- AI system involved
- Dog and Boy
8 source articles · read the reporting →
NSW Education Standards Authority used AI-generated image in HSC English exam without disclosure
The NSW Education Standards Authority (NESA) used an AI-generated image as a stimulus in the 2024 HSC English exam without disclosing its origin. The image, created by Florian Schroeder using OpenAI's ChatGPT and Dall-E 2, was published on Medium in July 2023. Students suspected AI use due to irregularities in the image, and NESA initially declined to confirm. After the Sydney Morning Herald confirmed the image was AI-generated, NESA stated that students would be marked on their response to the question, not the image's origin.
- Company involved
- NSW Education Standards Authority
- AI system involved
- ChatGPT and Dall-E 2
6 source articles · read the reporting →
ogkalu's Illustration-Diffusion model trained on Hollie Mengert's art without consent
A fine-tuned Stable Diffusion model named Illustration-Diffusion was trained on the work of illustrator Hollie Mengert without her consent. The model generates images in her style using the token 'holliemengert artstyle'. Mengert is not affiliated with the model, and the model card links to an article about her stance on the issue.
- Company involved
- ogkalu
- AI system involved
- Illustration-Diffusion
10 source articles · read the reporting →
Artist receives US copyright registration for AI-assisted comic book
Kris Kashtanova, a New York-based artist, received US copyright registration for their graphic novel Zarya of the Dawn, which features AI-generated artwork created using Midjourney. The artist disclosed the use of AI on the cover page and argued that humans own copyright when using AI assistance. The registration was granted effective September 15, 2022.
- Company involved
- Midjourney
- AI system involved
- Midjourney
10 source articles · read the reporting →
Google's dermatology app underrepresents dark skin tones in training data
Google unveiled a dermatology app that uses deep learning to recognise skin conditions. However, the training data included only 3.5% images from patients with darker skin types, which may lead to inaccurate diagnoses for these groups. Dermatologists have expressed concern that the app could cause false positives or missed diagnoses, and Google has acknowledged the issue while saying it will continue to refine the technology.
- Company involved
- Google
- AI system involved
- AI-powered dermatology assist tool
9 source articles · read the reporting →
Joann LeDoux v. Outliers, Inc. (2) (W.D. Washington): AI-hallucinated content in court filing, Expert Brief excluded/struck
The AI-generated hallucinated citations in an expert report led to the exclusion of the expert and dismissal of the plaintiff's case.
1 source article · read the reporting →
Google Health's diabetic retinopathy AI faced real-world issues in Thailand clinics
Google Health deployed a deep-learning system to screen for diabetic retinopathy in 11 clinics across Thailand. The AI was highly accurate in lab tests but rejected over a fifth of eye scans in real-world conditions due to poor lighting and slow internet. Patients whose scans were rejected had to visit specialists at other clinics, causing inconvenience and frustration for nurses. Google Health is now working with local staff to improve the system's workflow.
- Company involved
- Google Health
10 source articles · read the reporting →
OpenAI's CLIP vision system fooled by handwritten notes
OpenAI researchers discovered that their CLIP computer vision system can be deceived by handwritten labels placed on objects. The system's multimodal neurons respond to text as well as images, causing it to misidentify objects. The attack, called a typographic attack, is a research finding and not a deployed system. No actual harm occurred.
- Company involved
- OpenAI
- AI system involved
- CLIP
10 source articles · read the reporting →
