Google Derm Assist dermatology app accused of racial bias
Google’s Derm Assist, an AI-powered app designed to analy s e skin conditions, faced criticism over potential racial bias . Derm Assist automatically analyses skin conditions, asks questions, and suggests causes. The app is being tested in the US and has been approved for use as a medical tool in Europe. Google said the app can recognise 288 skin conditions; however, some doctors expressed concerns about the accuracy of the system, in part due poor image quality, and the potential for over-diagnosis of skin cancers. It also appeared the system was trained and tested on a dataset that underrepresents people with dark skin tones. The training dataset used for the app consisted of 64,837 images of 12,399 patients located in two US states . However, only 3.5 percent of these images came from patients with Fitzpatrick skin types V and VI, which represent brown skin and dark brown or black skin, respectively . The majority of the database was composed of people with fair skin, darker white skin, or light brown skin . The findings resulted in accusations of racial bias, and drew attention to perceived real-life Google workforce discrimination and inequality. Google responded by saying it would only save images to help train the Derm Assist algorithm if users gave them explicit permission to do so. Concerns had also been raised about the potential for Google using personal sensitive data for other purposes. System 🤖 Google Derm Assist Operator: Alphabet/Google Developer: Alphabet/Google Country: USA Sector: Health Purpose: Identify dermatological issues Technology: Computer vision; Deep learning Issue: Accuracy/reliability; Bias/discrimination - race; Privacy Resource s 📃 Google (2021). Interpreting skin conditions with AI Google (2021). Development and Assessment of an Artificial Intelligence–Based Tool for Skin Condition Diagnosis by Primary Care Physicians and Nurse
- Date it happened
- 2021-04-01
- Organisation involved
This incident was imported from AIAAIC and is used under CC BY-SA 4.0. Our additions to it — the structured fields, the translation, the checks against other reports — are published under the same licence.
This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.