Where automated decisions went wrong

"This is so sad": AI nurse scam costs Australian pensioner R2m - briefly.co.za

The AI-generated deepfake video persona convinced the pensioner to send money.

1 source article · read the reporting →

Apple temporarily disables notification summaries after BBC complaint

Apple's iOS 18.3 beta temporarily disables Apple Intelligence notification summaries for News & Entertainment apps following complaints from news outlets, including the BBC, about inaccurate summaries. The feature, which uses AI to summarize notifications, generated errors that misrepresented news content. Apple acknowledged the issue and introduced changes to clarify when summaries are AI-generated, as well as a warning that they 'may contain errors'. The summaries will be re-enabled after further refinement.

Company involved

Apple

AI system involved

Apple Intelligence notification summaries

9 source articles · read the reporting →

Resume prompt injection tricks AI hiring - moneywise.com

AI screening system determined which job applicants to advance to the next stage of recruitment.

1 source article · read the reporting →

Clinical algorithm showed racial bias in health care recommendations

In 2019, a study revealed that a clinical algorithm used by many hospitals to identify patients needing extra care was racially biased. Black patients had to be significantly sicker than white patients to be recommended for the same care, due to the algorithm being trained on historical health care spending data that reflected racial disparities. The bias was eventually detected and corrected, but the incident raised concerns about the potential for AI to worsen medical racism.

5 source articles · read the reporting →

Sydney students resit NAPLAN after predictive text glitch

Year 5 students at Waverley College and Kambala in Sydney were required to resit their NAPLAN writing exams after a technical glitch enabled Apple's predictive text and spellcheck features during the test. The issue was discovered by a teacher and students, and the schools reported it to NESA and ACARA. The students were given a new prompt and completed the test the following Monday. An ACARA spokesman stated the problem was not widespread, with over 880,000 tests submitted successfully.

Company involved

ACARA

AI system involved

NAPLAN

2 source articles · read the reporting →

St George's admissions algorithm biased against women and ethnic minorities

In the late 1970s and 1980s, St George's Hospital Medical School used a computer algorithm to automate the first round of its admissions process, scoring applicants based on historical data. The model was found to discriminate against female applicants and those with non-European-looking names, denying them interviews. The bias was discovered by two professors, and the university co-operated with a Commission for Racial Equality inquiry, subsequently contacting affected applicants and offering some places.

Company involved

St George's Hospital Medical School

4 source articles · read the reporting →

Acclarent's AI-Powered Surgery Tool Allegedly Injures Patients, Lawsuits Claim

The TruDi Navigation System by Acclarent, which uses AI to confirm device positions during sinus surgery, has been linked to at least ten patient injuries including strokes, according to FDA reports and two lawsuits. Plaintiffs Erin Ralph and Donna Fernihough allege that the AI was inaccurate and misled surgeons, causing carotid artery damage and strokes in 2022. Acclarent denies the allegations, stating there is no credible evidence of a causal connection. The lawsuits are ongoing.

Company involved

Acclarent

AI system involved

TruDi Navigation System

2 source articles · read the reporting →

Google Translate erases female historians and male nurses due to gender bias

An experiment by AlgorithmWatch found that Google Translate systematically changes the gender of occupation words when translating between gender-inflected languages, using English as a bridge. Female historians were rendered as male, and male nurses as female, reinforcing stereotypes. Google acknowledged the issue, explaining that gender is lost when translating to English, the bridging language. The feature affects hundreds of millions of European users who cannot easily avoid Google Translate on Android and Chrome.

Company involved

Google

AI system involved

Google Translate

10 source articles · read the reporting →

Physicians and students push to remove race from kidney function test eGFR

The eGFR kidney test, which estimates kidney function, has historically included race as a factor based on the assumption that Black people have higher muscle mass. This can overestimate kidney function in Black patients, potentially delaying treatment and transplant listing. Medical students and physicians have petitioned hospitals to stop using race, and some institutions have already changed their protocols. The National Kidney Foundation and American Society of Nephrology announced a task force to evaluate the use of race in the test.

AI system involved

estimated glomerular filtration rate (eGFR) test

3 source articles · read the reporting →

Montefiore Allegedly Plans to Replace Nurses with AI for Insurance Reviews

The New York State Nurses Association alleges that Montefiore Health System plans to lay off 12 utilization review nurses and replace them with AI-powered software from Datavant. The union warns that the move could compromise patient care by having AI review insurance denials without clinical judgement. Montefiore denies the claims, stating they are investing in technology for better outcomes. The plan has sparked a town hall and calls from elected officials to halt the layoffs.

Company involved

Montefiore Health System

3 source articles · read the reporting →

Blind people and advocates criticise AccessiBe for worsening website access

Blind users and disability advocates say AccessiBe, an automated web accessibility overlay, disrupts screen readers on websites, making some sites unnavigable and preventing users from paying rent, shopping or teaching. More than 400 people signed an open letter asking companies to stop using automated overlays. Some blind users have brought ADA lawsuits against companies that use AccessiBe; one case was referred to mediation and another was settled. AccessiBe denies that it was at fault, saying critics do not give specific examples.

Company involved

AccessiBe

AI system involved

AccessiBe

10 source articles · read the reporting →

CMS warns insurers against using AI to deny Medicare Advantage care

The Centers for Medicare & Medicaid Services (CMS) clarified that health insurers cannot use algorithms or AI to deny care to Medicare Advantage members. This follows lawsuits alleging UnitedHealth and Humana used an AI tool, nH Predict, to wrongfully deny post-acute care to elderly patients. The tool reportedly produced rigid estimates ignoring individual patient needs, leading to premature denials. CMS warned that non-compliance could result in penalties and increased audits.

Company involved

UnitedHealth, Humana

AI system involved

nH Predict

10 source articles · read the reporting →

Australian education ministers scrap NAPLAN robot marking

The Education Council decided not to use automated essay scoring for NAPLAN writing tests, dropping a proposal by ACARA to have computers mark English tasks. Teachers' unions had campaigned against the plan, and an academic warned that the software would reward verbose gibberish and could not assess creativity. The decision was made in December 2017 and announced in January 2018.

Company involved

Australian Curriculum, Assessment and Reporting Authority (ACARA)

10 source articles · read the reporting →

Nurse at St. Rose Dominican Hospital Averts AI-Sepsis Alert Error

A nurse at St. Rose Dominican Hospital in Henderson, Nevada, refused to follow an AI-generated sepsis alert that would have given an elderly dialysis patient IV fluids, which could have caused a life-threatening complication. The alert, part of the hospital's electronic system, prompted a charge nurse to order the fluids despite the patient's compromised kidneys. A physician intervened and ordered dopamine instead, averting harm. The incident highlights concerns about AI tools overriding clinical judgment.

Company involved

St. Rose Dominican Hospital

1 source article · read the reporting →

France: CNAF's discriminatory risk-scoring algorithm must be stopped

Amnesty International and coalition partners filed a complaint with the Council of State against CNAF's risk-scoring algorithm used to detect benefit overpayments. The algorithm assigns risk scores based on criteria that discriminate against vulnerable groups, including those with disabilities, single parents, and low-income households. The complaint alleges the system violates human rights to equality and privacy. The EU AI Act's social scoring ban may apply, but its definition remains unclear.

Company involved

CNAF

2 source articles · read the reporting →

China's vocational school students exploited as data annotators for AI

Vocational schools in China force students to work as data annotators for AI companies, paying subminimum wages and taking commissions. Students like Lucy in Shandong spent months labeling data for autonomous driving and content moderation systems, with little learning or career advancement. The practice continues despite new regulations.

2 source articles · read the reporting →

Nevada's AI grad score model cuts funding for low-income students

Nevada replaced its income-based school funding formula with a machine learning model developed by Infinite Campus that generates a 'grad score' for each student. The model reduced the number of students eligible for supplemental funding from 288,000 to 63,000, disproportionately affecting low-income students. Critics argue the model lacks transparency and de-emphasizes economic status, potentially leaving many high-poverty schools underfunded.

Company involved

Nevada Department of Education

AI system involved

Grad score model

7 source articles · read the reporting →

Dartmouth Medical School charges 17 students with cheating based on tracking system.

Dartmouth College's Geisel School of Medicine charged 17 students with cheating on remote exams during the pandemic. The charges were based on secret tracking of student activity on the learning management system. Critics say the system is prone to errors and inappropriate for monitoring students.

Company involved

Dartmouth College Geisel School of Medicine

10 source articles · read the reporting →

Google and HCA Healthcare partner to analyze 32 million patient records

Google Cloud has announced a partnership with HCA Healthcare to analyze around 32 million patient records. The anonymized data will be used to develop algorithms that could advise doctors on treatment options. Privacy advocates have raised concerns about the data sharing and the potential for AI to re-identify patients. HCA insists that patient-identifiable information will be stripped and that access will be tightly controlled.

Company involved

HCA Healthcare

AI system involved

Google Cloud

9 source articles · read the reporting →

UnitedHealthcare sued for using AI to deny Medicare Advantage patients care

A class action complaint alleges that UnitedHealthcare and its subsidiary naviHealth deployed an artificial intelligence model to override treating physicians' determinations of medically necessary care for elderly Medicare Advantage patients. The complaint claims the AI model has a known 90% error rate and wrongfully denied patients care. The lawsuit, filed in the U.S. District Court for the District of Minnesota, seeks damages on behalf of the estates of two deceased patients and all others similarly situated.

Company involved

UnitedHealthcare, Inc.

10 source articles · read the reporting →

NHS liver transplant algorithm causes longer waits for younger patients

An NHS computer algorithm that prioritises liver transplant patients most likely to die soon has led to younger patients waiting 156 days longer on average than those over 60. The algorithm, launched in 2018, gives a score based on 21 recipient and seven donor parameters. Patients like Sarah Meredith, 31, have been waiting two years for a transplant. NHS Blood and Transplant defends the system as saving lives but acknowledges the stress on younger patients.

Company involved

NHS Blood and Transplant

10 source articles · read the reporting →

Perth hospital group bans ChatGPT after doctor used it for a patient discharge summary

South Metropolitan Health Service, which runs five hospitals in Perth, ordered staff to stop using ChatGPT for work involving patient information after an email said some staff had used the AI chatbot to write medical notes. The service later clarified that only one doctor had used the tool to generate a patient discharge summary and that no confidential patient information had been breached. The Australian Medical Association has called for national regulations to ensure a human is involved in AI-assisted health decisions.

Company involved

South Metropolitan Health Service

AI system involved

ChatGPT

6 source articles · read the reporting →

NHS QCovid algorithm incorrectly labels young patients as high risk, causing vaccine call-up

The article reports that scientists are questioning the reliability of the QCovid risk prediction algorithm used by the NHS to prioritise COVID-19 vaccinations. Due to missing data on weight or ethnicity, the algorithm automatically assigns a high BMI and high-risk ethnicity, leading to young, healthy patients being incorrectly labelled as high risk and called for early vaccination. GPs have identified hundreds of such cases, causing anxiety and potentially delaying vaccinations for more vulnerable individuals. NHS Digital acknowledged the issue and said clinicians can remove patients from the shielding list.

Company involved

NHS

AI system involved

QCovid

10 source articles · read the reporting →

Four commercial large language models perpetuate race-based medical misconceptions

A study published in npj Digital Medicine tested four commercial large language models (Bard, ChatGPT, GPT-4, and Claude) for their tendency to propagate discredited race-based medical beliefs. When asked about kidney function, lung capacity, and skin thickness, the models sometimes endorsed debunked racial differences, particularly affecting Black patients. The study concludes that these biases pose a potential hazard and urges caution before using such models in clinical decision-making.

Company involved

Not named in article (refers to commercial LLMs generically as Google's Bard, OpenAI's ChatGPT and GPT-4, and Anthropic's Claude)

AI system involved

Bard, ChatGPT, GPT-4, Claude

6 source articles · read the reporting →