CMS warns insurers against using AI to deny Medicare Advantage care
The Centers for Medicare & Medicaid Services (CMS) clarified that health insurers cannot use algorithms or AI to deny care to Medicare Advantage members. This follows lawsuits alleging UnitedHealth and Humana used an AI tool, nH Predict, to wrongfully deny post-acute care to elderly patients. The tool reportedly produced rigid estimates ignoring individual patient needs, leading to premature denials. CMS warned that non-compliance could result in penalties and increased audits.
- Company involved
- UnitedHealth, Humana
- AI system involved
- nH Predict
10 source articles · read the reporting →
Estonia used AI to remove benefits from elderly without human involvement
The Council of Europe Human Rights Commissioner reported that Estonia's social security system used AI to establish incapacity levels and remove benefits from elderly people without human involvement. The system allegedly made decisions automatically, leading to complaints from NGOs.
8 source articles · read the reporting →
UK government using AI for benefits and immigration decisions
The UK government is reportedly using artificial intelligence to make decisions on welfare benefits and immigration applications. The system automates decisions previously made by human caseworkers. Concerns have been raised about the fairness and transparency of such automated decisions.
10 source articles · read the reporting →
INSS AI Denies Rural Worker's Pension After Misidentifying Her as a Man
A rural worker in Brazil, Josélia de Brito, had her pension application through the Meu INSS app automatically denied after the AI system misidentified her as a man. The INSS deployed the system to speed up benefit decisions, but experts say it struggles with complex rural cases. The case is cited as evidence that automation may exclude vulnerable people with limited digital access.
- Company involved
- Instituto Nacional do Seguro Social (INSS)
- AI system involved
- Meu INSS
4 source articles · read the reporting →
Australian government settles Robodebt class action for $1.7 billion
The Robodebt system, an automated debt recovery system used by Centrelink, calculated welfare debts using averaged income data from the ATO, leading to invalid debts. A class action was filed in November 2019 and settled in November 2020. The settlement provided refunds of over $751 million in invalid debts collected and $112 million in compensation to approximately 400,000 group members. The Australian government also dropped claims for over $1 billion in invalid debts.
- Company involved
- Australian Government (Centrelink)
- AI system involved
- Robodebt
10 source articles · read the reporting →
Ibrahim Diallo fired by automated HR system, locked out for three weeks
Ibrahim Diallo, a software engineer in California, was automatically terminated by an HR system after a script triggered by a missed contract renewal revoked his access and disabled his key card. Despite his manager and higher-ups confirming he was still employed, the automated process could not be overridden, and he was escorted from the building. It took three weeks for the company to resolve the issue by rehiring him as a new employee, after which he quit. The incident highlights the risks of fully automated termination systems without human override.
10 source articles · read the reporting →
Air Canada Chatbot Fabricates Discount, Leading to Court Case
Air Canada's customer service chatbot allegedly fabricated a discount during a conversation with a customer last year. The incident resulted in a court case against the airline. Insurer Armilla stated that its new AI mishap policy would have covered the loss from selling tickets at the discounted price if the chatbot was found to have underperformed.
- Company involved
- Air Canada
- AI system involved
- chatbot
6 source articles · read the reporting →
France: CNAF's discriminatory risk-scoring algorithm must be stopped
Amnesty International and coalition partners filed a complaint with the Council of State against CNAF's risk-scoring algorithm used to detect benefit overpayments. The algorithm assigns risk scores based on criteria that discriminate against vulnerable groups, including those with disabilities, single parents, and low-income households. The complaint alleges the system violates human rights to equality and privacy. The EU AI Act's social scoring ban may apply, but its definition remains unclear.
- Company involved
- CNAF
2 source articles · read the reporting →
AMS algorithm lacks transparency and may discriminate against job seekers
The Austrian Public Employment Service (AMS) uses an algorithm to classify job seekers into categories A, B, and C, determining their access to benefits and training. Scientists from TU Wien, WU Wien, and University of Vienna have criticised the algorithm for lacking transparency, as only two of 96 model variants have been published. They allege that the system may discriminate against women and people with migration background, and that job seekers are not informed about how the algorithm works or given a chance to appeal.
- Company involved
- AMS (Arbeitsmarktservice Österreich)
- AI system involved
- AMS-Algorithmus
10 source articles · read the reporting →
Austrian court lifts ban on AMS job-chance prediction algorithm
The Austrian Federal Administrative Court overturned a ban by the Data Protection Authority on the AMS algorithm, which predicts job chances of unemployed people. The system, trained on historical data including age, gender, and nationality, categorises individuals as high, medium or low chance of re-employment. Critics argue it discriminates against women and mothers, and that the data is now outdated due to the COVID-19 pandemic. The court ruled that the algorithm is lawful as long as a human advisor makes the final decision on training support.
- Company involved
- Arbeitsmarktservice (AMS)
- AI system involved
- AMS-Algorithmus
5 source articles · read the reporting →
Barclays pilot of Sapience monitoring software causes employee stress
Barclays introduced a pilot of employee monitoring software from Sapience in its product control department at Canary Wharf. The system monitors computer activity and admonishes staff if they are not deemed active enough, recording breaks as "unaccounted activity". Employees reported significant stress and worry about taking breaks. Barclays acknowledged the pilot and said it would listen to feedback.
- Company involved
- Barclays
- AI system involved
- Sapience employee monitoring software
10 source articles · read the reporting →
DWP's Universal Credit algorithm causes underpayment and hardship for claimants
The Department for Work and Pensions (DWP) uses an automated means-testing algorithm in its Universal Credit system that miscalculates earnings due to a design flaw, leading to underpayments and fluctuating benefit amounts. Claimants such as Janet R. received payments hundreds of pounds short, forcing them into debt and foodbank use. A Court of Appeal ruling in June 2020 ordered the DWP to fix the flaw, but the department had not yet fully implemented the remedy at the time of writing.
- Company involved
- Department for Work and Pensions
- AI system involved
- Universal Credit
10 source articles · read the reporting →
Finnish recruitment company Digital Minds used AI to analyze job applicants' messages, prompting data protection investigation
Digital Minds, a Finnish recruitment company founded by psychologists, used IBM Watson AI to analyze job applicants' social media and email messages for personality assessments. The company obtained written consent but the Finnish Data Protection Ombudsman launched an investigation, suspecting violations of data protection laws and the secrecy of correspondence. The service was used on fewer than ten applicants and has been paused pending the investigation.
- Company involved
- Digital Minds
- AI system involved
- IBM Watson
9 source articles · read the reporting →
Teleperformance plans AI webcam surveillance for home-working staff
Teleperformance, a global call centre company, told some staff it would install AI-powered webcams to monitor home-working infractions such as eating, phone use, or leaving desks. The system would randomly scan for breaches and send alerts to managers. After the Guardian inquired, the company said the remote scans would not be used in the UK, but the plan raised concerns from unions and MPs about invasive surveillance.
- Company involved
- Teleperformance
10 source articles · read the reporting →
CBSE introduces facial recognition system for students to access digital documents
The Central Board of Secondary Education (CBSE) has introduced a facial recognition system for Class 10 and 12 students to access their digital academic documents. A live image of the student is compared with the photograph on their CBSE admit card and, if the match succeeds, the certificate is emailed to them. The facility is available on Digi Locker for 2020 records and is expected to help foreign students and those unable to open a Digi Locker account.
- Company involved
- Central Board of Secondary Education (CBSE)
- AI system involved
- Facial Recognition System
10 source articles · read the reporting →
Amazon's automated HR wrongly fires sick workers with coronavirus
Amazon's automated human resources system reportedly denied sick-leave to workers with COVID-19 and wrongfully initiated termination proceedings against some who were sick or recovering. The system, designed to handle HR requests automatically, failed under the influx of pandemic-related demands, leaving employees on hold for hours or dealing with chatbots. Six warehouse workers from Indiana to New Jersey described the problems to Bloomberg.
- Company involved
- Amazon
- AI system involved
- Automated HR system (chatbots and automated termination tracking)
8 source articles · read the reporting →
PwC develops facial recognition tool to monitor employees working from home
Accounting giant PwC has developed a facial recognition tool that logs when employees are absent from their computer screens while working from home. The tool, intended for financial institutions, requires workers to provide written reasons for any absences, including toilet breaks. Commentators have criticised the tool as a huge invasion of privacy, with concerns about damage to trust and increased stress. PwC stated that the technology is designed to help regulated institutions meet compliance obligations and that voluntary consent of traders is essential.
- Company involved
- PwC
8 source articles · read the reporting →
Denmark's Udbetaling Danmark welfare algorithm illegally surveilled millions
Udbetaling Danmark (UDK), Denmark's centralized welfare payment agency, used automated algorithms to check eligibility and detect fraud. The system collected personal data on millions of beneficiaries and their relatives, including non-beneficiaries, in what a think-tank called 'systematic surveillance'. A complaint to the Danish Data Protection Authority led to a ruling that the data collection violated GDPR, and UDK eventually deleted the illegally obtained data. The system also caused erroneous payments, including underpaying 300,000 pensioners by 94 million euros annually.
- Company involved
- Udbetaling Danmark
10 source articles · read the reporting →
Company fires HR team after ATS auto-rejects manager's CV due to filtering error
A company's applicant tracking system (ATS) auto-rejected qualified candidates' resumes for three months because it was filtering for the outdated framework AngularJS instead of the required Angular framework. The manager discovered the flaw by submitting his own CV under a pseudonym and found it was rejected within seconds. After the manager reported the issue to upper management, the company investigated and dismissed half of its HR team. No legal action or regulatory involvement is reported.
4 source articles · read the reporting →
UnitedHealthcare sued for using AI to deny Medicare Advantage patients care
A class action complaint alleges that UnitedHealthcare and its subsidiary naviHealth deployed an artificial intelligence model to override treating physicians' determinations of medically necessary care for elderly Medicare Advantage patients. The complaint claims the AI model has a known 90% error rate and wrongfully denied patients care. The lawsuit, filed in the U.S. District Court for the District of Minnesota, seeks damages on behalf of the estates of two deceased patients and all others similarly situated.
- Company involved
- UnitedHealthcare, Inc.
10 source articles · read the reporting →
UnitedHealth used ALERT algorithm to limit mental health coverage, regulators found
ProPublica reports that UnitedHealth Group's Optum subsidiary used the ALERT algorithm to flag mental health patients and therapists as outliers, leading to therapy coverage denials. Regulators in California, Massachusetts and New York alleged this breached the federal Mental Health Parity and Addiction Equity Act, and UnitedHealth agreed to restrict the system in those jurisdictions. The company denies wrongdoing and says its programmes are compliant. Affected patients, including Medicaid enrollees, were left to pay out-of-pocket or go without care.
- Company involved
- UnitedHealth Group
- AI system involved
- ALERT
5 source articles · read the reporting →
IRCC uses AI triage for Temporary Resident Visa applications
Immigration, Refugees and Citizenship Canada (IRCC) uses an AI system called Advanced Analytics to triage Temporary Resident Visa applications from India and China. The system categorizes applications into tiers, with Tier 1 approved automatically and others sent to human officers. Critics allege the system lacks transparency and may introduce bias, leading to visa refusals without clear rationale. The author, a Canadian immigration lawyer, is filing Federal Court cases on behalf of clients affected by refusals.
- Company involved
- Immigration, Refugees and Citizenship Canada (IRCC)
- AI system involved
- Advanced Analytics Triage of Overseas Temporary Resident Visa Applications
10 source articles · read the reporting →
Humana sued for using AI to deny seniors rehabilitation care
Health insurer Humana is accused in a class-action lawsuit of using an AI algorithm to systematically deny rehabilitation care to Medicare Advantage patients, despite recommendations from their doctors. The lawsuit, filed on December 12, 2023, alleges that the AI tool restricted medically necessary care. This is the second major health insurer to face legal action over its use of AI to deny care.
- Company involved
- Humana
9 source articles · read the reporting →
California EDD's automated fraud detection wrongly suspended 600,000 legitimate unemployment claims
In January 2021, the California Employment Development Department used Thompson Reuters automated batch review software to flag 1.1 million unemployment claims as potentially fraudulent. EDD stopped payments on those claims without prior notice. Later, over 600,000 were confirmed as legitimate after claimants used ID.me to verify their identity. The incident highlights the trade-off between fraud prevention and timely benefit access.
- Company involved
- California Employment Development Department (EDD)
- AI system involved
- Thompson Reuters Automated Batch Review
7 source articles · read the reporting →
