Barclays pilot of Sapience monitoring software causes employee stress
Barclays introduced a pilot of employee monitoring software from Sapience in its product control department at Canary Wharf. The system monitors computer activity and admonishes staff if they are not deemed active enough, recording breaks as "unaccounted activity". Employees reported significant stress and worry about taking breaks. Barclays acknowledged the pilot and said it would listen to feedback.
- Company involved
- Barclays
- AI system involved
- Sapience employee monitoring software
10 source articles · read the reporting →
DWP's Universal Credit algorithm causes underpayment and hardship for claimants
The Department for Work and Pensions (DWP) uses an automated means-testing algorithm in its Universal Credit system that miscalculates earnings due to a design flaw, leading to underpayments and fluctuating benefit amounts. Claimants such as Janet R. received payments hundreds of pounds short, forcing them into debt and foodbank use. A Court of Appeal ruling in June 2020 ordered the DWP to fix the flaw, but the department had not yet fully implemented the remedy at the time of writing.
- Company involved
- Department for Work and Pensions
- AI system involved
- Universal Credit
10 source articles · read the reporting →
UK study finds AI cannot detect COVID-19 from cough sounds
The UK Department of Health and Social Care awarded contracts to Fujitsu to develop a "Cough In A Box" app that would use AI to detect COVID-19 from cough sounds, based on earlier claims of high accuracy. However, a study led by the Alan Turing Institute found that machine learning models could not accurately predict COVID-19 from coughs due to confounding variables. The study concluded that the technology does not work for COVID-19 diagnosis.
- Company involved
- UK Department of Health and Social Care
- AI system involved
- Cough In A Box
10 source articles · read the reporting →
UK councils use Covid OneView AI to harvest personal data for risk scoring
UK local authorities are using a system called Covid OneView, developed by data analytics firm Xantura, to harvest millions of personal details from council records. The system uses predictive analytics and AI to assign risk scores to households and individuals, aiming to identify those vulnerable to Covid or likely to break lockdown rules. Privacy campaigners and MPs have criticised the lack of transparency and the extent of data collection, which includes sensitive information such as debt levels, living arrangements, and even notes on unfaithful sex. Xantura and Barking and Dagenham Council have defended the system as compliant with data protection rules and focused on providing support.
- Company involved
- UK local authorities
- AI system involved
- Covid OneView
6 source articles · read the reporting →
Finnish recruitment company Digital Minds used AI to analyze job applicants' messages, prompting data protection investigation
Digital Minds, a Finnish recruitment company founded by psychologists, used IBM Watson AI to analyze job applicants' social media and email messages for personality assessments. The company obtained written consent but the Finnish Data Protection Ombudsman launched an investigation, suspecting violations of data protection laws and the secrecy of correspondence. The service was used on fewer than ten applicants and has been paused pending the investigation.
- Company involved
- Digital Minds
- AI system involved
- IBM Watson
9 source articles · read the reporting →
SQA exam algorithm disproportionately downgraded poorer pupils, report finds
The Scottish Qualifications Authority (SQA) used an algorithm to moderate 2020 exam results after COVID-19 cancelled exams. The algorithm disproportionately downgraded students from disadvantaged backgrounds, leading to a public outcry and a government U-turn. An independent report criticised the SQA and Scottish Government for failing to properly assess the equality impacts and for not making the algorithm available for analysis. The SQA has expressed no regret over the moderation approach.
- Company involved
- Scottish Qualifications Authority (SQA)
10 source articles · read the reporting →
Teleperformance plans AI webcam surveillance for home-working staff
Teleperformance, a global call centre company, told some staff it would install AI-powered webcams to monitor home-working infractions such as eating, phone use, or leaving desks. The system would randomly scan for breaches and send alerts to managers. After the Guardian inquired, the company said the remote scans would not be used in the UK, but the plan raised concerns from unions and MPs about invasive surveillance.
- Company involved
- Teleperformance
10 source articles · read the reporting →
Google and HCA Healthcare partner to analyze 32 million patient records
Google Cloud has announced a partnership with HCA Healthcare to analyze around 32 million patient records. The anonymized data will be used to develop algorithms that could advise doctors on treatment options. Privacy advocates have raised concerns about the data sharing and the potential for AI to re-identify patients. HCA insists that patient-identifiable information will be stripped and that access will be tightly controlled.
- Company involved
- HCA Healthcare
- AI system involved
- Google Cloud
9 source articles · read the reporting →
Denmark's Udbetaling Danmark welfare algorithm illegally surveilled millions
Udbetaling Danmark (UDK), Denmark's centralized welfare payment agency, used automated algorithms to check eligibility and detect fraud. The system collected personal data on millions of beneficiaries and their relatives, including non-beneficiaries, in what a think-tank called 'systematic surveillance'. A complaint to the Danish Data Protection Authority led to a ruling that the data collection violated GDPR, and UDK eventually deleted the illegally obtained data. The system also caused erroneous payments, including underpaying 300,000 pensioners by 94 million euros annually.
- Company involved
- Udbetaling Danmark
10 source articles · read the reporting →
Campaigners call for end to Home Office's IPIC 'robo-caseworker' AI tool
Migrants' rights campaigners have called for the UK government to stop using the IPIC (Identify & Prioritise Immigration Cases) AI system, which automatically identifies and recommends migrants for immigration decisions or enforcement action. Privacy International and the Migrants' Rights Network have raised concerns that the system could lead to racial bias and the 'rubberstamping' of algorithmic recommendations by caseworkers. The campaigners argue that people going through the immigration system may not know their information has been processed by an algorithm. The Home Office has not directly responded to the specific concerns, though Science Secretary Peter Kyle has defended the safe development of AI in public services.
- Company involved
- Home Office
- AI system involved
- IPIC (Identify & Prioritise Immigration Cases)
9 source articles · read the reporting →
UnitedHealthcare sued for using AI to deny Medicare Advantage patients care
A class action complaint alleges that UnitedHealthcare and its subsidiary naviHealth deployed an artificial intelligence model to override treating physicians' determinations of medically necessary care for elderly Medicare Advantage patients. The complaint claims the AI model has a known 90% error rate and wrongfully denied patients care. The lawsuit, filed in the U.S. District Court for the District of Minnesota, seeks damages on behalf of the estates of two deceased patients and all others similarly situated.
- Company involved
- UnitedHealthcare, Inc.
10 source articles · read the reporting →
UnitedHealth used ALERT algorithm to limit mental health coverage, regulators found
ProPublica reports that UnitedHealth Group's Optum subsidiary used the ALERT algorithm to flag mental health patients and therapists as outliers, leading to therapy coverage denials. Regulators in California, Massachusetts and New York alleged this breached the federal Mental Health Parity and Addiction Equity Act, and UnitedHealth agreed to restrict the system in those jurisdictions. The company denies wrongdoing and says its programmes are compliant. Affected patients, including Medicaid enrollees, were left to pay out-of-pocket or go without care.
- Company involved
- UnitedHealth Group
- AI system involved
- ALERT
5 source articles · read the reporting →
Perth hospital group bans ChatGPT after doctor used it for a patient discharge summary
South Metropolitan Health Service, which runs five hospitals in Perth, ordered staff to stop using ChatGPT for work involving patient information after an email said some staff had used the AI chatbot to write medical notes. The service later clarified that only one doctor had used the tool to generate a patient discharge summary and that no confidential patient information had been breached. The Australian Medical Association has called for national regulations to ensure a human is involved in AI-assisted health decisions.
- Company involved
- South Metropolitan Health Service
- AI system involved
- ChatGPT
6 source articles · read the reporting →
NHS QCovid algorithm incorrectly labels young patients as high risk, causing vaccine call-up
The article reports that scientists are questioning the reliability of the QCovid risk prediction algorithm used by the NHS to prioritise COVID-19 vaccinations. Due to missing data on weight or ethnicity, the algorithm automatically assigns a high BMI and high-risk ethnicity, leading to young, healthy patients being incorrectly labelled as high risk and called for early vaccination. GPs have identified hundreds of such cases, causing anxiety and potentially delaying vaccinations for more vulnerable individuals. NHS Digital acknowledged the issue and said clinicians can remove patients from the shielding list.
- Company involved
- NHS
- AI system involved
- QCovid
10 source articles · read the reporting →
Polish DPO investigates OpenAI over ChatGPT false data and lack of transparency
The Polish data protection authority (UODO) is investigating a complaint against OpenAI concerning ChatGPT. The complainant alleges that ChatGPT generated false information about him, and that OpenAI failed to correct it or disclose what data it holds, violating GDPR principles of lawfulness, fairness and transparency. The complainant also claims OpenAI did not fulfil its information obligations under Article 12 and Article 5(1)(a) GDPR. UODO has stated it will examine the systemic compliance of OpenAI's data processing with European data protection law.
- Company involved
- OpenAI
- AI system involved
- ChatGPT
9 source articles · read the reporting →
EviCore denied heart catheterization for patient using algorithm
In fall 2021, Little John Cupp's doctor requested a left heart catheterization exam. EviCore, a company hired by UnitedHealthcare, denied the request twice using an algorithm called 'the dial' that adjusts thresholds for review. The algorithm flagged the request for review, and EviCore's doctors determined it was not medically necessary. Cupp did not receive the procedure and his symptoms continued.
- Company involved
- EviCore (a Cigna company)
- AI system involved
- the dial
6 source articles · read the reporting →
IRCC uses AI triage for Temporary Resident Visa applications
Immigration, Refugees and Citizenship Canada (IRCC) uses an AI system called Advanced Analytics to triage Temporary Resident Visa applications from India and China. The system categorizes applications into tiers, with Tier 1 approved automatically and others sent to human officers. Critics allege the system lacks transparency and may introduce bias, leading to visa refusals without clear rationale. The author, a Canadian immigration lawyer, is filing Federal Court cases on behalf of clients affected by refusals.
- Company involved
- Immigration, Refugees and Citizenship Canada (IRCC)
- AI system involved
- Advanced Analytics Triage of Overseas Temporary Resident Visa Applications
10 source articles · read the reporting →
Cigna StressWaves Test found unreliable and invalid in independent study
A study published in Scientific Reports evaluated the Cigna StressWaves Test (CSWT), an AI tool that claims to assess psychological stress from speech. The study found that the CSWT had poor test-retest reliability and poor validity compared to the Perceived Stress Scale. The authors warned that widespread availability of the tool could lead to misleading results and negative consequences for users making healthcare decisions. Cigna has not publicly responded to the findings.
- Company involved
- Cigna
- AI system involved
- Cigna StressWaves Test
4 source articles · read the reporting →
California EDD's automated fraud detection wrongly suspended 600,000 legitimate unemployment claims
In January 2021, the California Employment Development Department used Thompson Reuters automated batch review software to flag 1.1 million unemployment claims as potentially fraudulent. EDD stopped payments on those claims without prior notice. Later, over 600,000 were confirmed as legitimate after claimants used ID.me to verify their identity. The incident highlights the trade-off between fraud prevention and timely benefit access.
- Company involved
- California Employment Development Department (EDD)
- AI system involved
- Thompson Reuters Automated Batch Review
7 source articles · read the reporting →
Chattr.ai exposed job applicant data due to insecure Firebase rules
A security researcher discovered that Chattr.ai, an AI hiring system used by many fast food chains, had a Firebase database with insecure security rules. By registering a new user, the researcher gained full read/write access to the database, exposing personal data of job applicants and employees, including names, phone numbers, emails, and some plaintext passwords. The vulnerability was reported to Chattr.ai on 9 January and patched the next day, but the company did not provide further contact or thanks.
- Company involved
- Chattr.ai
- AI system involved
- Chattr.ai
6 source articles · read the reporting →
Teleperformance deploys AI to neutralise Indian call centre agents' accents
Teleperformance, the world's largest call centre operator, has announced it is using AI from Sanas to modify the accents of its Indian employees in real time. The technology, called accent translation, aims to make agents sound more neutral to native English speakers. The company invested $13 million in Sanas and gained exclusive rights. No specific incident of harm has been reported.
- Company involved
- Teleperformance
- AI system involved
- Sanas AI
6 source articles · read the reporting →
Serco Leisure issued enforcement notices for unlawful biometric monitoring of employees
Serco Leisure and associated trusts used facial recognition and fingerprint scanning to monitor employee attendance at 38 leisure facilities. The ICO found they unlawfully processed biometric data of over 2,000 employees. Enforcement notices were issued ordering them to stop.
- Company involved
- Serco Leisure Operating Limited
8 source articles · read the reporting →
NHS plans AI to listen to appointments and generate notes
The UK's National Health Service announced plans to use AI to automatically generate notes from patient appointments. Health Secretary Victoria Atkins said the scheme would reduce admin time. Privacy campaigners raised concerns about data security and accuracy, citing an incident where AI misheard the chief medical officer's name. The Department of Health and Social Care stated that patient confidentiality remains a top priority.
- Company involved
- National Health Service (NHS)
3 source articles · read the reporting →
Swedish welfare agency's AI system flags marginalized groups for fraud investigations
Försäkringskassan, Sweden's Social Insurance Agency, uses an AI risk-scoring system to flag welfare applicants for fraud investigations. The system disproportionately targets women, individuals with foreign backgrounds, low-income earners, and those without university degrees, according to an investigation by Lighthouse Reports and Svenska Dagbladet. Amnesty International has called for the system to be discontinued, citing violations of the right to equality and non-discrimination.
- Company involved
- Försäkringskassan (Swedish Social Insurance Agency)
6 source articles · read the reporting →
