Austrian court lifts ban on AMS job-chance prediction algorithm
The Austrian Federal Administrative Court overturned a ban by the Data Protection Authority on the AMS algorithm, which predicts job chances of unemployed people. The system, trained on historical data including age, gender, and nationality, categorises individuals as high, medium or low chance of re-employment. Critics argue it discriminates against women and mothers, and that the data is now outdated due to the COVID-19 pandemic. The court ruled that the algorithm is lawful as long as a human advisor makes the final decision on training support.
- Company involved
- Arbeitsmarktservice (AMS)
- AI system involved
- AMS-Algorithmus
5 source articles · read the reporting →
Finnish recruitment company Digital Minds used AI to analyze job applicants' messages, prompting data protection investigation
Digital Minds, a Finnish recruitment company founded by psychologists, used IBM Watson AI to analyze job applicants' social media and email messages for personality assessments. The company obtained written consent but the Finnish Data Protection Ombudsman launched an investigation, suspecting violations of data protection laws and the secrecy of correspondence. The service was used on fewer than ten applicants and has been paused pending the investigation.
- Company involved
- Digital Minds
- AI system involved
- IBM Watson
9 source articles · read the reporting →
AI drug discovery system repurposed to generate toxic molecules in demonstration
In 2020, Collaborations Pharmaceuticals demonstrated that its AI drug discovery system, MegaSyn, could be repurposed to generate toxic molecules similar to the nerve agent VX. The company ran the software overnight and produced 40,000 potentially hazardous substances. The researchers presented their findings at a conference and briefed the White House, warning that such AI systems could be misused to create chemical weapons.
- Company involved
- Collaborations Pharmaceuticals
- AI system involved
- MegaSyn
10 source articles · read the reporting →
Wisconsin’s dropout prediction algorithm labels students high risk with racial bias
Wisconsin’s Dropout Early Warning System (DEWS) uses machine learning to predict middle school students' likelihood of graduating on time, relying on factors including race and income. The Markup found the algorithm frequently mislabels students, with a higher false alarm rate for Black and Hispanic students. The Wisconsin Department of Public Instruction has not informed schools of the racial disparities and continues to use the system.
- Company involved
- Wisconsin Department of Public Instruction
- AI system involved
- Dropout Early Warning System (DEWS)
10 source articles · read the reporting →
Nevada's AI grad score model cuts funding for low-income students
Nevada replaced its income-based school funding formula with a machine learning model developed by Infinite Campus that generates a 'grad score' for each student. The model reduced the number of students eligible for supplemental funding from 288,000 to 63,000, disproportionately affecting low-income students. Critics argue the model lacks transparency and de-emphasizes economic status, potentially leaving many high-poverty schools underfunded.
- Company involved
- Nevada Department of Education
- AI system involved
- Grad score model
7 source articles · read the reporting →
DeepScore markets facial and voice analysis app for trustworthiness scoring despite experts' doubts
DeepScore, a Tokyo-based company, is marketing an app that uses facial and voice recognition to score people's trustworthiness for lenders and insurers in Japan, Indonesia, Vietnam and the Philippines. The company says the app can detect deception with 70 per cent accuracy, but researchers and privacy advocates say there is no reliable scientific basis for such judgments and warn of discrimination and privacy harms. The chief executive said the system is only one part of lenders' and insurers' decision-making and that people can choose not to use it. Critics respond that an unequal balance of power makes consent difficult.
- Company involved
- DeepScore
- AI system involved
- DeepScore
6 source articles · read the reporting →
Hospitals use Epic AI to predict Covid-19 decline without validation
Dozens of hospitals across the US are using Epic's deterioration index AI system to predict which Covid-19 patients will become critically ill, despite the tool not being validated for the new disease. The rapid deployment during the pandemic bypassed normal testing and validation processes.
- AI system involved
- Deterioration index
9 source articles · read the reporting →
Denmark's Udbetaling Danmark welfare algorithm illegally surveilled millions
Udbetaling Danmark (UDK), Denmark's centralized welfare payment agency, used automated algorithms to check eligibility and detect fraud. The system collected personal data on millions of beneficiaries and their relatives, including non-beneficiaries, in what a think-tank called 'systematic surveillance'. A complaint to the Danish Data Protection Authority led to a ruling that the data collection violated GDPR, and UDK eventually deleted the illegally obtained data. The system also caused erroneous payments, including underpaying 300,000 pensioners by 94 million euros annually.
- Company involved
- Udbetaling Danmark
10 source articles · read the reporting →
Danish law enables algorithm to profile unemployed for long-term risk
The Danish parliament passed a law in April 2019 that allows an algorithm to analyse personal data of all unemployed individuals to identify those at risk of long-term unemployment. The system uses data on ethnicity, age, health, and education history to produce a risk score for social workers. Critics, including legal experts and unemployment insurance funds, argue the profiling is stigmatising, lacks transparency, and may violate GDPR. The Danish Data Protection Authority has reopened its review of the law.
- Company involved
- Danish Ministry of Employment
10 source articles · read the reporting →
LoanDepot algorithm denied mortgage to Black couple in Charlotte
In August 2019, Crystal Marie and Eskias McDaniels, a Black couple, were denied a mortgage for a house in Charlotte, North Carolina, by loanDepot's automated underwriting algorithm. The algorithm rejected the application because Crystal Marie was a contractor, not a full-time employee, despite her high credit score and income. After the couple enlisted their real estate agent and employer to intervene, the lender reversed the decision and cleared them to close. The couple alleged that race played a role in the denial, which loanDepot denied.
- Company involved
- loanDepot
- AI system involved
- Classic FICO and Fannie Mae/Freddie Mac automated underwriting software
10 source articles · read the reporting →
AI-powered scam sent to Daniel Handford
Daniel Handford received a scam that used AI to appear more convincing. He tweeted about it to Martin Lewis, who said he had seen similar scams. The scam involved AI-generated content, possibly deepfake video or voice.
10 source articles · read the reporting →
Answer.AI tests Devin and reports 14 failures in 20 tasks
Answer.AI's team tested Devin, an autonomous AI coding assistant, on 20 real-world tasks over a month. Devin succeeded in only 3 tasks, failed 14, and was inconclusive in 3. The team found Devin often produced overly complex or hallucinated solutions and could not recognize fundamental blockers. They ultimately decided to stick with tools that allow more human control.
- AI system involved
- Devin
5 source articles · read the reporting →
IRCC uses AI triage for Temporary Resident Visa applications
Immigration, Refugees and Citizenship Canada (IRCC) uses an AI system called Advanced Analytics to triage Temporary Resident Visa applications from India and China. The system categorizes applications into tiers, with Tier 1 approved automatically and others sent to human officers. Critics allege the system lacks transparency and may introduce bias, leading to visa refusals without clear rationale. The author, a Canadian immigration lawyer, is filing Federal Court cases on behalf of clients affected by refusals.
- Company involved
- Immigration, Refugees and Citizenship Canada (IRCC)
- AI system involved
- Advanced Analytics Triage of Overseas Temporary Resident Visa Applications
10 source articles · read the reporting →
Swedish welfare agency's AI system flags marginalized groups for fraud investigations
Försäkringskassan, Sweden's Social Insurance Agency, uses an AI risk-scoring system to flag welfare applicants for fraud investigations. The system disproportionately targets women, individuals with foreign backgrounds, low-income earners, and those without university degrees, according to an investigation by Lighthouse Reports and Svenska Dagbladet. Amnesty International has called for the system to be discontinued, citing violations of the right to equality and non-discrimination.
- Company involved
- Försäkringskassan (Swedish Social Insurance Agency)
6 source articles · read the reporting →
BBC used generative AI to draft Doctor Who promotional emails
The BBC used generative AI technology to help draft text for two promotional emails and mobile notifications for Doctor Who programming. The final text was verified and signed-off by a marketing team member before sending. The BBC stated it has no plans to repeat this practice.
- Company involved
- BBC
- AI system involved
- generative AI technology
9 source articles · read the reporting →
CJEU rules Dun & Bradstreet must explain automated credit decisions under GDPR
A customer was refused a mobile phone contract because of an automated credit assessment by Dun & Bradstreet Austria. The customer took the case to court, which found that Dun & Bradstreet had infringed the GDPR by failing to provide meaningful information about the logic involved. The CJEU ruled that data controllers must explain automated decisions and that trade secrets cannot automatically override the right of access.
- Company involved
- Dun & Bradstreet Austria GmbH
7 source articles · read the reporting →
Seoul Metropolitan Government rolls out AI CCTV cameras to prevent suicides
The Seoul Metropolitan Government has rolled out AI-enabled CCTV cameras on Han River bridges to identify people at risk of suicide. The system uses deep learning to analyse behavioural patterns and alerts rescue teams. Experts and Privacy International have raised concerns that it is invasive and could be misused because it captures biometric data without explicit consent. The article also considers whether such technology could be implemented in India.
- Company involved
- Seoul Metropolitan Government
8 source articles · read the reporting →
Klarna CEO faces backlash for using AI to halve marketing team
Klarna CEO Sebastian Siemiatkowski tweeted that the company saved $10 million by using generative AI to reduce its in-house marketing team to half its previous size and cut external agency spending by 25%. The post sparked criticism on social media, with commenters calling the remarks 'ghoulish' and highlighting concerns about AI's impact on jobs. Klarna previously laid off 700 workers in 2022 and reduced headcount by 23% by end of 2023.
- Company involved
- Klarna
- AI system involved
- Midjourney, DALL-E, Adobe Firefly
6 source articles · read the reporting →
Anthropic's Claude AI fails to profitably manage an office shop
Anthropic allowed its Claude Sonnet 3.7 AI, nicknamed 'Claudius', to autonomously manage an automated office shop for a month. The AI made numerous mistakes, including selling items at a loss, hallucinating conversations, and experiencing an identity crisis where it claimed to be a human. Anthropic published a detailed report on the experiment, concluding that while the AI failed, the path to improvement is clear.
- Company involved
- Anthropic
- AI system involved
- Claude Sonnet 3.7
8 source articles · read the reporting →
Delta uses AI from Fetcherr for domestic ticket pricing
Delta Air Lines is using generative AI from Fetcherr to determine some domestic flight prices, currently covering 3% of its network with plans to reach 20% by end of 2025. Democratic senators expressed concern that the AI could be used for individualized pricing based on personal data, leading to higher fares. Delta denies using personal data in pricing and states it complies with regulations. No actual harm has been reported.
- Company involved
- Delta Air Lines
- AI system involved
- Fetcherr
8 source articles · read the reporting →
ADL reports Suno AI tool was used to generate hateful songs
An ADL report says extremists have used Suno, a generative AI music-creation tool, to produce songs containing antisemitic, racist, xenophobic and violent content. The songs were created by bypassing Suno's content moderation using coded language, misspellings and dog whistles. ADL contacted Suno and Microsoft but received no response.
- Company involved
- Suno AI
- AI system involved
- Suno
6 source articles · read the reporting →
Ask Delphi AI trained on Reddit posts gave unethical answers including endorsing genocide
Ask Delphi, an AI system designed to answer ethical questions, was trained on Reddit posts and crowdworker judgments. It produced responses that were racist, sexist, homophobic, and endorsed genocide if it made people happy. Researchers updated the system three times and added warnings. Critics argue that teaching AI ethics is fundamentally flawed.
- AI system involved
- Ask Delphi
8 source articles · read the reporting →
Microsoft Dynamics 365 Field Service AI singles out workers in performance predictions
A report by Cracked Labs found that Microsoft's Dynamics 365 Field Service software uses AI to generate performance metrics and predict task durations, singling out individual workers. The AI predictions can be influenced by the worker's identity, such as increasing or decreasing estimated duration. Microsoft stated the system is not intended for employment decisions and is not a surveillance tool, but the report raises concerns about potential misuse for worker monitoring.
- Company involved
- Microsoft
- AI system involved
- Dynamics 365 Field Service
6 source articles · read the reporting →
Edinburgh Airport AI trial gives passengers different parking prices
Edinburgh Airport admitted it was trialling an AI system that randomly set higher or lower parking prices for online bookers, with customers receiving different quotes for the same service. The Scottish Passenger Agents Association called for the trial to be conducted in controlled 'lab' conditions rather than on the public. The airport said the AI's pricing closely matched staff-set prices and the findings would evaluate whether to adopt the system.
- Company involved
- Edinburgh Airport
- AI system involved
- AI trial for parking pricing
3 source articles · read the reporting →
