UK welfare fraud AI system criticised as biased and opaque

An AI-powered system used by the UK government to detect welfare fraud has been found to exhibit bias based on age, disability, marital status and nationality, contradicting earlier assurances that it posed no discrimination concerns. What happened Internal assessments revealed by a Guardian investigation revealed that "statistically significant outcome disparities" in the AI system's recommendations for fraud investigations. Designed to identify fraudulent claims for Universal Credit, the system disproportionately flags individuals from certain demographic groups. The incident sparked outrage and distrust in the UK government's approach to AI , with campaigners criticising the Depa rtment of Work and Pen sions ' (DWP) "hurt first, fix later" approach and calling for increased transparency and accountability in government's use of th e technology . Why it happened The DWP failed to conduct comprehensive fairness analyses for all protected characteristics. No evaluations have been made regarding potential biases related to race, sex, sexual orientation, religion, pregnancy and gender reassignment status. Critics argue that this lack of thorough assessment undermines the system's fairness and inclusivity. What it means The case highlights the need for stronger regulatory frameworks and comprehensive fairness analyses before deploying AI systems in public services. System 🤖 Advances Developer: Department for Work and Pensions (DWP) Country: UK Sector: Govt - welfare Purpose: Detect fraud Technology: Machine learning Issue: Fairness ; Transparency

Date it happened

2024-12-01

Organisation involved

Department for Work and Pensions (DWP)

Product, system or model

Advances

This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.