Audit finds RisCanvi justice algorithm is opaque, unreliable, and unfair

An audit revealed serious shortcomings associated with an algorithm used in Catalonia's criminal justice system to predict how likely prisoners are to reoffend. What happened An "adversarial" audit of RisCanvi conducted by algorithmic auditing company Eticas concluded that the algorithm suffered from “substantial deficiencies” in the reliability of its decisions by failing to standardise outcomes and for being potentially biased aga inst specific types of peo ple, p articularly those with challenging backgrounds. Eticas also found that the system had not adhered to Spain’s regulations on the use of automated decision-making systems, which have required audits since 2016, and that it had not undergone an Al gorithmic Impact Assessment or rigorous testing. Furthermore, the audit concluded that inmates, who are the people most affected by the system, were unaware of its existence, while staff using it lacked proper training and understanding of how it worked. Why it happened RisCanvi's shortcomings appear to stem from the poor design and training of the system, and from inadequate oversight and poor transparency. It is unclear whether its transparency and accountability shortcomings constituted a deliberate attempt to conceal the system's inadequacies. What it means The case highlights the need for greater transparency and accountability of so-called "high-risk" systems that can have a major impact on people's lives. It also suggests the RisCanvi algorithm needs to be comprehensively overhauled and more rigorous testing and oversight introduced. System 🤖 RisCanvi Developer: Antonio Andrés Pueyo Country: Spain Sector: Govt - justice Purpose: Predict prisoner recidivism risk Technology: Risk assessment algorithm Issue: Accountability; Accuracy/reliability; Bias/discrimination; Transparency Investigations, assessments, audits 👁️ Eticas. RisCanvi Adversarial Audit Eticas. Case study

Date it happened

2024-07-01

Organisation involved

Catalonian Department of Justice

Product, system or model

RisCanvi

This is a record of what was reported, not a finding that anyone broke the law. If it names your organisation and you believe it is wrong, the corrections process is free and open to everyone.