Airbnb smart-pricing algorithm increased racial revenue gap, study finds
A study by Carnegie Mellon University found that Airbnb's smart-pricing algorithm increased the revenue gap between White and Black hosts, even though it narrowed the gap among hosts who adopted it. The algorithm, which sets daily prices automatically, was adopted by fewer Black hosts, so its suggested prices were closer to the optimum for White hosts. The researchers said the tool could reduce racial disparities only if more Black hosts adopted it.
- Company involved
- Airbnb
- AI system involved
- Airbnb smart-pricing algorithm
10 source articles · read the reporting →
University of Chicago report finds US healthcare algorithms rife with bias
A University of Chicago report alleges that algorithms used by hospitals, insurers and other businesses across the US are biased along racial and economic lines. The algorithms help triage patients, predict diabetes and flag those needing extra care, but the report says flawed products were introduced with little oversight. It says inequitable treatment has continued for more than a decade.
7 source articles · read the reporting →
Thomas Raynard James v. Detective Kevin Conley, et al. (S.D. Florida): AI-hallucinated content in court filing, Bar Referral
AI generated hallucinated content in a court filing, affecting the legal process and the lawyers who filed it.
1 source article · read the reporting →
TOV Realty, LLC v. Suarez; Kosel Equity, LLC v. MacGregor (SC Connecticut): AI-hallucinated content in court filing, 6 hours CLE;…
The AI generated legal briefs containing fabricated citations, which were submitted to the Connecticut Supreme Court.
- Company involved
- GLG Law LLC
- AI system involved
- ChatGPT
1 source article · read the reporting →
Study finds credit score algorithms less accurate for minorities
A study of 50 million US consumers found that credit scoring algorithms used by mortgage lenders are less accurate for minority and low-income applicants due to sparse credit data, leading to higher rejection rates. The inaccuracy is due to noise in the data, not bias, so fairer algorithms cannot fix it. The study suggests that adjusting for bias had no effect, and that addressing the inaccuracy could reduce disparities by 50%.
- Company involved
- Mortgage lenders (unnamed)
- AI system involved
- Credit scoring algorithms
6 source articles · read the reporting →
LoanDepot algorithm denied mortgage to Black couple in Charlotte
In August 2019, Crystal Marie and Eskias McDaniels, a Black couple, were denied a mortgage for a house in Charlotte, North Carolina, by loanDepot's automated underwriting algorithm. The algorithm rejected the application because Crystal Marie was a contractor, not a full-time employee, despite her high credit score and income. After the couple enlisted their real estate agent and employer to intervene, the lender reversed the decision and cleared them to close. The couple alleged that race played a role in the denial, which loanDepot denied.
- Company involved
- loanDepot
- AI system involved
- Classic FICO and Fannie Mae/Freddie Mac automated underwriting software
10 source articles · read the reporting →
Company fires HR team after ATS auto-rejects manager's CV due to filtering error
A company's applicant tracking system (ATS) auto-rejected qualified candidates' resumes for three months because it was filtering for the outdated framework AngularJS instead of the required Angular framework. The manager discovered the flaw by submitting his own CV under a pseudonym and found it was rejected within seconds. After the manager reported the issue to upper management, the company investigated and dismissed half of its HR team. No legal action or regulatory involvement is reported.
4 source articles · read the reporting →
LAPD told officers to collect social media data on every civilian stopped
The Los Angeles Police Department directed officers to record the social media accounts of every civilian they interviewed, including those not arrested or accused of a crime, according to records obtained by the Brennan Center for Justice. The field interview cards, which also ask for social security numbers, have raised civil liberties concerns. Prosecutors have accused three officers of using the cards to falsely label civilians as gang members. The LAPD said the policy was being updated but did not respond to questions about its use of social media monitoring tools including Media Sonar and Geofeedia.
- Company involved
- Los Angeles Police Department (LAPD)
- AI system involved
- Field interview cards, Media Sonar, Geofeedia
10 source articles · read the reporting →
Met Police buys £3m retrospective facial recognition system
The Metropolitan Police Service (MPS) has awarded a £3 million, four-year contract to Northgate Public Services for a new retrospective facial-recognition (RFR) system to be deployed within three months. RFR processes biometric information from historic CCTV, social media, and other images to identify suspects and missing persons, operating retroactively unlike live facial recognition. The procurement was approved by the Mayor's Office for Policing and Crime in August 2021. The MPS states that human-in-the-loop decision-making will be used, but digital rights groups and a former biometrics commissioner have raised concerns about potential discrimination, overrepresentation of marginalised groups in watch lists, and lack of a legislative framework.
- Company involved
- Metropolitan Police Service
- AI system involved
- Retrospective facial-recognition software
9 source articles · read the reporting →
JMOR Properties v. Artist Alley Townhomes et al. (CA Florida (4d)): AI-hallucinated content in court filing, Bar Referral
The AI generated fabricated legal citations that were included in a court filing, affecting the court and the opposing party.
1 source article · read the reporting →
Southland Homes & Real Estate and Investment, LLC v. Lam (SC California): AI-hallucinated content in court filing, Monetary Sanctions; Bar…
The AI system generated legal briefs containing fabricated case citations, which were filed in court, affecting the court and the opposing party.
1 source article · read the reporting →
California DMV suspends Cruise LLC driverless and deployment permits
The California Department of Motor Vehicles suspended Cruise LLC's autonomous vehicle deployment and driverless testing permits, saying the vehicles were not safe for public operation and that Cruise had misrepresented safety information. The suspension took effect immediately under California regulations. Cruise may apply to reinstate the permits once it satisfies the department's requirements. Its permit for testing with a safety driver was not affected.
- Company involved
- Cruise LLC
10 source articles · read the reporting →
Immigration NZ uses data model to predict likely troublemakers among overstayers
Immigration New Zealand has been piloting a data modelling system that uses demographic information such as age, gender, and ethnicity of overstayers to predict which individuals are likely to commit crime or incur hospital debts. The system then targets those individuals for accelerated deportation. Critics, including immigration lawyers and a lobby group, have likened the programme to 'Minority Report' and allege it amounts to racial profiling. Immigration Minister Iain Lees-Galloway only learned of the programme on the day of the article and said he would seek a full briefing, but maintained it is not racial profiling and does not breach the Human Rights Act.
- Company involved
- Immigration New Zealand
10 source articles · read the reporting →
BC Tribunal Confirms Companies Remain Liable for AI Chatbot-Created Information - Lexology
The AI chatbot provided inaccurate information to a user.
1 source article · read the reporting →
Lyft approved driver with terrorism conviction due to background check failure
Lyft allowed a driver with a conviction for aiding terrorism to drive for the ride-sharing service. The driver, Raja L. Khan, had served seven years and was denied by Uber and the city of Chicago, but Lyft's independent background check provider approved him. After WGN Investigates reported the issue, Lyft deactivated the driver and apologised, stating the approval was unacceptable and that it would re-run background checks.
- Company involved
- Lyft
7 source articles · read the reporting →
Audit of RisCanvi finds biases and reliability issues in criminal justice system
Eticas conducted an adversarial audit of RisCanvi, an AI risk assessment tool used in Catalonia's criminal justice system. The audit uncovered biases in risk classifications against specific demographics and significant reliability issues. The findings call for fairer practices in criminal justice AI.
- Company involved
- Catalonia's criminal justice system
- AI system involved
- RisCanvi
4 source articles · read the reporting →
Dutch probation service's OXREC algorithm flawed, leading to incorrect recidivism risk assessments
The Dutch Inspectorate of Justice and Security (Inspectie JenV) published a report finding that the probation service's (Reclassering) OXREC algorithm contains serious flaws, including swapped formulas and incorrect numbers, causing about a quarter of risk assessments to be wrong. The algorithm, used since 2018 for about 44,000 cases per year, also uses variables that can lead to discrimination, such as neighborhood score and income. The Inspectorate recommended immediate correction or temporary suspension. The probation service announced it would temporarily stop using OXREC.
- Company involved
- Reclassering Nederland
- AI system involved
- OXREC
4 source articles · read the reporting →
NHTSA investigation PE24016: Unexpected ADS behavior
Waymo's automated driving system exhibited unexpected behavior during driving.
- Company involved
- Waymo
1 source article · read the reporting →
Citizens Advice finds ethnicity penalty in car insurance pricing
Citizens Advice conducted exploratory research into car insurance pricing and found that people of colour may be paying £250 more per year than White people. The research suggests that areas with large communities of colour may be identified as more risky by algorithms, even when objective risk factors are controlled. Citizens Advice has called on the Financial Conduct Authority to investigate the issue.
9 source articles · read the reporting →
TransUnion AI tenant screening denied 75-year-old man apartment due to mistaken littering conviction
Chris Robinson, then 75, applied for a senior living apartment in California. The property manager used an AI screening program from TransUnion, which assigned him a low score based on a mistaken conviction for littering that belonged to a different person with the same name in Texas. Robinson lost the apartment and his application fee. A federal class-action lawsuit against TransUnion moved toward a $11.5 million settlement in 2023.
10 source articles · read the reporting →
MAA, JBG to pay DC $9.3M total to settle RealPage case - Multifamily Dive
The system set rent prices for tenants, causing them to pay higher rents.
- Company involved
- MAA (Mid-America Apartments) and JBG Smith
- AI system involved
- RealPage Revenue Management Software
1 source article · read the reporting →
Pasco Sheriff's Office used algorithm to target potential future criminals and schoolchildren
The Pasco Sheriff's Office operates an intelligence-led policing programme that uses an algorithm to identify people who might break the law based on criminal histories and social networks. Deputies are sent to the homes of those flagged, even without evidence of a crime, and former deputies allege they were ordered to make targets' lives miserable. The agency also keeps a list of more than 400 schoolchildren predicted to 'fall into a life of crime', built from data such as grades and child welfare records, without informing the children or their parents. Civil liberties groups are considering lawsuits and public advocacy campaigns, and experts have called the programmes 'morally repugnant'.
- Company involved
- Pasco Sheriff's Office
10 source articles · read the reporting →
Insurer drops California homeowner after aerial photo analysis
Cindy Picos was dropped by her home insurer after the company used aerial photos to assess her roof. The insurer refused to let her see the photos, and she believes her roof is in fine shape. The incident highlights the growing use of drones and aerial imagery by insurers to inspect properties without in-person visits.
2 source articles · read the reporting →
Broward College student flagged by Honorlock and accused of cheating
A 17-year-old Black student at Broward College was flagged by Honorlock, a remote proctoring system, during an online biology exam in February 2022. Her professor reviewed the recording and accused her of academic dishonesty for looking down and away from the screen; the college found her responsible for noncompliance and gave her a zero and a warning. The student says she was only thinking and fidgeting, and the article reports that the video is ambiguous. Honorlock says it does not definitively identify cheaters and the final decision rests with the school.
- Company involved
- Broward College
- AI system involved
- Honorlock
2 source articles · read the reporting →
