AMS algorithm lacks transparency and may discriminate against job seekers
The Austrian Public Employment Service (AMS) uses an algorithm to classify job seekers into categories A, B, and C, determining their access to benefits and training. Scientists from TU Wien, WU Wien, and University of Vienna have criticised the algorithm for lacking transparency, as only two of 96 model variants have been published. They allege that the system may discriminate against women and people with migration background, and that job seekers are not informed about how the algorithm works or given a chance to appeal.
- Company involved
- AMS (Arbeitsmarktservice Österreich)
- AI system involved
- AMS-Algorithmus
10 source articles · read the reporting →
Southwest Airlines holiday meltdown due to crew scheduling software failure
In December 2022, Southwest Airlines experienced a catastrophic operational meltdown after its crew scheduling software failed to assign pilots and flight attendants to flights during a winter storm. The system, SkySolver and Crew Web Access, could not handle the volume of schedule changes, leading to over 15,800 flight cancellations and stranding thousands of passengers and crew. Southwest's CEO acknowledged the technology failure and promised upgrades, but the incident remained unresolved at the time of reporting.
- Company involved
- Southwest Airlines
- AI system involved
- SkySolver and Crew Web Access
10 source articles · read the reporting →
Trivago loses Australian appeal over misleading hotel rate rankings
The Federal Court of Australia has dismissed Trivago's appeal against a ruling that the online travel company breached Australian consumer law. The court found that Trivago's website used an algorithm that gave prominence to hotels paying higher fees, so the most prominent offers were not always the cheapest for consumers. Consumers may therefore have paid more for rooms, and hotels lost direct bookings. The case returns to the Federal Court for consideration of penalties and other orders sought by the Australian Competition and Consumer Commission.
- Company involved
- Trivago
10 source articles · read the reporting →
Austrian court lifts ban on AMS job-chance prediction algorithm
The Austrian Federal Administrative Court overturned a ban by the Data Protection Authority on the AMS algorithm, which predicts job chances of unemployed people. The system, trained on historical data including age, gender, and nationality, categorises individuals as high, medium or low chance of re-employment. Critics argue it discriminates against women and mothers, and that the data is now outdated due to the COVID-19 pandemic. The court ruled that the algorithm is lawful as long as a human advisor makes the final decision on training support.
- Company involved
- Arbeitsmarktservice (AMS)
- AI system involved
- AMS-Algorithmus
5 source articles · read the reporting →
Shipt's new V2 algorithm reduces pay for 41% of workers, study finds
A study by MIT Media Lab and Coworker.org found that Shipt's new V2 payment algorithm reduced pay for 41% of workers by an average of 11% per shop. The algorithm, rolled out in September 2020, replaced a transparent payment system with a black-box model. Workers pooled data to analyze the impact, revealing uneven distribution of earnings.
- Company involved
- Shipt
- AI system involved
- V2
10 source articles · read the reporting →
UBC students raise concerns over Proctorio's privacy breach and discrimination
The University of British Columbia (UBC) uses Proctorio, an algorithmic test proctoring software, for remote exams. In June 2020, a UBC student reported an issue with Proctorio, and the company's CEO released the student's support chat logs, sparking privacy concerns. The AMS of UBC published an open letter alleging that Proctorio discriminates against students of colour, those with disabilities, and other groups by flagging 'abnormal' behaviours and denying access. The letter calls on UBC to end its contract with Proctorio and conduct an external audit.
- Company involved
- University of British Columbia
- AI system involved
- Proctorio
10 source articles · read the reporting →
Proctorio's exam proctoring system allows human agents to review student feeds despite privacy claims
A security researcher analyzed Proctorio's Chrome extension and found evidence that the system allows human agents to review students' room scans and live ID checks, contrary to Proctorio's claims that only professors can access recordings. The system flags students based on behavioral metrics and can terminate exams. The researcher also raised concerns about discrimination against low-income students and privacy violations.
- Company involved
- Proctorio
- AI system involved
- Proctorio
10 source articles · read the reporting →
Allstate's proposed retention model charged big spenders more in Maryland
Allstate proposed an auto insurance rate adjustment plan in Maryland using a retention model algorithm that varied premium changes based on existing premiums. The algorithm would have increased rates up to 20% for customers already paying the highest premiums while capping increases at 5% for others, and would have denied meaningful discounts to thousands owed reductions. The Maryland Insurance Administration rejected the plan as discriminatory, but Allstate claims it was withdrawn and has continued proposing similar models elsewhere. The proposal never took effect in Maryland.
- Company involved
- Allstate Corporation
- AI system involved
- retention model
9 source articles · read the reporting →
New York City's McKinsey-led jail violence program manipulated data, violence increased
New York City paid McKinsey & Company $27.5 million to reduce violence at Rikers Island jail complex. McKinsey designed a predictive algorithm called the Housing Unit Balancer and Restart housing units, but jail officials and McKinsey consultants stacked the units with compliant inmates to artificially lower violence numbers. Violence actually increased by nearly 50% during the project. The city eventually decided to close Rikers.
- Company involved
- New York City Department of Correction
- AI system involved
- Housing Unit Balancer (HUB)
10 source articles · read the reporting →
Portland Metro ends Replica partnership over data privacy concerns
Portland Metro, an elected regional government in Oregon, ended its pilot project with movement data company Replica after a disagreement about data sharing. Portland Metro requested raw, disaggregated data, which Replica refused to provide, citing user privacy concerns. The partnership was terminated without payment.
- Company involved
- Portland Metro
- AI system involved
- Replica
10 source articles · read the reporting →
Uber algorithm reinforced gender pay gap in compensation offers to new hires
Uber used an algorithm to determine compensation offers for new hires, allegedly paying women less than men for similar roles. The algorithm was designed to protect existing shareholders and save money, but it reinforced existing gender pay gaps. The system is now being altered, according to The Information.
- Company involved
- Uber
7 source articles · read the reporting →
Airbnb smart-pricing algorithm increased racial revenue gap, study finds
A study by Carnegie Mellon University found that Airbnb's smart-pricing algorithm increased the revenue gap between White and Black hosts, even though it narrowed the gap among hosts who adopted it. The algorithm, which sets daily prices automatically, was adopted by fewer Black hosts, so its suggested prices were closer to the optimum for White hosts. The researchers said the tool could reduce racial disparities only if more Black hosts adopted it.
- Company involved
- Airbnb
- AI system involved
- Airbnb smart-pricing algorithm
10 source articles · read the reporting →
PwC develops facial recognition tool to monitor employees working from home
Accounting giant PwC has developed a facial recognition tool that logs when employees are absent from their computer screens while working from home. The tool, intended for financial institutions, requires workers to provide written reasons for any absences, including toilet breaks. Commentators have criticised the tool as a huge invasion of privacy, with concerns about damage to trust and increased stress. PwC stated that the technology is designed to help regulated institutions meet compliance obligations and that voluntary consent of traders is essential.
- Company involved
- PwC
8 source articles · read the reporting →
USPS algorithm RRECS causes pay cuts for two-thirds of rural mail carriers
The United States Postal Service (USPS) implemented a new algorithm, RRECS, to evaluate rural carrier routes and determine pay. Due to flaws in how carriers scanned packages, the algorithm underestimated route times, resulting in pay cuts for 66 per cent of rural carriers, some losing thousands of dollars annually. Carriers report that they were not adequately trained on the system and that the cuts are scheduled to take effect, though they have been postponed multiple times. The USPS stated that the system is the result of a nationally negotiated agreement.
- Company involved
- United States Postal Service
- AI system involved
- RRECS
9 source articles · read the reporting →
Amazon sellers lose thousands after RepricerExpress pricing glitch
A software glitch in RepricerExpress's automated repricing system caused products sold by Amazon sellers to be priced at 1p, leading to significant financial losses for the sellers. RepricerExpress apologised but did not offer compensation, while Amazon said it had reached out to affected sellers but many reported no response. Some sellers have instructed lawyers to take legal action against RepricerExpress and Amazon.
- Company involved
- RepricerExpress
- AI system involved
- RepricerExpress
10 source articles · read the reporting →
Bruce Springsteen addresses Ticketmaster dynamic pricing outrage
Bruce Springsteen addressed fan outrage over Ticketmaster's dynamic pricing for his 2023 tour, which led to ticket prices skyrocketing up to $5,000. Springsteen said he told his team to do what other artists do and that most tickets were affordable. He offered refunds to fans who complained.
- Company involved
- Ticketmaster
- AI system involved
- Ticketmaster dynamic pricing
9 source articles · read the reporting →
ScaleFactor reportedly failed to deliver promised automated bookkeeping software
ScaleFactor, an Austin-based startup, is reported to have failed to deliver the automated, real-time bookkeeping tools it promised customers, instead relying on human bookkeepers and a Filipino contract accounting firm. The company told Forbes in June that it was shutting down, initially blaming the pandemic, but Forbes later reported that its problems predated Covid-19. Investors reportedly came to see the company as more of a services business than a software platform, and pulled funding after a pivot to a marketplace model. No legal or regulatory action is reported.
- Company involved
- ScaleFactor
10 source articles · read the reporting →
UnitedHealth used ALERT algorithm to limit mental health coverage, regulators found
ProPublica reports that UnitedHealth Group's Optum subsidiary used the ALERT algorithm to flag mental health patients and therapists as outliers, leading to therapy coverage denials. Regulators in California, Massachusetts and New York alleged this breached the federal Mental Health Parity and Addiction Equity Act, and UnitedHealth agreed to restrict the system in those jurisdictions. The company denies wrongdoing and says its programmes are compliant. Affected patients, including Medicaid enrollees, were left to pay out-of-pocket or go without care.
- Company involved
- UnitedHealth Group
- AI system involved
- ALERT
5 source articles · read the reporting →
Presto Automation uses off-site human agents to double-check AI drive-thru orders
Presto Automation Inc, which markets an AI voice assistant for drive-thru ordering, used off-site human agents in countries including the Philippines to double-check orders in more than 70% of customer interactions, according to SEC filings reported by Bloomberg. The company told Bloomberg that the process helps train its system and should reduce human intervention over time. Presto's drive-thru AI is used in more than 400 restaurants, including Del Taco, Carl's Jr and Checkers, and its stock fell more than 10% after the reports.
- Company involved
- Presto Automation Inc.
8 source articles · read the reporting →
AI-generated article misstates Roadzen's Q1 revenue expectations
An AI-generated article on The Motley Fool incorrectly stated that Roadzen's analyst expectations for Q1 revenue were over $21 million, implying a revenue miss of more than 50%. Roadzen clarified that these figures were never issued by its covering analysts and that its actual revenue of $10.9 million was in line with estimates. The Motley Fool later corrected the article and added an editor's note acknowledging the error.
- Company involved
- The Motley Fool
4 source articles · read the reporting →
Delta uses AI from Fetcherr for domestic ticket pricing
Delta Air Lines is using generative AI from Fetcherr to determine some domestic flight prices, currently covering 3% of its network with plans to reach 20% by end of 2025. Democratic senators expressed concern that the AI could be used for individualized pricing based on personal data, leading to higher fares. Delta denies using personal data in pricing and states it complies with regulations. No actual harm has been reported.
- Company involved
- Delta Air Lines
- AI system involved
- Fetcherr
8 source articles · read the reporting →
GMB members report declining earnings since Uber introduced dynamic pricing
GMB Union has welcomed a report into Uber's dynamic pricing that appears to show it benefits Uber more than drivers. GMB members have reported declining earnings since the introduction of dynamic pricing. The union has called for greater transparency around earnings in the gig economy and will review the report before consulting members on next steps.
- Company involved
- Uber
- AI system involved
- dynamic pricing
8 source articles · read the reporting →
Kroger's digital price tags investigated for surge pricing concerns
U.S. Senators Elizabeth Warren and Bob Casey sent a letter to Kroger CEO Rodney McMullen raising concerns about the company's use of Electronic Shelving Labels (ESLs) for dynamic pricing. The system, called EDGE shelf and developed with Microsoft, can change prices based on time of day or weather, potentially leading to price gouging. The senators warned that this could harm consumers by increasing grocery costs and requested information from Kroger about its use of the technology.
- Company involved
- Kroger
- AI system involved
- EDGE shelf (Electronic Shelving Labels)
10 source articles · read the reporting →
Audit of RisCanvi finds biases and reliability issues in criminal justice system
Eticas conducted an adversarial audit of RisCanvi, an AI risk assessment tool used in Catalonia's criminal justice system. The audit uncovered biases in risk classifications against specific demographics and significant reliability issues. The findings call for fairer practices in criminal justice AI.
- Company involved
- Catalonia's criminal justice system
- AI system involved
- RisCanvi
4 source articles · read the reporting →
