Where automated decisions went wrong

Amnesty International reveals Serbian spyware targeting journalists and activists

Amnesty International's Security Lab found that Serbian authorities used Cellebrite tools and a previously unknown spyware named 'NoviSpy' to covertly infect the phones of independent journalist Slaviša Milanov and several activists. The infections occurred while devices were unattended during police or BIA interviews. The report alleges this is part of a wider crackdown on civil society, violating rights to privacy and free expression.

Company involved

Serbian Security Information Agency (BIA)

AI system involved

NoviSpy

7 source articles · read the reporting →

Microsoft funded Israeli facial recognition firm surveilling West Bank Palestinians

Microsoft invested in AnyVision, an Israeli facial recognition company whose technology powers a secret military surveillance project in the West Bank. The system, called Better Tomorrow, identifies and tracks Palestinians in live camera feeds. Microsoft said it would audit AnyVision for compliance with its ethical principles.

Company involved

Israeli Defense Forces

AI system involved

Better Tomorrow

10 source articles · read the reporting →

King's Cross developer scraps facial recognition after public outcry

The developer of the King's Cross site in London admitted it had used facial recognition software in two cameras on a busy pedestrian street between May 2016 and March 2018 without public consent. Following a backlash and questions from the Information Commissioner's Office, the developer announced it would not deploy the technology in the future and had abandoned plans for wider use. The system was used to help police detect crime, but critics highlighted the lack of transparency and privacy concerns.

Company involved

King's Cross development (Argent-led consortium)

10 source articles · read the reporting →

Swedish police fined for unlawful use of Clearview AI facial recognition

Sweden's data protection authority IMY fined the Swedish Police Authority €250,000 for unlawfully using Clearview AI's facial-recognition app. The police used the app between autumn 2019 and March 2020 without authorisation or a required data protection impact assessment. The IMY ordered the police to ensure Clearview AI deletes the data and to train employees to avoid future breaches.

Company involved

Swedish Police Authority

AI system involved

Clearview AI

3 source articles · read the reporting →

Verkada employees used surveillance system to capture and share images of female colleagues with sexual comments

A group of male employees at Verkada accessed the company's video surveillance system to capture images of female employees without their knowledge and shared them in a Slack channel with graphic sexual comments. The incident, which occurred in 2019, highlighted a frat-like culture and alleged discrimination against women and minorities. Verkada responded by taking disciplinary action against some employees, including financial penalties and termination, but critics say the response was insufficient.

Company involved

Verkada

AI system involved

Verkada video surveillance system

5 source articles · read the reporting →

HP webcam facial tracking fails to follow black man's face

A black man, Desi, posted a YouTube video showing that HP's webcam facial tracking software failed to follow his face while it tracked a white woman's face. HP acknowledged the issue and said it was working with partners to investigate. The company attributed the problem to insufficient lighting affecting contrast detection.

Company involved

HP

AI system involved

HP Webcam facial tracking software

10 source articles · read the reporting →

Nvidia releases AI-powered Eye Contact feature for Broadcast app

Nvidia has released a beta version of Eye Contact, an AI-powered feature for its Broadcast app that replaces a user's eyes in video streams with simulated eyeballs that always face the camera. The feature is intended to maintain eye contact even when the user looks away, but social media response has been largely negative. A former TV news anchor said that unbroken eye contact can be regarded as creepy or uncomfortable.

Company involved

Nvidia

AI system involved

Eye Contact

8 source articles · read the reporting →

Southern Co-op uses facial recognition with Hikvision cameras

Southern Co-op, a UK grocery chain, deployed a facial recognition system using Hikvision cameras in its stores. The system is used to identify and track customers, but the grocer did not inform customers or obtain their consent. Privacy advocates have raised concerns about compliance with GDPR and ethical risks. The grocer has stated it is committed to a high standard of privacy.

Company involved

Southern Co-op

4 source articles · read the reporting →

IARPA Janus Benchmark C dataset used faces from YouTube without consent

The IARPA Janus Benchmark C (IJB-C) dataset, published in 2017, contains 21,294 images and names of 3,531 individuals scraped from YouTube, Flickr, and Wikimedia without their consent. The dataset was created for face recognition benchmarking to improve intelligence analysis. Jillian York, a digital rights activist, had 41 frames of her face taken from a YouTube video without her knowledge or permission. The dataset violates YouTube's terms of service and raises privacy concerns.

Company involved

IARPA (Intelligence Advanced Research Projects Activity)

AI system involved

IARPA Janus Benchmark C (IJB-C)

6 source articles · read the reporting →

Clearview AI tested facial recognition surveillance cameras with UFT and Rudin

Clearview AI, the facial recognition company that scraped billions of photos from social media, developed a surveillance camera system under the name Insight Camera. The system was tested by the United Federation of Teachers and Rudin Management in New York City. The UFT used it to identify individuals who had made threats and prevent them from entering its offices. Clearview did not respond to requests for comment.

Company involved

Clearview AI

AI system involved

Insight Camera

9 source articles · read the reporting →

Chinese tech companies patented Uyghur detection analytics

IPVM reported that Huawei, Megvii, SenseTime, and other Chinese tech companies filed patents in China for AI systems that can detect Uyghur ethnicity. The patents included Uyghur as a detectable attribute for facial recognition and image retrieval. The companies responded by saying they would amend or withdraw the patents, claiming the references were misunderstood or regrettable, while the technology is used by police for surveillance and targeting of Uyghurs.

10 source articles · read the reporting →

Teleperformance plans AI webcam surveillance for home-working staff

Teleperformance, a global call centre company, told some staff it would install AI-powered webcams to monitor home-working infractions such as eating, phone use, or leaving desks. The system would randomly scan for breaches and send alerts to managers. After the Guardian inquired, the company said the remote scans would not be used in the UK, but the plan raised concerns from unions and MPs about invasive surveillance.

Company involved

Teleperformance

10 source articles · read the reporting →

AnyVision patents drone facial recognition technology

AnyVision, an Israeli private surveillance company, filed a US patent for drone technology that uses facial recognition to identify people on the ground. The patent describes adaptive positioning of drones to find optimal angles for accurate facial recognition. The technology is not yet in production, but the company's CEO stated it will become a reality soon. Potential privacy concerns have been raised.

Company involved

AnyVision

AI system involved

Adaptive positioning of drones for enhanced facial recognition

9 source articles · read the reporting →

Kohler, BMW, MaxMara secretly collected customers' facial recognition data

During the 2021 CCTV 3·15 Gala, it was revealed that Kohler, BMW, and MaxMara stores had installed facial recognition cameras from vendors such as Wandianzhang and Youluoke. These cameras captured customers' facial data without their knowledge or consent, and the data was used to track customer visits and inform sales strategies. The systems were deployed in thousands of stores across China, collecting over 100 million facial records. The companies did not inform customers or obtain consent, violating Chinese privacy laws.

Company involved

Kohler (China) Investment Co., Ltd., ZhengTong Auto (BMW dealerships), MaxMara

AI system involved

Facial recognition cameras from Wandianzhang, Youluoke, Yaliang, Ruiwei

10 source articles · read the reporting →

Huawei and Megvii tested Uyghur alarm facial recognition system

Huawei and Megvii conducted an interoperability test in 2018 that verified a 'Uyghur alarm' function in Megvii's facial recognition system, according to a confidential report found by IPVM. The system could identify Uyghur minorities and alert police. Huawei denied real-world application and deleted the report after IPVM inquired. Megvii denied targeting ethnic groups. The test report was publicly accessible on Huawei's European website.

Company involved

Huawei

AI system involved

Megvii Dynamic Face Recognition

10 source articles · read the reporting →

ViaQuatro's facial recognition system in São Paulo metro challenged in court

In April 2018, ViaQuatro installed the Digital Interactive Doors System, developed by AdMobilize, on the São Paulo metro's yellow line. The system used cameras to detect passengers' faces and claimed to infer their emotion, age, and gender in order to target advertisements. The Brazilian Institute of Consumer Protection (IDEC) filed a public civil action alleging that the system violated consumer and data protection laws by processing biometric data without consent and making pseudoscientific and discriminatory inferences. A judge ordered the cameras removed in August 2018, and the case is pending a final ruling.

Company involved

ViaQuatro

AI system involved

Digital Interactive Doors System (DID system)

10 source articles · read the reporting →

Adobe uses Creative Cloud user data to train AI

Adobe's content analysis feature may scan Creative Cloud and Document Cloud files to train machine learning models, including object recognition in Lightroom and Liquid Mode in Acrobat. The company says it may analyze images, audio, video, text and other documents stored on its servers. Adobe offers an opt-out in account privacy settings, but the article notes the company did not ask first.

Company involved

Adobe

AI system involved

Content analysis

10 source articles · read the reporting →

Canadian Tire chided by BC privacy commissioner for facial recognition without consent

Canadian Tire deployed facial recognition in at least four British Columbia stores between 2018 and 2021 to curb theft and increase staff safety. The BC Privacy Commissioner investigated and found the retailer failed to show a reasonable purpose and did not obtain customer consent, violating the province's Personal Information Protection Act. After the investigation was announced, Canadian Tire discontinued use of the system at all 12 locations and deleted customer data. The commissioner recommended stronger privacy policies and government regulation of biometric technologies.

Company involved

Canadian Tire

10 source articles · read the reporting →

Clearview AI to pull out of Canada and stop working with RCMP amid privacy investigation

Clearview AI, the US facial recognition company, has agreed to stop offering its services in Canada amid a joint investigation by Canadian privacy regulators. The company's technology, which matches images against billions of photos scraped from the internet, was used by the RCMP and more than 20 other police services across Canada. The Office of the Privacy Commissioner of Canada announced the 'indefinite suspension' of Clearview's contract with the RCMP, its last remaining Canadian client. The investigation into whether the technology breaches Canadian privacy laws will continue.

Company involved

Clearview AI

AI system involved

Clearview AI

10 source articles · read the reporting →

Canon uses AI smile recognition cameras to control office access

Canon Information Technology deployed AI cameras at its Beijing headquarters that only allow smiling employees to enter the office or book conference rooms. The system uses smile recognition to enforce a positive atmosphere. Some workers have complained about the intrusiveness of the technology. Canon defended the system, saying it encourages a positive work environment.

Company involved

Canon Information Technology (Canon China)

9 source articles · read the reporting →

CBSA tested facial recognition on millions at Toronto Pearson Airport in 2016

In 2016, the Canada Border Services Agency (CBSA) secretly tested facial recognition technology on millions of travellers at Toronto Pearson Airport without their knowledge. The system, supplied by Face4 Systems, matched faces against a database of previously deported individuals. The CBSA stated that no one was deported as a result of the pilot, and the technology was removed after six months. Privacy advocates raised concerns about lack of consent, transparency, and potential racial bias.

Company involved

Canada Border Services Agency

AI system involved

Faces on the Move

8 source articles · read the reporting →

Serbia's Ministry of Interior deploys thousands of Huawei facial recognition cameras in Belgrade

In early 2019, Serbia's Ministry of Interior announced the deployment of a thousand smart surveillance cameras with facial and license plate recognition capabilities, supplied by Huawei, across Belgrade. The Ministry refused to release information about the project, citing confidentiality, and failed to conduct a legally required Data Protection Impact Assessment. Citizens responded by crowdsourcing camera locations via the hashtag #hiljadekamera and a website. The system remains in operation with no public debate or transparency.

Company involved

Ministry of Interior

AI system involved

Huawei smart surveillance cameras

10 source articles · read the reporting →

FTC settles with IntelliVision over deceptive facial recognition claims

The Federal Trade Commission (FTC) took action against IntelliVision Technologies Corp. for making false, misleading, or unsubstantiated claims about its AI-powered facial recognition software. The company allegedly claimed its software had one of the highest accuracy rates on the market and was free of gender or racial bias, without supporting evidence. The FTC also alleged that IntelliVision did not train its software on millions of faces as claimed, but on images of approximately 100,000 individuals. Under a proposed consent order, IntelliVision is prohibited from making such misrepresentations without competent and reliable testing.

Company involved

IntelliVision Technologies Corp.

AI system involved

IntelliVision facial recognition software

9 source articles · read the reporting →

Met Police buys £3m retrospective facial recognition system

The Metropolitan Police Service (MPS) has awarded a £3 million, four-year contract to Northgate Public Services for a new retrospective facial-recognition (RFR) system to be deployed within three months. RFR processes biometric information from historic CCTV, social media, and other images to identify suspects and missing persons, operating retroactively unlike live facial recognition. The procurement was approved by the Mayor's Office for Policing and Crime in August 2021. The MPS states that human-in-the-loop decision-making will be used, but digital rights groups and a former biometrics commissioner have raised concerns about potential discrimination, overrepresentation of marginalised groups in watch lists, and lack of a legislative framework.

Company involved

Metropolitan Police Service

AI system involved

Retrospective facial-recognition software

9 source articles · read the reporting →