Christopher Gatlin spent 17 months in a Missouri jail after a facial recognition search run on Amazon Rekognition flagged him as a candidate from a grainy, partially masked surveillance photo. Last week, an amended federal complaint added AWS as a defendant. The instinct in most coverage is to ask "was the algorithm wrong?" That's the wrong question. The more useful one is: where, specifically, did the system and the process around it fail, and was that failure predictable from the statistics involved? It was — and that's the part worth actually understanding.
Failure point 1 — No enforced quality gate at the input
AWS's own technical guidance says Rekognition's face-comparison feature isn't meant to be used on images that are blurry, distant, or have large portions of the face obscured. The photo in this case had all three problems: shot from a distance and above, a hood over part of the forehead, a mask over the lower face. The system still returned ranked candidates instead of rejecting the query outright. That's consistent with how similarity-matching systems are generally designed — they return the best available matches, not a refusal — but it means the quality gate that should stop a bad query from ever reaching an investigator doesn't exist as an enforced technical control. It exists only as a line in the documentation that the deploying agency has to know about and choose to follow.
Failure point 2 — No enforced confidence threshold
A face-comparison result includes a similarity score. Nothing in the pipeline as deployed appears to have required a minimum score before a candidate could be acted on. AWS's guidance says match results should be treated as one input alongside independent evidence — not the sole basis for action affecting someone's liberty. That's a policy statement, not a system constraint. There was no mechanism that would have technically prevented a low-confidence match from being handed to detectives and treated identically to a high-confidence one.
Failure point 3 — Circular verification
The photo lineup that "confirmed" Gatlin as a suspect was built because the algorithm flagged him. The witness — who had already told police his memory of the attack was poor — was shown a lineup containing the algorithmically-generated candidate and, after an officer prompted him to focus on specific features, selected Gatlin. That selection was then used as independent corroboration of the algorithm's output. It isn't independent. It's the same signal, laundered through a second step that looks like verification but isn't, because the second step's input set was constructed using the first step's output. This is the investigative equivalent of validating a model on its own training data — the result will look like confirmation no matter what, because the test wasn't capable of returning a different answer.
Failure point 4 — The base-rate problem, with real numbers behind it
Here's the statistical failure underneath all of it, and it's not specific to this one case — it's a property of searching for a rare event in a large population, regardless of how good the underlying algorithm is.
People hear "highly accurate" and assume that means "if it flags someone, they're probably guilty."
The correct question is Positive Predictive Value (PPV): of everyone the system flags, what fraction are actually the true match? That depends on the size of the search population as much as on the algorithm's accuracy.
Using Bayes' theorem, with illustrative figures chosen to show the mechanism (not AWS's undisclosed real performance numbers for the system used in 2021):
P(is suspect | flagged) = P(flagged | suspect) × P(suspect)
─────────────────────────────────
P(flagged)
Sensitivity = 99% → P(flagged | suspect) = 0.99
False positive rate = 0.1% → P(flagged | not suspect) = 0.001
Database size = 250,000 → P(suspect) = 1/250,000
P(flagged) = (0.99 × 1/250,000) + (0.001 × 249,999/250,000) ≈ 0.001004
P(is suspect | flagged) ≈ 0.00000396 / 0.001004 ≈ 0.4%Even a system with a 99.9%-reliable false-positive rate produces roughly 250 false alarms for every one true match when searched against a 250,000-person database (0.1% of 250,000). One real signal, competing against 250 false ones — a 0.4% chance any given flag is correct, no matter how "accurate" the headline number sounds. This is the same math behind why screening tests for rare diseases produce so many false positives; it's not a defect specific to Rekognition, it's what happens whenever you search for something rare across something large without narrowing the population first.
Failure point 5 — Demographic disparity isn't uniform, but it's also not a myth
NIST's federal testing (FRVT) found that many algorithms have substantially higher false-positive rates for Asian and African American faces compared to Caucasian faces — in some cases by a factor of 10 to 100. That's real and documented. The more complete picture: NIST also found that the most accurate algorithms show far smaller demographic gaps — sometimes near-undetectable. That means the disparity is not an unavoidable law of facial recognition; it's largely a function of which algorithm and what confidence threshold an agency chooses to deploy. That's an important, honest nuance — it turns "facial recognition is inherently racist" (an overstatement) into the sharper, more defensible claim: procurement and threshold choices determine whether demographic disparity shows up in practice, and unaudited deployments have no way of knowing which side of that line they're on.
Wayward Fowl's audit perspective
If Wayward Fowl were to audit this AI system, the audit would likely examine input-quality controls at the point of query, the presence and enforcement of a minimum confidence threshold before results reach an investigator, and the independence of the verification step used to corroborate any algorithmic candidate. Potential findings could include an absence of an enforced image-quality gate despite documented vendor guidance against low-quality inputs, no technical control preventing a below-threshold match from being treated the same as a high-confidence one, and a verification procedure — the lineup — that was constructed using the algorithm's own output rather than independent evidence, which undermines its value as a check on the system. We would recommend strengthening input and confidence-threshold controls at the technical layer, improving procurement governance by requiring NIST-tested algorithms with published, demographic-specific error rates for the exact configuration deployed, and implementing mandatory audit logging together with a hard separation between algorithmic output and the evidence used to verify it, to reduce the likelihood of similar failures.
The 0.4% PPV isn't a flaw to patch — it's math that will be true of any system used this way, on a population this large, for an event this rare. The actual fix isn't a better algorithm. It's a process that treats every match as a hypothesis to test, never as a conclusion to act on.