Coinbase Replay Finds Fraud Coverage Fell After AI Upgrades

Coinbase reported Oct. 7 that newer versions of three AI model families detected fewer fraudulent payments and a smaller share of fraud value in a historical replay of its Onramp payment-screening system. The payment-decision policy remained unchanged.

The replay covered 16,140 transactions involving 7,293 users, including 813 confirmed fraudulent transactions. The data came from nine weeks before Coinbase deployed its risk agent. It included matured fraud cases and a sample of legitimate transactions.

The test compared Opus 4.5 with Opus 5, Sonnet 4.6 with Sonnet 5, and GPT-5.4 with GPT-5.6 (sol). Each newer model recorded lower recall, F1 and dollar-weighted recall. Recall measures the share of fraud cases detected. Dollar-weighted recall measures the share of total fraudulent value detected. F1 combines precision and recall.

Sonnet’s recall fell 22.2 percentage points, while its dollar-weighted recall declined 22.9 points. Opus recorded a 0.8-point drop in recall, and its precision also fell. Precision measures the share of transactions flagged as fraudulent that were actually fraudulent.

GPT’s precision rose 11.5 percentage points. Its recall fell 20.7 points, and its dollar-weighted recall declined 21.8 points. The model’s fraud alerts were more accurate, while it detected fewer fraud cases and less fraudulent value in the replay.

The models used the same instructions and the same policy for converting risk classifications into payment decisions. Coinbase described the test as an evaluation of model behavior within that setup, rather than a comparison of redesigned screening systems. The results did not establish whether the tested versions caused customer losses, and Coinbase could not determine the reason for the declines.

In a separate disclosure on Oct. 8, Coinbase reported results from a custom Qwen3.5-9B model post-trained with historical fraud outcomes and deterministic rewards. Compared with Opus 4.5, the model’s F1 score improved by 9.6 percentage points and its dollar-weighted recall rose by 35.4 points.

A separate production measurement recorded median end-to-end request latency of 0.683 seconds for the custom model, compared with 1.515 seconds for Opus 4.5. Coinbase calculated the difference as a 55% relative reduction. The latency and fraud-detection results came from separate evaluations.

An earlier Coinbase online experiment tested selective large-language-model reviews alongside existing models and rules. That agent-enabled process detected 30% fewer fraudulent transactions and 22% less fraud value. It did not compare newer versions of the three model families.

The evaluation used a proprietary dataset that cannot be released for independent replication, according to the researchers’ stated limitations.