Bitcoin txindex can shrink by 40 GB after a rebuild

A redesign of Bitcoin Core’s optional -txindex was merged into the master branch via pull request #35531 on Aug. 15. In one contributor’s mainnet test the rebuilt txindex fell from about 66 GB to 26 GB, a reduction of roughly 61%. The change is upstream in master, but the first stable release that will ship the redesign has not been specified.

The -txindex option builds a database that lets a node find transactions by their full transaction ID. Under the previous format each full 32-byte transaction ID was stored as a database key along with the transaction’s disk-position data. The redesign writes a much shorter lookup key made up of a five-byte prefix derived from a salted SipHash and a six-byte suffix that encodes block sequence and transaction offset. The software still checks full transaction IDs after candidates are located, so short prefixes cannot produce false matches.

When a lookup uses the compact key, the node scans entries that share the same five-byte prefix, uses the block index to locate candidate blocks, reads candidate transactions from disk and compares their full IDs to confirm matches. The extra disk reads and ID checks handle any collisions created by the shorter prefix. In the contributor’s tests lookup latency stayed near 0.2 milliseconds per query.

The contributor’s mainnet rebuild completed in 1 hour 19 minutes with the compact format versus 1 hour 50 minutes with the prior format. Reported numbers apply only to the optional txindex; the full blockchain and the remainder of a node’s data directory were not included in the measurement. Hardware, storage type, chain height and software version can affect index size and rebuild time.

Upgrading a node to a release that includes the change does not force an immediate conversion of an existing txindex. Existing legacy entries remain readable after an upgrade, so operators must recreate the txindex to write entries in the compact format and realize the reduced disk footprint. If an operator downgrades to an older release after rebuilding the txindex, the older release cannot read entries written in the compact format; returning to an older version therefore requires another txindex rebuild in the legacy format.

Because the redesign is currently in master, operators should wait for the stable release that contains the change and follow the migration notes included with that release for exact upgrade and downgrade procedures.