rocksdb.md

Data storage in RocksDB

Blockbook stores data the key-value store RocksDB. As there are multiple indexes, Blockbook uses RocksDB column families feature to store indexes separately.

The database structure is described in golang pseudo types in the form (name type).

Operators used in the description:

• -> mapping from key to value.
• + concatenation,
• [] array

Types used in the description:

• []byte - variable length array of bytes
• [32]byte - fixed length array of bytes (32 bytes long in this case)
• uint32 - unsigned integer, stored as array of 4 bytes in big endian*
• vint, vuint - variable length signed/unsigned int
• addrDesc - address descriptor, abstraction of an address. For Bitcoin type coins it is the transaction output script, stored as variable length array of bytes. For Ethereum type coins it is fixed size array of 20 bytes.
• bigInt - unsigned big integer, stored as length of the array (1 byte) followed by array of bytes of big int, i.e. (int_len byte)+(int_value []byte). Zero is stored as one byte of value 0.
• float32 - float32 number stored as uint32
• string - string stored as (len vuint)+(value []byte)

Database structure:

The database structure described here is of Blockbook version 0.5.0 (internal data format version 7).

The database structure for Bitcoin type and Ethereum type coins is different. Column families used for both types:

• default, height, addresses, transactions, blockTxs, fiatRates

Column families used only by Bitcoin type coins:

• addressBalance, txAddresses

Column families used only by Ethereum type coins:

• addressContracts, internalData, contracts, functionSignatures, blockInternalDataErrors, addressAliases

Column families description:

• default

  Stores internal state in json format, under the key internalState.

  Most important internal state values are:

  • coin - which coin is indexed in DB
  • data format version - currently 6
  • dbState - closed, open, inconsistent

  Blockbook is checking on startup these values and does not allow to run against wrong coin, data format version and in inconsistent state. The database must be recreated if the internal state does not match.

• height

  Maps block height to block hash and additional data about block.

  (height uint32) -> (hash [32]byte)+(time uint32)+(nr_txs vuint)+(size vuint)
  

• addresses

  Maps addrDesc+block height to array of transactions with array of input/output indexes.

  The block height in the key is stored as bitwise complement ^ of the height to sort the keys in the order from newest to oldest.

  As there can be multiple inputs/outputs for the same address in one transaction, each txid is followed by variable length array of input/output indexes. The index values in the array are multiplied by two, the last element of the array has the lowest bit set to 1. Input or output is distinguished by the sign of the index, output is positive, input is negative (by operation bitwise complement ^ performed on the number).

  (addrDesc []byte)+(^height uint32) -> []((txid [32]byte)+[](index vint))
  

• addressBalance (used only by Bitcoin type coins)

  Maps addrDesc to number of transactions, sent amount, total balance and a list of unspent transactions outputs (UTXOs), ordered from oldest to newest

  (addrDesc []byte) -> (nr_txs vuint)+(sent_amount bigInt)+(balance bigInt)+
                       []((txid [32]byte)+(vout vuint)+(block_height vuint)+(amount bigInt))
  

• txAddresses (used only by Bitcoin type coins)

  Maps txid to block height and array of input addrDesc with amounts and array of output addrDesc with amounts, with flag if output is spent. In case of spent output, addrDesc_len is negative (negative sign is achieved by bitwise complement ^).

  (txid []byte) -> (height vuint)+
                   (nr_inputs vuint)+[]((addrDesc_len vuint)+(addrDesc []byte)+(amount bigInt))+
                   (nr_outputs vuint)+[]((addrDesc_len vint)+(addrDesc []byte)+(amount bigInt))
  

• addressContracts (used only by Ethereum type coins)

  Maps addrDesc to total number of transactions, number of non contract transactions, number of internal transactions and array of contracts with number of transfers of given address.

  (addrDesc []byte) -> (total_txs vuint)+(non-contract_txs vuint)+(internal_txs vuint)+(contracts vuint)+
                       []((contractAddrDesc []byte)+(type+4*nr_transfers vuint))+
                       <(value bigInt) if ERC20> or
                         <(nr_values vuint)+[](id bigInt) if ERC721> or
                         <(nr_values vuint)+[]((id bigInt)+(value bigInt)) if ERC1155>
  

  This is a dense counted-entry record:

  • a small fixed prefix with address-level counters
  • a counted list of per-contract entries
  • each contract entry stores a compact standard-discriminated payload

  It is optimized for compactness and hot-path account reads, not for extensibility.

  • Contract ordering & hotness lookup

  Contract entries are appended in discovery order (they are not sorted). Lookups are normally a linear scan, but for mid-size lists we lazily build an in-memory index map when an address becomes "hot" (frequently looked up within the current block). A size-limited LRU keeps hot addresses; once the cache is full, the least-recently used hot address is evicted and will fall back to linear scans until it becomes hot again.

  • Large addressContracts cache

  To reduce repeated RocksDB reads/writes for very large entries, Blockbook caches addressContracts blobs whose packed size exceeds address_contracts_cache_min_size. The cache is flushed periodically, and also flushed early when its total size crosses the active cache cap. Chain-tip sync uses address_contracts_cache_max_bytes; bulk connect uses address_contracts_cache_bulk_max_bytes. Early flush avoids unbounded memory growth at the cost of more frequent writes.

• internalData (used only by Ethereum type coins)

  Maps txid to type (CALL 0 | CREATE 1), addrDesc of created contract for CREATE type, array of type (CALL 0 | CREATE 1 | SELFDESTRUCT 2), from addrDesc, to addrDesc, value bigInt and possible error.

  (txid []byte) -> (type+2*nr_transfers vuint)+<(addrDesc []byte) if CREATE>+
                   []((type byte)+(fromAddrDesc []byte)+(toAddrDesc []byte)+(value bigInt))+
                   (error []byte)
  

• blockTxs

  Maps block height to data necessary for blockchain rollback. Only last 300 (by default) blocks are kept. The content of value data differs for Bitcoin and Ethereum types.

  • Bitcoin type

  The value is an array of txids and input points in the block.

  (height uint32) -> []((txid [32]byte)+(nr_inputs vuint)+[]((txid [32]byte)+(index vint)))
  

  • Ethereum type

  The value is an array of transaction data. For each transaction is stored txid, from and to address descriptors and array of contract transfer infos consisting of from, to and contract address descriptors, type (ERC20 0 | ERC721 1 | ERC1155 2) and value (or list of id+value for ERC1155)

  (height uint32) -> [](
                        (txid [32]byte)+(from addrDesc)+(to addrDesc)+(nr_contracts vuint)+
                        []((from addrDesc)+(to addrDesc)+(contract addrDesc)+(type byte)+
                        <(value bigInt) if ERC20 or ERC721> or
                          <(nr_values vuint)+[]((id bigInt)+(value bigInt)) if ERC1155>)
                       )
  

• transactions

  Transaction cache, txdata is generated by coin specific parser function PackTx.

  (txid []byte) -> (txdata []byte)
  

• fiatRates

  Stored daily fiat rates, one day as one entry.

  (timestamp YYYYMMDDhhmmss) -> (nr_currencies vuint)+[]((currency string)+(rate float32))+
                                (nr_tokens vuint)+[]((tokenContract string)+(tokenRate float32))
  

• contracts (used only by Ethereum type coins)

  Maps contract addrDesc to indexed contract metadata. Sync owns this column family; API code does not write here. Protocol-specific detection records (e.g. ERC-4626 vault status) live in ercProtocols so API-time writes cannot collide with sync's whole-row writes.

  (addrDesc []byte) -> (name string)+(symbol string)+(type string)+(decimals vuint)+
                       (createdInBlock vuint)+(destroyedInBlock vuint)
  

• ercProtocols (used only by EVM coins)

  Per-protocol detection records keyed by contract address. Decoupled from contracts so API-driven protocol writes never clobber sync-driven contract metadata, and so disconnect can revert protocol records independently.

  Two prefixes share the column family:

  (0x00 || protocolId byte || addrDesc []byte) -> (persistHeight vuint)+(payload []byte)
  (0x01 || protocolId byte || persistHeight uint32 || addrDesc []byte) -> ()
  

  • byContract (prefix 0x00) is the read path: one row per (contract, protocolId), value carries the persist-height and the protocol-specific payload.
  • byHeight (prefix 0x01) is the secondary index used by DisconnectBlockRangeEthereumType: a small range scan over the disconnected height range yields exactly the rows whose persistence is no longer canonical, and both rows are deleted in the same batch as the rest of the disconnect.

  Reserved protocol IDs:

  • protocolId = 1 (erc4626): payload is the ERC-4626 vault's underlying asset address.

    erc4626 payload := (underlyingAssetContract string)
    

    Presence of the row implies the contract was observed as a vault at persistHeight; absence means either not-a-vault or never observed. The asset address is captured by the contractInfo API path on a successful Multicall3 probe of asset() and totalAssets(); indexing itself does not mark vaults.

  Future protocol IDs append the next free byte. 0x00 is reserved.

• functionSignatures (used only by Ethereum type coins)

  Database of four byte signatures downloaded from https://www.4byte.directory/.

  (fourBytes uint32)+(id uint32) -> (signatureName string)+[]((parameter string))
  

• blockInternalDataErrors (used only by Ethereum type coins)

  Errors when fetching internal data from backend. Stored so that the action can be retried.

  (blockHeight uint32) -> (blockHash [32]byte)+(retryCount byte)+(errorMessage []byte)
  

• addressAliases (used only by Ethereum type coins)

  Maps address to address ENS name.

  (address []byte) -> (ensName []byte)
  

Note: The txid field as specified in this documentation is a byte array of fixed size with length 32 bytes ([32]byte), however some coins may define other fixed size lengths.