Persistent block store with linear history
The cemented block store is a store where blocks are stored linearly (by level) in chunks. Blocks in this store should not be reorganized anymore and are thus *cemented*. As these blocks should not be accessed regularly and especially their optionally stored metadata, the later are compressed using a zip format to save disk space. For each chunk of blocks, a dedicated file is used. Moreover, to enable easy access and to prevent too much on-disk reading, two indexed maps are used to retrieve blocks hash from their level and their level from the block hash.
The cemented block store contains a set of files updated each time a new chunk is added to the store. These files indicate which interval of blocks (w.r.t. their levels) are stored in it.
This store is expected to respect the following invariants:
i ; j (with i <= j) contains | j - i + 1 | blocks and are ordered from i to j in the file.∀f_x=(i,j) ∈ F ∧ x < |F|, ∃f_y =(i', j'), x = y - 1 ∧ j + 1 = j'
meaning the concatenation of every chunk must be continuous.
The cemented block store is composed of the following files:
| <n> × <offset> | <n> × <block> |
where n is (j - i + 1), <offset> is a 4 bytes integer representing the absolute offset of a block where the k-th (with 0 <= k < n) offset stands for the absolute offset of the k-th block in the file and with <block>, a Block_repr.t value encoded using Block_repr.encoding (thus prefixed by the its size).
module Cemented_block_level_index :
Index.S with type key = Block_key.t and type value = Block_level.tOn-disk index of block's hash to level.
module Cemented_block_hash_index :
Index.S with type key = Block_level.t and type value = Block_key.tOn-disk index of block's level to hash.
type cemented_blocks_file = {start_level : int32;end_level : int32;file : [ `Cemented_blocks_file ] Naming.file;}The type for cemented block chunks file description
val init :
?log_size:int ->
[< `Chain_dir ] Naming.directory ->
readonly:bool ->
(t, Tezos_error_monad.TzCore.error list) Stdlib.result Lwt.tinit ?log_size ~cemented_blocks_dir ~readonly creates or loads an existing cemented block store at path cemented_blocks_dir. cemented_blocks_dir will be created if it does not exists. If readonly is true, cementing blocks will result in an error. log_size determines the index cache size.
val close : t -> unitclose cemented_store closes the cemented_store opened files: its indexes.
val cemented_blocks_files : t -> cemented_blocks_file array optioncemented_blocks_files cemented_store returns the current list of cemented blocks chunks files.
val cemented_block_level_index : t -> Cemented_block_level_index.tcemented_block_level_index block_store returns the hash to level index.
val cemented_block_hash_index : t -> Cemented_block_hash_index.tcemented_block_hash_index block_store returns the level to hash index.
val load_table :
[ `Cemented_blocks_dir ] Naming.directory ->
(cemented_blocks_file array option, Tezos_error_monad.TzCore.error list)
Stdlib.result
Lwt.tload_table ~cemented_blocks_dir reads the cemented_blocks_dir directory and instantiate the cemented blocks chunks files.
val find_block_file : t -> int32 -> cemented_blocks_file optionfind_block_file cemented_store block_level lookups the cemented_store to find the cemented block chunk file that contains the block at level block_level. Returns None if the block cannot be found.
val is_cemented : t -> Tezos_crypto.Block_hash.t -> boolis_cemented cemented_store block_hash checks if the block_hash is stored in the cemented_store.
val get_cemented_block_level : t -> Tezos_crypto.Block_hash.t -> int32 optionget_cemented_block_level cemented_store block_hash returns the level of the block_hash if present in cemented_store. Returns None otherwise.
val get_cemented_block_hash : t -> int32 -> Tezos_crypto.Block_hash.t optionget_cemented_block_hash cemented_store block_level returns the hash of the block at block_level if present in cemented_store. Returns None otherwise.
val read_block_metadata :
t ->
int32 ->
(Block_repr.metadata option, Tezos_error_monad.TzCore.error list)
Stdlib.result
Lwt.tread_block_metadata cemented_store block_level returns the metadata of the block at block_level if present in cemented_store. Returns None otherwise.
val cement_blocks_metadata :
t ->
Block_repr.t list ->
(unit, Tezos_error_monad.TzCore.error list) Stdlib.result Lwt.tcement_blocks_metadata cemented_store chunk compresses and stores the metadata of blocks present in chunk. If no block of the given chunk contains metadata, nothing is done. Otherwise, for every block containing metadata, an entry is written in the dedicated .zip metadata file.
We assume that the blocks containing metadata are contiguous and if at least one block has metadata, then the blocks from that block with metadata to the last block of chunk must have metadata. However, we do not check the validity of this assumption.
val get_lowest_cemented_level : t -> int32 optionget_lowest_cemented_level cemented_store returns the lowest cemented block in cemented_store, if it exists.
val get_highest_cemented_level : t -> int32 optionget_highest_cemented_level cemented_store returns the highest cemented block in cemented_store if it exists.
val get_cemented_block_by_level :
t ->
read_metadata:bool ->
int32 ->
(Block_repr.block option, Tezos_error_monad.TzCore.error list) Stdlib.result
Lwt.tget_cemented_block_by_level cemented_store ~read_metadata level reads the cemented block at level in cemented_store, if it exists. It also retrieves the metadata depending on read_metadata.
val get_cemented_block_by_hash :
read_metadata:bool ->
t ->
Tezos_crypto.Block_hash.t ->
(Block_repr.block option, Tezos_error_monad.TzCore.error list) Stdlib.result
Lwt.tget_cemented_block_by_hash cemented_store hash reads the cemented block of hash in cemented_store, if it exists. It also retrieves the metadata depending on read_metadata.
val cement_blocks :
?check_consistency:bool ->
t ->
write_metadata:bool ->
Block_repr.t list ->
(unit, Tezos_error_monad.TzCore.error list) Stdlib.result Lwt.tcemented_blocks ?check_consistency cemented_store ~write_metadata
chunk stores the chunk of blocks and write their metadata if the flag write_metadata is set. check_consistency (default is true) ensures that the blocks in the given chunk are contiguous.
val trigger_gc : t -> Tezos_shell_services.History_mode.t -> unit Lwt.ttrigger_gc cemented_store history_mode garbage collects metadata chunks and/or chunks from the cemented_store depending on the History_mode.t:
Archive mode, nothing is done;Full offset mode, only offset chunks of metadata are kept;Rolling offset mode, only offset chunks of metadata and chunks are kept.Important: when purging chunks of blocks, it is necessary to rewrite the index to remove garbage collected blocks. Therefore, the higher the offset is, the longest the GC phase will last.
val iter_cemented_file :
(Block_repr.block -> unit Lwt.t) ->
cemented_blocks_file ->
(unit, Tezos_error_monad.TzCore.error list) Stdlib.result Lwt.titer_cemented_file ~cemented_block_dir f block_file reads from the cemented block_file located in cemented_block_dir and applies f on every block.
val check_indexes_consistency :
?post_step:(unit -> unit Lwt.t) ->
?genesis_hash:Tezos_crypto.Block_hash.t ->
t ->
(unit, Tezos_error_monad.TzCore.error list) Stdlib.result Lwt.tcheck_indexes_consistency ?post_step ?genesis_hash cemented_store
history_mode iterates over a partially initialized cemented_store that contains both chunks of blocks and indexes then check the consistency of each block: (hashes, predecessors and levels). The hash is not checked for genesis_hash and post_step is called after each treated chunk. This is used for snapshot imports.