package reed-solomon-erasure

Bookkeeper for shard by shard encoding.

This is useful for avoiding incorrect use of encode_single_typ, and encode_single_sep_typ functions.

Naming conventions

typ represents bytes, str, and bigstr, e.g. encode_typ represents encode_bytes, encode_str, and encode_bigstr.

Use cases

Shard by shard encoding is useful for streamed data encoding where you do not have all the needed data shards immediately, but you want to spread out the encoding workload rather than doing the encoding after everything is ready.

A concrete example would be network packets encoding, where encoding packet by packet as you receive them may be more efficient than waiting for N packets then encode them all at once.

Example

open Reed_solomon_erasure

let () =
  let r = ReedSolomon.make 3 2 in

  let sbs = ShardByShard.make r in

  let shards = [|Bytes.of_string "\000\001\002\003\004";
                 Bytes.of_string "\005\006\007\008\009";
                 (* say we don't have the 3rd data shard yet
                    and we want to fill it in later *)
                 Bytes.of_string "\000\000\000\000\000";
                 Bytes.of_string "\000\000\000\000\000";
                 Bytes.of_string "\000\000\000\000\000"|] in

  (* encode 1st and 2nd data shard *)
  ShardByShard.encode_bytes sbs shards;
  ShardByShard.encode_bytes sbs shards;

  (* fill in 3rd data shard *)
  Bytes.set shards.(2) 0 '\010';
  Bytes.set shards.(2) 1 '\011';
  Bytes.set shards.(2) 2 '\012';
  Bytes.set shards.(2) 3 '\013';
  Bytes.set shards.(2) 4 '\014';

  (* now do the encoding*)
  ShardByShard.encode_bytes sbs shards;

  (* above is equivalent to doing ReedSolomon.encode_bytes shards *)

  assert (ReedSolomon.verify_bytes r shards)

Ownership of shards

The shard by shard codec is just a wrapper around the core codec type reed_solomon, so all potential shard ownership issues in functions provided by ReedSolomon are carried over.

See ReedSolomon, Ownership of shards section for details.