hesela.devHitchhiker
Hitchhiker is an erasure-coded storage design that couples Reed-Solomon instances to reduce single-data-unit repair traffic without adding parity units.
Piggybacking adds functions of earlier code instances to later ones. Decoding the earlier instance allows those additions to be removed; the underlying recovery property is retained. Useful repair savings depend on the chosen functions, not merely on coupling instances. [2]
Hitchhiker distinguishes XOR, XOR+ and nonXOR variants. XOR+ requires an all-XOR parity in its base code. Parity repair is not the optimized data-repair case; hop-and-couple addresses scattered substripe reads. [1]
The versioned decoder selects its special path only for one erased data unit; parity or multiple erasures take another path. That implementation does not establish a default HDFS deployment: the 3.4.3 policy guide lists RS and XOR codecs. [3] [4]
In the fixed 10+4 accounting example, XOR+ reads 6.5 block-equivalents per missing data block and 10 for parity. Uniformly weighting ten data roles and four parity roles gives 7.5, a 25% reduction from the 10-block baseline, not the data-only 35%. These are independent repair counts, not a full-stripe outage or measured speedup. [5]
Hop-length controls a locality trade-off: half a block makes each substripe contiguous, but a short degraded read may also reconstruct its unrequested coupled bytes. Smaller hops reduce that separation at the cost of less contiguous substripe reads. Whole-block accounting does not establish small-read cost. [1]
Relations
- related
- Erasure coding, Reed–Solomon codes (storage), Repair-read amplification, Repair locality, Subpacketization
Advantages
- Retains the base code's recovery property while allowing a different repair-read pattern. [2]
Limitations
Citations
- Rashmi et al. A Hitchhiker's Guide to Fast and Efficient Data Reconstruction in Erasure-coded Data Centers. SIGCOMM 2014, sections 3-7 and Appendix.
- Rashmi, Shah and Ramchandran. A Piggybacking Design Framework for Read-and Download-efficient Distributed Storage Codes, section II.
- Apache Hadoop 3.4.3: HHXORErasureDecodingStep, single-data versus multiple/parity dispatch.
- Apache Hadoop 3.4.3: HDFS Erasure Coding, supplied policies and supported codecs.
- Hesela: Hitchhiker read plans and role-weighted recovery accounting.
Sources & caveats
- Status denotes a versioned implementation component, not a default filesystem policy. The public accounting module does not execute an encoder, decoder or HDFS cluster.