hesela.devLSM compaction
LSM compaction reorganizes sorted storage components, commonly merging selected runs or SST files and removing obsolete entries when visibility rules permit.
Compaction is not compression: it changes the organization and retained versions of data, whereas compression changes their representation. A snapshot can keep an older version necessary. Background merges consume CPU and I/O even when foreground queries perform no writes. [1] Some nonoverlapping files can move between levels without rewriting their contents, so compaction need not rewrite every selected byte. [2] A strategy specifies trigger, layout, granularity and data-movement policy, not just leveled versus tiered organization. [3]
Relations
- related
- lsm-tree, sstable, leveled-compaction, tiered-compaction, database-write-amplification
Citations
- RocksDB project. Overview: multi-threaded compaction and snapshot visibility.
- RocksDB project. Compaction: leveled, tiered and hybrid policies.
- Sarkar, Staratzis, Zhu and Athanassoulis. Constructing and Analyzing the LSM Compaction Design Space. PVLDB 14(11), 2216-2229 (2021), DOI: 10.14778/3476249.3476274; updated arXiv:2202.04522v2 (2022), sections 2-5.
- Hesela. Bloom filters: optimize the lookup, not each filter; compaction and measurement boundaries.