Data deduplication
Data deduplication replaces duplicate content with references to shared stored blocks or chunks within a defined matching scope.
Sharing content does not determine physical replica count or guarantee fast restores. Restore locality and cache behavior must be assessed separately from capacity savings. Matching scope, index cost and handling of existing data are implementation-specific; these boundaries depend on the filesystem. [3] [4] Chunk boundaries are another independent choice. Copying snapshots between repositories with different chunker parameters can reduce reuse; the documented setup copies parameters when initializing a destination. [5] Confidentiality also depends on the matching layer: a trusted client's plaintext index is not cross-user ciphertext matching. [1]
Candidate-limited lookup can retain duplicates that a complete index would find; sparse indexing makes this trade-off explicit. [7]
Deduplicating sampled positions need not recover the full population's ratio; a duplication-frequency histogram records the observed repetition distribution. [8]
Deleting a backup need not release its logical size. Chunks may still be shared, and unreferenced payload can remain inside live containers. Snapshot removal with forget and collection with prune are separate operations. [9] A last-reference marker alone also cannot resolve every selective-retention policy; the linked derivation states the exact chronological-suffix condition. [10]
Collection policy can leave unused payload after a successful run. Borg 1.4.1 documents a default 10% segment-compaction threshold; compact --dry-run does nothing rather than forecasting reclaimable space. [11]
Chunk-audit coverage does not determine incomplete-file impact. [12]
Relations
- related
- Resemblance detection, Delta compression, HyperLogLog, Thin provisioning, Object storage, All-flash array, Chunk fragmentation, Container capping, Forward assembly area, Content-defined chunking, Rolling hash, Cryptographic hash function, Hash collision, Convergent encryption, Deduplication side channel, Proof of ownership (deduplication), Sparse indexing (deduplication), FIDEDUPERANGE, Duplication-frequency histogram, WARC revisit record, ZFS deduplication table, iDedup, Destor, RevDedup, DedupSearch, IDEA index, Deduplication ratio, Content-addressed storage, Birthday bound, MinHash, Jaccard similarity, GogetaFS
Citations
- Hesela: Encrypted deduplication: who can test a file guess?
- OpenZFS documentation: Deduplication
- Deduplication ratio does not predict restore cost
- Restic 0.19.1 documentation: Ensuring deduplication for copied snapshots.
- Lillibridge et al. Sparse Indexing: Large Scale, Inline Deduplication Using Sampling and Locality. USENIX FAST 2009, pp. 111-123.
- Harnik, Khaitzin and Sotnikov. Estimating Unseen Deduplication. FAST 2016, sections 2.1 and 3.3.
- Restic v0.19.1: snapshot removal and pruning
- Hesela: backup deletion and reclaimable containers
- Borg 1.4.1: compact threshold and dry-run behavior.
- Hesela: exact synthetic chunk-audit and restore-impact examples, code, native probe and source provenance.
- Bhagwat et al. Providing High Reliability in a Minimum Redundancy Archival Storage System. MASCOTS 2006, sections 4-6.