deduplication · shipping · also dedup

Data deduplication

Data deduplication replaces duplicate content with references to shared stored blocks or chunks within a defined matching scope.

Sharing content does not determine physical replica count or guarantee fast restores. Restore locality and cache behavior must be assessed separately from capacity savings. Matching scope, index cost and handling of existing data are implementation-specific; these boundaries depend on the filesystem. [3] [4] Chunk boundaries are another independent choice. Copying snapshots between repositories with different chunker parameters can reduce reuse; the documented setup copies parameters when initializing a destination. [5] Confidentiality also depends on the matching layer: a trusted client's plaintext index is not cross-user ciphertext matching. [1]

Candidate-limited lookup can retain duplicates that a complete index would find; sparse indexing makes this trade-off explicit. [7]

Deduplicating sampled positions need not recover the full population's ratio; a duplication-frequency histogram records the observed repetition distribution. [8]

Deleting a backup need not release its logical size. Chunks may still be shared, and unreferenced payload can remain inside live containers. Snapshot removal with forget and collection with prune are separate operations. [9] A last-reference marker alone also cannot resolve every selective-retention policy; the linked derivation states the exact chronological-suffix condition. [10]

Collection policy can leave unused payload after a successful run. Borg 1.4.1 documents a default 10% segment-compaction threshold; compact --dry-run does nothing rather than forecasting reclaimable space. [11]

Chunk-audit coverage does not determine incomplete-file impact. [12]

Relations

related
Resemblance detection, Delta compression, HyperLogLog, Thin provisioning, Object storage, All-flash array, Chunk fragmentation, Container capping, Forward assembly area, Content-defined chunking, Rolling hash, Cryptographic hash function, Hash collision, Convergent encryption, Deduplication side channel, Proof of ownership (deduplication), Sparse indexing (deduplication), FIDEDUPERANGE, Duplication-frequency histogram, WARC revisit record, ZFS deduplication table, iDedup, Destor, RevDedup, DedupSearch, IDEA index, Deduplication ratio, Content-addressed storage, Birthday bound, MinHash, Jaccard similarity, GogetaFS

Citations

  1. Hesela: Encrypted deduplication: who can test a file guess? (accessed 2026-10-18)
  2. OpenZFS documentation: Deduplication (accessed 2026-10-11)
  3. Deduplication ratio does not predict restore cost (accessed 2026-10-11)
  4. Restic 0.19.1 documentation: Ensuring deduplication for copied snapshots. (accessed 2026-10-12)
  5. Lillibridge et al. Sparse Indexing: Large Scale, Inline Deduplication Using Sampling and Locality. USENIX FAST 2009, pp. 111-123. (accessed 2026-10-19)
  6. Harnik, Khaitzin and Sotnikov. Estimating Unseen Deduplication. FAST 2016, sections 2.1 and 3.3. (accessed 2026-11-11)
  7. Restic v0.19.1: snapshot removal and pruning (accessed 2026-10-23)
  8. Hesela: backup deletion and reclaimable containers (accessed 2026-10-23)
  9. Borg 1.4.1: compact threshold and dry-run behavior. (accessed 2026-10-23)
  10. Hesela: exact synthetic chunk-audit and restore-impact examples, code, native probe and source provenance. (accessed 2026-11-22)
  11. Bhagwat et al. Providing High Reliability in a Minimum Redundancy Archival Storage System. MASCOTS 2006, sections 4-6. (accessed 2026-11-22)

Further reading

  1. Data deduplication (accessed 2026-03-26)
  2. Content-defined chunking: boundaries are not identity (accessed 2026-10-12)
Sources & caveats
  • The snapshot-removal and pruning distinction refers to restic v0.19.1 at the pinned documentation commit. [9]
  • The audit examples use synthetic equal-size chunks, not fleet loss rates. Historical replication work weighted chunks by dependent files or bytes. [13]

JSON: /data/terms/deduplication.json

Published by Hesela team · Updated

Report a correction · Editorial standards · Reuse guidance

Cite this page
Hesela team. Data deduplication. Hesela. 2026-11-22. https://hesela.dev/terms/deduplication/