idedup · research · also latency-aware inline deduplication

iDedup

iDedup is a research inline deduplication design that trades some capacity savings for longer duplicate-block sequences and memory-resident fingerprint lookup.

The FAST 2012 design matches runs consecutive in both file-block order and existing disk-block locations. A tunable sequence-length threshold limits fragmentation. Its LRU fingerprint cache can miss duplicates and competes with the file buffer cache. Deduplication runs during NVRAM destaging, before disk allocation. [1]

Memory-resident lookup does not make all metadata volatile. Shared blocks retain persistent reference counts, whose updates can require disk I/O. A fingerprint-cache miss is not proof that no duplicate exists. [1]

The evaluation replayed two enterprise CIFS traces on three spinning disks. The headline 60-70% concerns achievable deduplication relative to a maximum, not a 60-70% reduction of all input bytes. The reported 2-4% mean-response-time differences occur at Threshold-8 with a 1 GB metadata cache, not as a universal bound or p99 guarantee. [1]

A Hesela illustration reads the same twelve block addresses in three orders: one run, three runs or twelve runs. Unique-block count stays twelve. Runs describe address adjacency, not measured seeks or elapsed time; caching, scheduling and device geometry are excluded. This is not an iDedup implementation or paper reproduction. [3]

Relations

related
Data deduplication, Chunk fragmentation, Sparse indexing (deduplication), GogetaFS

Citations

  1. Srinivasan et al. iDedup: Latency-aware, Inline Data Deduplication for Primary Storage. FAST 2012, updated 10 February. (accessed 2026-11-17)
  2. Hesela: reproducible synthetic block-layout examples (JSON). (accessed 2026-11-17)

Further reading

  1. USENIX FAST 2012 publication record and updated-paper link. (accessed 2026-11-17)
  2. Hesela: block-layout illustration source (JavaScript). (accessed 2026-11-17)
  3. Hesela: Deduplication ratio does not predict restore cost. (accessed 2026-11-17)
Sources & caveats
  • Paper sections 3.2-3.4, 4.1.4, 4.2.3 and 5.2-5.7. Historical research, not a claim about present NetApp products. The full updated paper and evaluation figures were inspected; no native benchmark or trace replay was performed.
  • The synthetic illustration uses an explicit adjacency rule and no fitted parameters. Execute the linked JavaScript with Node.js to print its JSON; it does not infer disk latency.

JSON: /data/terms/idedup.json

Published by Hesela team · Updated

Report a correction · Editorial standards · Reuse guidance

Cite this page
Hesela team. iDedup. Hesela. 2026-11-21. https://hesela.dev/terms/idedup/