iDedup
iDedup is a research inline deduplication design that trades some capacity savings for longer duplicate-block sequences and memory-resident fingerprint lookup.
The FAST 2012 design matches runs consecutive in both file-block order and existing disk-block locations. A tunable sequence-length threshold limits fragmentation. Its LRU fingerprint cache can miss duplicates and competes with the file buffer cache. Deduplication runs during NVRAM destaging, before disk allocation. [1]
Memory-resident lookup does not make all metadata volatile. Shared blocks retain persistent reference counts, whose updates can require disk I/O. A fingerprint-cache miss is not proof that no duplicate exists. [1]
The evaluation replayed two enterprise CIFS traces on three spinning disks. The headline 60-70% concerns achievable deduplication relative to a maximum, not a 60-70% reduction of all input bytes. The reported 2-4% mean-response-time differences occur at Threshold-8 with a 1 GB metadata cache, not as a universal bound or p99 guarantee. [1]
A Hesela illustration reads the same twelve block addresses in three orders: one run, three runs or twelve runs. Unique-block count stays twelve. Runs describe address adjacency, not measured seeks or elapsed time; caching, scheduling and device geometry are excluded. This is not an iDedup implementation or paper reproduction. [3]
Relations
Citations
- Srinivasan et al. iDedup: Latency-aware, Inline Data Deduplication for Primary Storage. FAST 2012, updated 10 February.
- Hesela: reproducible synthetic block-layout examples (JSON).
Further reading
- USENIX FAST 2012 publication record and updated-paper link.
- Hesela: block-layout illustration source (JavaScript).
- Hesela: Deduplication ratio does not predict restore cost.
Sources & caveats
- Paper sections 3.2-3.4, 4.1.4, 4.2.3 and 5.2-5.7. Historical research, not a claim about present NetApp products. The full updated paper and evaluation figures were inspected; no native benchmark or trace replay was performed.
- The synthetic illustration uses an explicit adjacency rule and no fitted parameters. Execute the linked JavaScript with Node.js to print its JSON; it does not infer disk latency.