hesela.devZFS deduplication table
A ZFS deduplication table maps block signatures and properties to stored block locations and reference counts so compatible writes can share existing data.
OpenZFS keeps pool-level tables by checksum type, keyed by checksum and block properties. This metadata is not a recovery copy. [4]
In the OpenZFS v2.3 pool properties, dedup_table_size counts on-disk table bytes, whereas dedupcached reports the table content resident in ARC. dedup_table_quota limits on-disk growth with approximate enforcement; it is not a RAM cap. Lowering it does not prune existing entries. [3]
In OpenZFS 2.3.4, over-quota lookups can update existing entries. New-entry writes fall back to non-deduplicated storage, subject to allocation and I/O success. [4]
Setting dedup=off changes subsequent writes, not previously shared blocks or their tracking. [5]
Pruning is a separate operation: ddtprune removes eligible older single-reference entries, not the corresponding payload blocks. [6] [7]
With fast_dedup, in-memory log entries have a separate zfs_dedup_log_mem_max budget, initialized from 1% of total RAM by default. This is not a bound on all deduplication memory. [1] [2]
In OpenZFS 2.3.4, the lookup key includes logical size, physical size, compression and the encryption-use flag alongside the checksum. Matching checksum bytes alone do not establish an interchangeable block representation. [4]
In v2.3, dedup=verify adds a byte-for-byte check with SHA-256. A read failure can also trigger fallback; ordinary storage after verification is not proof of a hash collision. This write-path check is not a pool-wide scrub. [8] [9]
In OpenZFS v2.3, Direct I/O writes bypass deduplication even with dedup=on. Requesting O_DIRECT does not prove that path: direct=disabled and unaligned write portions use buffered I/O. [10]
Relations
- related
- Data deduplication, ZFS, Checksum
Citations
- OpenZFS v2.3: zfs(4), dedup log memory, flushing and import-time trade-offs
- OpenZFS 2.3.4: dedup log memory initialization
- OpenZFS v2.3: zpoolprops(7), DDT size, cache residency and quota
- OpenZFS 2.3.4: deduplication table entries and quota fallback
- OpenZFS: Deduplication, changing the property and existing blocks
- OpenZFS v2.3: zpool-ddtprune(8), single-reference entry pruning
- OpenZFS 2.3.4: zio_ddt_free handles previously pruned entries
- OpenZFS v2.3: zfsprops(7), dedup verification and checksum selection
- OpenZFS 2.3.4: zio_ddt_collision and write fallback
- OpenZFS v2.3: Direct I/O write paths and deduplication
Sources & caveats
- Source review of OpenZFS v2.3 manuals and the 2.3.4 implementation at commit 34f96a15c73eab27dd6ad17bb5f1263bf26e37d7; not a claim about every ZFS version.
- No pool experiment, memory-sizing measurement or performance benchmark was performed.