hdf5-raw-data-chunk-cache · shipping · also RDCC, HDF5 chunk cache

HDF5 raw-data chunk cache

The HDF5 raw-data chunk cache retains decoded dataset chunks in memory subject to byte capacity, hash-slot collisions and eviction policy.

Separate limits: colliding chunks cannot coexist in the cache even when byte capacity remains. Cache bytes and slot count therefore constrain different resources. [1]

Measured example: four compressed chunks and 32 scalar selections required 32 raw chunk reads with one slot, but four with 17 slots and enough bytes. Grouping the same selections by chunk also needed four reads with one slot. These are in-memory file-driver counts, not disk traffic or a timing speedup. [3]

Scope: each dataset has its own cache. Effective settings and library version matter; the byte budget is not a cap on all application memory. [2]

Relations

related

hdf5, hdf5-chunked-layout, page-cache, cache-pollution

Citations

1. The HDF Group: Chunking in HDF5, chunk caching and filters (accessed 2026-10-17)
2. h5py 3.15: File objects, Python file-like objects and chunk cache (accessed 2026-10-17)
3. Hesela: HDF5 chunk cache, measured file-driver reads (accessed 2026-10-17)

JSON: /data/terms/hdf5-raw-data-chunk-cache.json