hdf5-chunked-layout · shipping · also HDF5 chunked storage, HDF5 dataset chunk

HDF5 chunked layout

An HDF5 chunked layout divides a dataset into fixed-shape array blocks that can be stored and filtered separately.

Geometry: chunk dimensions are chosen at dataset creation. Compression and other filters operate on chunks; selecting a few values can require processing a whole filtered chunk. A different chunk shape requires rewriting the dataset. [1]

Byte accounting: a 32 by 32 chunk of four-byte integers expands to 4,096 bytes, regardless of its smaller compressed size. In a measured scalar-read trace, a cache large enough for the compressed bytes but smaller than the decoded chunk still reread that chunk for each selection. [2]

Terminology: array chunk geometry is distinct from content-defined chunking, which places boundaries based on byte content for purposes such as deduplication. [1] [3]

Relations

related

hdf5, hdf5-raw-data-chunk-cache, content-defined-chunking

Citations

1. h5py 3.15: Datasets, chunked storage and filter pipeline (accessed 2026-10-17)
2. Hesela: HDF5 chunk cache, measured file-driver reads (accessed 2026-10-17)
3. Hesela: Content-defined chunking boundaries (accessed 2026-10-17)

JSON: /data/terms/hdf5-chunked-layout.json