Measured library reads / synthetic array

HDF5 chunk-cache read observations

Six cache and access-order cases, repeated three times, measure raw chunk reads for 32 scalar selections from a synthetic compressed HDF5 array.

Metadata / Full trace / Runner / Analysis

Method

Fresh HDF5 open per case over the same synthetic in-memory file. Trace read/readinto calls during 32 scalar selections; match exact stored chunk extents; retain metadata reads separately in each step. The array is 64 by 64 unsigned four-byte integers; each 32 by 32 chunk is 4,096 decoded bytes. All six cases return the same 32 selected values, in either round-robin or grouped order. Each case starts with a fresh HDF5 open.

Interpretation

More byte capacity does not fix the forced one-slot collisions. Seventeen slots plus enough decoded-byte capacity gives four reads, as does grouping selections with a one-chunk cache. A two-KiB cache fits the compressed chunk sizes, but not their decoded images. The article explains these separate controls and why no timing speedup follows from the counts.

Reproduce

Download the runner, tests and requirements into one directory. The runner creates its file image in memory, validates payloads and effective settings, and rejects ambiguous read overlaps. The CSV/metadata generator verifies source hashes before flattening the raw trace.

python3 -m venv .venv
.venv/bin/pip install --only-binary=:all: -r hdf5-chunk-cache-requirements.txt
.venv/bin/python test_hdf5_chunk_cache.py
.venv/bin/python hdf5-chunk-cache.py --repetitions 3 > observations.json

Environment: Python 3.12.3, h5py 3.15.0, HDF5 1.14.6, NumPy 2.3.3, zlib 1.3; driver fileobj over BytesIO. File-format bounds: v110/v110. No random seed is needed for the deterministic integer array.

Limits

• One library build and synthetic trace, not a workload sample or paper replication.
• No disk, OS page-cache, latency, throughput, power-loss or device-failure measurement.
• Decoded bytes are inferred as raw chunk reads times uncompressed chunk size; filter callbacks were not instrumented.
• One slot deliberately forces conflicts; 17 slots is a measured control for this layout, not a universal optimum.

Column definitions

repetition

One-based repeat on the same host, not an independent performance sample.

case

Stable case identifier.

order

Scalar coordinate order: round-robin across chunks, or grouped by chunk.

rdccNslots

Effective raw-data chunk cache hash-table slots.

rdccNbytes

Effective raw-data chunk cache byte capacity, excluding other memory.

rdccW0

Effective eviction parameter, fixed at zero.

scalarSelections

Number of scalar application reads.

selectedBytes

Returned application bytes: scalar selections times four.

rawChunkReads

File-driver read callbacks exactly matching a stored chunk extent.

rawChunkBytes

Sum of stored payload bytes in those callbacks, excluding metadata.

chunkDecodeBytesInferred

Arithmetic inference: rawChunkReads times 4096, not measured filter callbacks.

Sources and license

CC0-1.0 for Hesela's original code and observations; upstream libraries and documents retain their licenses.

1. The HDF Group (2015): Improving I/O Performance When Working with HDF5 Compressed Datasets, pp. 7-20
2. The HDF Group: Chunking in HDF5, chunk caching and filters
3. h5py 3.15: File objects, Python file-like objects and chunk cache
4. h5py 3.15: Datasets, chunked storage and filter pipeline
5. h5py low-level API: DatasetID.get_chunk_info_by_coord

Definitions: Raw-data chunk cache, chunked layout and HDF5.