hdf5-swmr · shipping · also HDF5 single-writer/multiple-reader

HDF5 SWMR

HDF5 SWMR is an access mode that permits one writer and multiple readers to use an HDF5 file with ordered metadata flushing under the required filesystem semantics.

Related Observatory article: HDF5 SWMR on hesela.com

Mechanism: metadata flush dependencies prevent readers following references to unflushed child structures. This requires compatible POSIX write semantics. [2]

Reader protocol: h5py writers flush datasets and readers refresh them; notifications are separate. Objects are created before switching to SWMR. [1]

Application boundary: Hesela observed unequal lengths in two readable arrays, and showed that a publication count does not prevent mutation of previously published values. This was a paused-writer observation, not a crash or power-loss test. [3]

Metadata retries: HDF5 can retry inconsistent checksummed metadata reads under a bounded attempt policy. The retry information API describes library metadata reads, not NAND recovery counters or device-failure rates; the linked Hesela experiment does not measure retries. [4]

Process-exit boundary: a separate post-flush experiment retained write/SWMR status flags after os._exit. Ordinary opens were rejected while SWMR reads worked; readable but unpublished payloads still selected no pairs. This is not power-loss evidence, and status flags do not establish whether a writer is alive. [5] [6]

Relations

related

hdf5, crash-consistency, advisory-file-lock

Limitations

• The documented SWMR scope forbids adding or deleting objects while SWMR writing is active; plan groups, datasets and attributes before switching modes. [2, 1]
• The HDF Group's cited scope excludes modification or append operations on variable-length, string and region-reference datatypes. [2]
• Correct SWMR behavior requires the documented POSIX write semantics; an HDF5-readable path alone does not establish filesystem suitability. [2, 1]
• Reader visibility is not a cross-dataset transaction or proof of recovery after power loss; the measured example assumes immutable published rows. [3]

Citations

1. h5py 3.15: Single Writer Multiple Reader (accessed 2026-10-17)
2. The HDF Group: SWMR metadata flush dependencies (accessed 2026-10-17)
3. HDF5 SWMR: a readable file is not a complete measurement (accessed 2026-10-17)
4. The HDF Group: H5Fget_metadata_read_retry_info and metadata read attempts (accessed 2026-10-17)
5. Hesela: HDF5 process-exit observations (accessed 2026-10-17)
6. The HDF Group: File locking and superblock status (accessed 2026-10-17)

JSON: /data/terms/hdf5-swmr.json