Measured API behavior / synthetic input values

HDF5 SWMR visibility observations

Nine ordered checkpoints repeated 3 times distinguish readable HDF5 arrays from complete published pairs, using synthetic integer inputs and a live separate reader.

Metadata / Raw observations / Python harness / Full analysis

Method

One live writer and one spawned reader; pipe handshakes pause writes at each observation; all datasets created before SWMR; fixed-size integers, one-element chunks, no compression. All three repetitions produced identical observations. Counts are records, not bytes; integers are invented test payloads, not sensor measurements.

What the checks establish

The reader sees unequal lengths after x is extended, then equal lengths before publication advances. A count-first selection excludes the incomplete tail. An out-of-range count is rejected, but overwriting an already published x value is not detected by count and length checks. This demonstrates why immutability is a separate application requirement.

Reproduce

Download the harness and requirements into one directory. The harness creates only private temporary files, uses bounded waits, checks every expected checkpoint even with Python optimization enabled and prints JSON. Select a filesystem with --temp-dir. The CSV is a flattened view of that JSON, with the generator retained for provenance.

python3 -m venv .venv
.venv/bin/pip install --only-binary=:all: -r hdf5-swmr-requirements.txt
.venv/bin/python hdf5-swmr.py --repetitions 3 > observations.json

Environment: Python 3.12.3, h5py 3.15.0, HDF5 1.14.6, NumPy 2.3.3, Linux 6.8.0-146-generic, sec2 driver. The source SHA-256 and raw/CSV hashes are in metadata. The latest format bound is relative to the installed library.

Limits

• One host. Synthetic values are test inputs, not sensor measurements. No power loss, process crash, fsync audit, network filesystem, competing writers or performance test. Watermark ordering is an application demonstration, not a crash-safe transaction protocol.
• Only 3 repeats on one host; no failure probability or confidence interval is estimated.
• Final two checkpoints intentionally violate the publication rule. The count is not a snapshot, checksum or crash-safe commit.
• Filesystem magic does not identify exact ext-family version or mount options.
• A deterministic pause exposes a possible intermediate state, not its production frequency.

Separate process-exit supplement

The CSV above remains the live-writer experiment. A separate four-case experiment, repeated three times, ends a writer after explicit HDF5 flushes with or without normal close. It records the version-3 status byte, fresh ordinary/SWMR open outcomes, full payloads and publication count. It does not simulate power loss, kernel failure or interrupted device I/O.

After an unclean SWMR exit, the ordinary open was rejected while the SWMR open read the payload. The unpublished case still selected no pairs. The non-SWMR control rejected both opens. The comparison table and interpretation keep these results distinct from the 27 CSV checkpoints.

Process-exit JSON / Python source / Pinned requirements. The JSON includes environment, method, limits and SHA-256 hashes. Original supplement code and observations use CC0-1.0. Run with --repetitions 3; only private temporary files are created.

Column definitions

repetition

One-based repeated run on the same host; not an independent reliability sample.

checkpoint

One-based ordered checkpoint within the run.

phase

Operation boundary chosen by the harness.

xRecords

Refreshed x dataset length in records.

yRecords

Refreshed y dataset length in records.

watermarkRecords

Application-selected published prefix length in records.

rejected

True if the count exceeds available paired rows.

selectedPrefix

Selected integer pairs, or null when rejected; test values have no physical units.

Sources and license

CC0-1.0 for Hesela's original harness and observations; upstream libraries and documents retain their own licenses.

1. h5py 3.15: Single Writer Multiple Reader
2. The HDF Group: SWMR metadata flush dependencies
3. The HDF Group: H5Dflush and H5Drefresh
4. The HDF Group: File locking and superblock flags
5. The HDF Group: h5clear is not a general repair tool
6. The HDF Group: HDF5 Data Model and File Structure

Definitions: HDF5 and HDF5 SWMR.