Controlled native-library observations / synthetic data

HDF5 checksum coverage and reader-state observations

Controlled HDF5 1.14.6 observations separate declared Fletcher32, partial-edge coverage, failure callbacks and reuse of unchecked cached values.

Metadata / Coverage observations / Reader observations / Interpretation

Method

One-dimensional synthetic uint32 arrays, eight-element chunks, one-bit payload mutations and intact controls. Seven coverage cases, eight fresh-reader combinations and four cache sequences, each repeated three times; three growth sequences and one fill control remain in raw JSON. Values start at 100. The coverage fixture has shape 12, maximum shape 16 and chunk length 8. Each full chunk holds 32 decoded bytes. Only the first stored payload byte of the selected chunk is mutated, by XOR with 1. Every case includes an intact-value control.

The CSV combines three explicitly labelled phases, not one homogeneous sample. Growth sequences and the fill control remain in complete JSON. returned-values describes successful I/O, not agreement with the original array. Inspect the raw returned values, callbacks and exceptions before interpreting success.

Coverage controls

First repetition shown; all three agreed. Every dataset declares Fletcher32. Edge skipping is explicitly enabled only in the named cases.

A zero filter mask coexisted with an unfiltered partial edge on this build. When that edge grew to a full chunk after a mutation, the changed value remained in the newly protected representation. This does not demonstrate ordinary default writes omitting Fletcher32, nor does it estimate a production failure rate.

Reader-state controls

First repetition of each cache sequence. The strict read has EDC enabled and no failure callback.

A callback that continues after failure differs from disabling detection: the full observations record Fletcher32 failure events with EDC still enabled. A later strict call may reuse decoded cached data. In these controls, the fresh-reader rejection occurs against identical stored bytes, not a different or repaired file. See the cache-state interpretation.

Reproduce

The file manifest and instructions list both runners, both test files, replay tools, requirements and source matrices. Keep those files together. The programs create synthetic images in memory; they do not accept production input paths. Use an isolated Python environment.

python -m unittest -v test_hdf5_filter_coverage.py test_hdf5_reader_policy.py
python verify_evidence.py
python verify_reader_policy.py

Runtime: Python 3.12.3, h5py 3.15.0, HDF5 1.14.6, NumPy 2.3.3, Linux x86_64. Exact replay also checks environment and source identity; a mismatch on another build is evidence to investigate, not a reason to overwrite the recorded result.

The CSV/metadata generator verifies runner hashes before deriving rows from the two saved traces. Neither CSV nor displayed numeric results are manually transcribed.

Column definitions

phase

Coverage mutation, fresh-reader policy, or strict read after permissive cache priming.

case

Designed case identifier; reader-policy cases include clean/damaged fixture.

repetition

Repeat on one build, not an independent population sample.

mask

Stored chunk filter mask; zero alone does not establish checksum coverage.

storedBytes

Stored target-chunk bytes, including checksum when present.

edcEnabled

Requested checking policy, not proof that checking ran.

cacheBytes

Explicit budget in reader cases; null in coverage cases, whose cache budget was not recorded.

outcome

returned-values or read-error; success does not establish agreement with the clean fixture.

Limits

• One Linux x86_64 build: h5py 3.15.0, HDF5 1.14.6, NumPy 2.3.3. Fileobj/BytesIO and v110/v110 format bounds.
• Designed cases establish behavior, not corruption incidence, confidence intervals, throughput or hardware-fault results.
• Edge skipping, EDC disabling and callback continuation are explicit opt-outs, not ordinary h5py defaults.
• Mask inspection is a coverage precondition, not checksum verification; raw chunk reads bypass filters.
• No SWMR, parallel I/O, compression chains, third-party filters, power failure or cross-version claim.
• Later handle/refresh anomaly research is deliberately excluded from this dataset and its recommendations.

Sources and reuse

CC0-1.0 for Hesela's original code and observations; upstream materials retain their licenses.

1. HDF Group: H5Pset_chunk_opts, Dataset Creation Properties
2. HDF5 1.14.6 H5Dchunk.c: chunk flush and partial-edge read paths
3. HDF Group: H5Dwrite_chunk and H5Dread_chunk
4. h5py DatasetID API: chunk information and direct chunk methods
5. HDF Group: H5Pset_edc_check and H5Pget_edc_check
6. HDF5 1.14.6 H5Zfletcher32.c
7. h5py 3.15.1: Datasets and filter pipeline
8. HDF5 1.14.6 H5Dpublic.h
9. HDF5 1.14.6: H5Z_pipeline filter-failure callback

Related definitions: chunked layout, raw-data chunk cache and checksum.