hesela.devHDF5 external storage
HDF5 external storage maps a contiguous dataset's raw bytes onto an ordered list of byte ranges in non-HDF5 files.
Mapping: each external-file-list segment specifies a filename, byte offset and reserved size; it is not a VDS mapping or a link to an HDF5 object. h5py exposes the list through Dataset.external. [1] [3]
EOF behavior: the HDF5 1.14.6 reader zeroes the remaining bytes when a referenced file reaches EOF; an open failure takes an error path. This byte-level behavior differs from substituting a dataset fill element. [2]
Measured example: truncating a little-endian int32 fixture inside its second integer changed 287454020 to 13124 without a read error, even with fillvalue=-999. Big-endian storage returned 287440896. Missing files errored; empty files returned zeros. These are controlled synthetic observations on one build. [4]
Preservation: validate the required mapped ranges, offsets and source identity together. In the fixture, a 16-byte raw file equalled Dataset.nbytes but lacked eight mapped bytes because data started at offset eight. Same-length replacement passed the range check but changed values. The full analysis and downloadable harness retain the evidence and limits. [5]
Resolution: external raw files use HDF5_EXTFILE_PREFIX, which overrides the dataset-access prefix. Default relative lookup uses the process working directory; ${ORIGIN} in the prefix supports resolution relative to the containing HDF5 file. Preserve the reader settings as well as the stored filename. [6]
Relations
Citations
- The HDF Group: Dataset guide, External Storage Properties
- HDF5 1.14.6: H5D__efl_read EOF and open-error handling
- h5py 3.15.0: Dataset.external and Dataset.nbytes
- Hesela: external raw-file truncation observations (42 fixtures, HDF5 1.14.6)
- Hesela: HDF5 dependencies, truncated external raw files
- The HDF Group: H5Pset_efile_prefix and HDF5_EXTFILE_PREFIX