{
  "status": "shipping",
  "supersedes": [],
  "supersededBy": [],
  "firstSeen": null,
  "lastSeen": null,
  "updated": "2026-10-21",
  "slug": "hdf5-external-storage",
  "term": "HDF5 external storage",
  "aliases": [
    "HDF5 external raw storage",
    "HDF5 external file list"
  ],
  "definition": "HDF5 external storage maps a contiguous dataset's raw bytes onto an ordered list of byte ranges in non-HDF5 files.",
  "note": "Mapping: each external-file-list segment specifies a filename, byte offset and reserved size; it is not a VDS mapping or a link to an HDF5 object. h5py exposes the list through Dataset.external. [h5-external-guide] [h5py-external]\n\nEOF behavior: the HDF5 1.14.6 reader zeroes the remaining bytes when a referenced file reaches EOF; an open failure takes an error path. This byte-level behavior differs from substituting a dataset fill element. [h5-external-source]\n\nMeasured example: truncating a little-endian int32 fixture inside its second integer changed 287454020 to 13124 without a read error, even with fillvalue=-999. Big-endian storage returned 287440896. Missing files errored; empty files returned zeros. These are controlled synthetic observations on one build. [hesela-efl]\n\nPreservation: validate the required mapped ranges, offsets and source identity together. In the fixture, a 16-byte raw file equalled Dataset.nbytes but lacked eight mapped bytes because data started at offset eight. Same-length replacement passed the range check but changed values. The full analysis and downloadable harness retain the evidence and limits. [hesela-efl-analysis]\n\nResolution: external raw files use HDF5_EXTFILE_PREFIX, which overrides the dataset-access prefix. Default relative lookup uses the process working directory; ${ORIGIN} in the prefix supports resolution relative to the containing HDF5 file. Preserve the reader settings as well as the stored filename. [h5-efile-prefix]",
  "related": [
    "hdf5",
    "hdf5-virtual-dataset",
    "hdf5-fill-value",
    "backup-verification"
  ],
  "citations": [
    {
      "id": "h5-external-guide",
      "title": "The HDF Group: Dataset guide, External Storage Properties",
      "url": "https://support.hdfgroup.org/documentation/hdf5/latest/_h5_d__u_g.html",
      "accessed": "2026-10-21"
    },
    {
      "id": "h5-external-source",
      "title": "HDF5 1.14.6: H5D__efl_read EOF and open-error handling",
      "url": "https://github.com/HDFGroup/hdf5/blob/hdf5_1.14.6/src/H5Defl.c#L236-L344",
      "accessed": "2026-10-21"
    },
    {
      "id": "h5py-external",
      "title": "h5py 3.15.0: Dataset.external and Dataset.nbytes",
      "url": "https://docs.h5py.org/en/3.15.0/high/dataset.html#h5py.Dataset.external",
      "accessed": "2026-10-21"
    },
    {
      "id": "hesela-efl",
      "title": "Hesela: external raw-file truncation observations (42 fixtures, HDF5 1.14.6)",
      "url": "https://hesela.dev/research/hdf5-external-eof.json",
      "accessed": "2026-10-21"
    },
    {
      "id": "hesela-efl-analysis",
      "title": "Hesela: HDF5 dependencies, truncated external raw files",
      "url": "https://hesela.com/analyses/hdf5-virtual-datasets-missing-sources/#external-eof",
      "accessed": "2026-10-21"
    },
    {
      "id": "h5-efile-prefix",
      "title": "The HDF Group: H5Pset_efile_prefix and HDF5_EXTFILE_PREFIX",
      "url": "https://portal.hdfgroup.org/documentation/hdf5/latest/group___d_a_p_l.html",
      "accessed": "2026-10-21"
    }
  ],
  "advantages": [],
  "limitations": []
}