hesela.devZarr consolidated metadata
Zarr consolidated metadata collects metadata for a hierarchy into a consolidated representation to reduce separate metadata reads.
Purpose: consolidation reduces metadata requests, especially for hierarchies on high-latency storage. It does not itself capture chunk values as a dataset snapshot. [1]
Freshness: metadata changes can make the consolidated representation stale; applications must coordinate publication and reader refresh with their update protocol. Bypassing consolidation does not establish an atomic multi-object snapshot. [1]
Measured boundary: in Hesela's ordered Zarr-Python 3.1.3 format-2 experiment, a stale consolidated reader saw an eight-element shape with a newly overwritten first value. A fresh unconsolidated reader saw twelve elements; re-consolidation did not refresh an already retained handle. This is version-scoped local evidence, not a format-3 concurrency result. [2]
Relations
- related
- Zarr, Zarr fill value, Backup verification