hesela.devArrow IPC file
An Arrow IPC file adds a footer with schema and block locations to support random access to serialized Arrow record batches.
Structure: the footer locates record batches and dictionaries for random access. Dictionary replacement is unsupported in the file format, while deltas extend a dictionary. This differs from sequential stream replacement. [1]
Conversion: in Arrow 21, unify_dictionaries applies to write_table. Native controls with changed dictionary order succeeded when both batches were in one table, but failed across separate write_batch or write_table calls. The successful writer remapped indices while preserving the intended ordered string values. [2] [3] [4]
Deltas: a prefix extension succeeded with emit_dictionary_deltas enabled; reordering did not. File and stream readers can attach different dictionary arrays while returning the same tested string values, so physical dictionary identity is a different acceptance target. [4]
Integrity: a readable footer and full structural validation do not compare values with an independent original. Preserve the intended artifact digest or an explicitly defined semantic expectation. [5]
Relations
Citations
- Apache Arrow 21.0: IPC streaming and file format, dictionary messages
- PyArrow 21.0: table-scoped dictionary unification and delta options
- Apache Arrow 21.0.0: WriteTable and WriteDictionaries implementation
- Hesela: retained native Arrow dictionary observations and message hashes
- Hesela: Arrow IPC reproduction, acceptance rules and limitations