hesela.devArrow IPC stream
An Arrow IPC stream serializes a schema and a sequence of dictionary and record-batch messages for sequential reading.
Completion: a stream can end with an explicit EOS message or input closure. Successful exhaustion does not establish the producer's intended row count or whole-container coverage. [1]
Interpretation: dictionary-encoded indices depend on the effective dictionary state. A replacement changes that state; a delta appends entries. Preserve the applicable state and message order when restarting or reconstructing a stream. Equal indices in different batches need not denote equal values; compare decoded values or remap to a verified common dictionary before reconciling records. [1]
Native control: a stale replacement preserved an 880-byte, six-row stream and both complete record-batch messages but changed three decoded labels. PyArrow 21.0.0 read_all and full validation passed. Omitting the initial dictionary caused reading to fail; omitting a required delta allowed reading but failed full validation. [2]
Acceptance: retain the intended producer identity, generation and whole-object digest under an explicit trust policy. A digest computed after a wrong reconstruction cannot establish the earlier expectation. Permitted re-encoding needs a defined semantic comparison as well as structural validation. [3]
Relations
Citations
- Apache Arrow 21.0: IPC streaming and file format, dictionary messages
- Hesela: retained native Arrow dictionary observations and message hashes
- Hesela: Arrow IPC reproduction, acceptance rules and limitations