# Arrow IPC: completeness and dictionary interpretation

## What these controls establish

In the retained Arrow 21.0.0 controls, a well-formed shortened stream reads
successfully with fewer rows. Separately, a stale dictionary preserves all
record-batch messages, six rows and an 880-byte stream length, but changes three
decoded labels despite successful full validation.

These are deliberate synthetic reconstructions, not new vulnerabilities or
estimates of production loss. The experiments use one release, host and
architecture. They test neither Arrow Flight nor crashes, performance, nulls,
nested or extension types. The non-dictionary integer study and dictionary
string study are separate populations; their case counts must not be pooled
into an incident rate.

The original study retains 15 fixtures through two reading APIs and two
same-host process captures (60 operation records). The dictionary supplement
retains nine writer configurations and four reconstructed streams, each in
normal and optimized Python processes. All original fixture bytes and
observations are included. Repetition here is not independent replication.

## Reproduce

Use an isolated Python 3.12 environment, then from the extracted directory:

```sh
python -m pip install -r requirements.txt
sha256sum -c SHA256SUMS
python -B verify.py
python -B native_dictionary.py --verify dictionary-native/capture-normal
python -B -O native_dictionary.py --verify dictionary-native/capture-optimized
python -B -m unittest -q test_experiment.py test_native_dictionary.py test_dictionary_examples.py test_review_evidence.py
python -B -O -m unittest -q test_experiment.py test_native_dictionary.py test_dictionary_examples.py test_review_evidence.py
```

The requirements pin PyArrow 21.0.0. Exact dictionary replay additionally checks
native-library hashes; another wheel/platform is a new replication, not this
exact replay. Record a separate capture with
`python -B native_dictionary.py --output NEW_DIRECTORY` when comparing builds.
The output directory must not exist. Do not relabel different observations as
the retained evidence.

Without installing Arrow, `python -B -m unittest -q test_dictionary_examples.py
test_review_evidence.py` checks finite illustrations and retained evidence
derivation only, not native execution.

## Acceptance and interpretation

Retain the identity, generation and complete-object digest from the intended
completed producer under an explicit trust policy. Validate the decoded
structure as well. A digest made after a wrong reconstruction cannot recover
the earlier expectation. The archive's SHA256SUMS detects differences from this
package; it does not independently authenticate who produced it.

If re-encoding is permitted, specify semantic equivalence separately. The
dictionary examples compare ordered non-null string values, not arbitrary
Arrow types or ordered categoricals. Dictionary arrays may differ while these
values agree. Optional end-of-stream removal can preserve values while changing
the whole-object digest.

The historical case ID `file-truncated-footer` is retained unchanged. It removes
the last byte of the final ARROW1 signature; footer metadata is unchanged.
It does not test arbitrary footer corruption. A source cursor position is not
a measure of byte-validation coverage.

## Related query research

Liu, Pavlenko, Interlandi and Haynes, [PVLDB 2023](https://www.vldb.org/pvldb/vol16/p3044-liu.pdf),
Section 8.1.1, translate string predicates through each chunk's dictionary
before comparing integer codes. This gives a query-processing reason to retain
the applicable dictionary, alongside the recovery dependency demonstrated here.
Their Section 4 uses Arrow 5.0.0 and non-nested relational workloads; it does
not validate this Arrow 21 reconstruction or establish an archive-loss rate.
We did not reproduce their performance experiments.

## Sources and reuse

Primary contracts: [Arrow 21 format](https://arrow.apache.org/docs/21.0/format/Columnar.html),
[writer options](https://arrow.apache.org/docs/21.0/python/generated/pyarrow.ipc.IpcWriteOptions.html)
and the [versioned writer](https://github.com/apache/arrow/blob/apache-arrow-21.0.0/cpp/src/arrow/ipc/writer.cc).
Acquisition receipts are included; third-party snapshots and libraries are not.

Original harnesses, synthetic fixtures, observations and publication derivations:
CC0-1.0, https://creativecommons.org/publicdomain/zero/1.0/.
Third-party libraries retain their own licenses. The source packet is preserved
at commit 7d5d816db7dd12e239578ec197992012f2c675ab.