{
  "slug": "arrow-ipc-stream",
  "term": "Arrow IPC stream",
  "aliases": [],
  "definition": "An Arrow IPC stream serializes a schema and a sequence of dictionary and record-batch messages for sequential reading.",
  "note": "Completion: a stream can end with an explicit EOS message or input closure. Successful exhaustion does not establish the producer's intended row count or whole-container coverage. [format]\n\nInterpretation: dictionary-encoded indices depend on the effective dictionary state. A replacement changes that state; a delta appends entries. Preserve the applicable state and message order when restarting or reconstructing a stream. Equal indices in different batches need not denote equal values; compare decoded values or remap to a verified common dictionary before reconciling records. [format]\n\nNative control: a stale replacement preserved an 880-byte, six-row stream and both complete record-batch messages but changed three decoded labels. PyArrow 21.0.0 read_all and full validation passed. Omitting the initial dictionary caused reading to fail; omitting a required delta allowed reading but failed full validation. [evidence]\n\nAcceptance: retain the intended producer identity, generation and whole-object digest under an explicit trust policy. A digest computed after a wrong reconstruction cannot establish the earlier expectation. Permitted re-encoding needs a defined semantic comparison as well as structural validation. [replay]",
  "related": [
    "arrow-ipc-file",
    "backup-verification",
    "restore-test",
    "parquet",
    "checksum"
  ],
  "citations": [
    {
      "id": "format",
      "title": "Apache Arrow 21.0: IPC streaming and file format, dictionary messages",
      "url": "https://arrow.apache.org/docs/21.0/format/Columnar.html",
      "accessed": "2026-11-04"
    },
    {
      "id": "evidence",
      "title": "Hesela: retained native Arrow dictionary observations and message hashes",
      "url": "https://hesela.dev/research/arrow-ipc-completeness/dictionary-native/capture-normal/observations.json",
      "accessed": "2026-11-04"
    },
    {
      "id": "replay",
      "title": "Hesela: Arrow IPC reproduction, acceptance rules and limitations",
      "url": "https://hesela.dev/research/arrow-ipc-completeness/REPRODUCE.md",
      "accessed": "2026-11-04"
    }
  ],
  "provenance": [
    "Synthetic native observations on one PyArrow/Arrow C++ 21.0.0 build and host; not field rates, a current-release guarantee or a new vulnerability. [evidence]",
    "String equivalence means ordered non-null values in these controls, not arbitrary Arrow schemas, ordered categoricals or extension types. [replay]"
  ],
  "status": "shipping",
  "advantages": [],
  "limitations": [],
  "supersedes": [],
  "supersededBy": [],
  "firstSeen": null,
  "lastSeen": null,
  "updated": "2026-11-04"
}