{
  "slug": "arrow-ipc-file",
  "term": "Arrow IPC file",
  "aliases": [],
  "definition": "An Arrow IPC file adds a footer with schema and block locations to support random access to serialized Arrow record batches.",
  "note": "Structure: the footer locates record batches and dictionaries for random access. Dictionary replacement is unsupported in the file format, while deltas extend a dictionary. This differs from sequential stream replacement. [format]\n\nConversion: in Arrow 21, unify_dictionaries applies to write_table. Native controls with changed dictionary order succeeded when both batches were in one table, but failed across separate write_batch or write_table calls. The successful writer remapped indices while preserving the intended ordered string values. [options] [writer] [evidence]\n\nDeltas: a prefix extension succeeded with emit_dictionary_deltas enabled; reordering did not. File and stream readers can attach different dictionary arrays while returning the same tested string values, so physical dictionary identity is a different acceptance target. [evidence]\n\nIntegrity: a readable footer and full structural validation do not compare values with an independent original. Preserve the intended artifact digest or an explicitly defined semantic expectation. [replay]",
  "related": [
    "arrow-ipc-stream",
    "backup-verification",
    "restore-test",
    "parquet",
    "checksum"
  ],
  "citations": [
    {
      "id": "format",
      "title": "Apache Arrow 21.0: IPC streaming and file format, dictionary messages",
      "url": "https://arrow.apache.org/docs/21.0/format/Columnar.html",
      "accessed": "2026-11-04"
    },
    {
      "id": "options",
      "title": "PyArrow 21.0: table-scoped dictionary unification and delta options",
      "url": "https://arrow.apache.org/docs/21.0/python/generated/pyarrow.ipc.IpcWriteOptions.html",
      "accessed": "2026-11-04"
    },
    {
      "id": "writer",
      "title": "Apache Arrow 21.0.0: WriteTable and WriteDictionaries implementation",
      "url": "https://github.com/apache/arrow/blob/apache-arrow-21.0.0/cpp/src/arrow/ipc/writer.cc",
      "accessed": "2026-11-04"
    },
    {
      "id": "evidence",
      "title": "Hesela: retained native Arrow dictionary observations and message hashes",
      "url": "https://hesela.dev/research/arrow-ipc-completeness/dictionary-native/capture-normal/observations.json",
      "accessed": "2026-11-04"
    },
    {
      "id": "replay",
      "title": "Hesela: Arrow IPC reproduction, acceptance rules and limitations",
      "url": "https://hesela.dev/research/arrow-ipc-completeness/REPRODUCE.md",
      "accessed": "2026-11-04"
    }
  ],
  "provenance": [
    "Synthetic native observations on one PyArrow/Arrow C++ 21.0.0 build and host; not field rates, a current-release guarantee or a new vulnerability. [evidence]",
    "String equivalence means ordered non-null values in these controls, not arbitrary Arrow schemas, ordered categoricals or extension types. [replay]"
  ],
  "status": "shipping",
  "advantages": [],
  "limitations": [],
  "supersedes": [],
  "supersededBy": [],
  "firstSeen": null,
  "lastSeen": null,
  "updated": "2026-11-04"
}