{
  "status": "shipping",
  "supersedes": [],
  "supersededBy": [],
  "firstSeen": null,
  "lastSeen": null,
  "updated": "2026-10-17",
  "slug": "parquet-page-checksum",
  "term": "Parquet page checksum",
  "aliases": [],
  "definition": "A Parquet page checksum is an optional CRC32 value used to detect changes in the serialized page body, excluding its page header.",
  "note": "Scope: the checksum covers page bytes as stored, not the page header; it is not a whole-file or footer integrity certificate. [parquet-crc]\n\nVersion-specific controls: PyArrow 21.0 defaults both write_page_checksum and page_checksum_verification to false. The reader checks an eligible page only if verification is enabled and the CRC field is present. [arrow-read] [arrow-write] [arrow-code]\n\nInterpretation: enabling verification does not require every page to carry a checksum, nor does reading selected columns validate every column. Writer policy, reader policy and read coverage must be recorded separately. [arrow-code] [arrow-read]\n\nSelection metadata: a page CRC does not validate footer statistics used for row-group pruning. In Hesela's controlled PyArrow 21.0.0 example, narrowing the stored minimum excluded a real matching row while a full CRC-enabled scan returned every original value; a separate statistics-versus-values audit detected the inconsistency. This deliberately invalid footer is not a field-error rate estimate or a claim about every reader. [arrow-pruning] [hesela-statistics]",
  "related": [
    "parquet",
    "checksum",
    "backup-verification",
    "zone-map",
    "parquet-row-group",
    "parquet-column-chunk"
  ],
  "advantages": [],
  "limitations": [],
  "citations": [
    {
      "id": "parquet-crc",
      "title": "Apache Parquet: page checksumming",
      "url": "https://parquet.apache.org/docs/file-format/data-pages/checksumming/",
      "accessed": "2026-10-17"
    },
    {
      "id": "arrow-read",
      "title": "Apache Arrow 21.0: read_table checksum verification and column selection",
      "url": "https://arrow.apache.org/docs/21.0/python/generated/pyarrow.parquet.read_table.html",
      "accessed": "2026-10-17"
    },
    {
      "id": "arrow-write",
      "title": "Apache Arrow 21.0: write_table page checksum option",
      "url": "https://arrow.apache.org/docs/21.0/python/generated/pyarrow.parquet.write_table.html",
      "accessed": "2026-10-17"
    },
    {
      "id": "arrow-code",
      "title": "Apache Arrow 21.0.0: SerializedPageReader checksum presence guard",
      "url": "https://github.com/apache/arrow/blob/apache-arrow-21.0.0/cpp/src/parquet/column_reader.cc#L433-L443",
      "accessed": "2026-10-17"
    },
    {
      "id": "parquet-encoding",
      "title": "Apache Parquet: PLAIN INT64 encoding",
      "url": "https://parquet.apache.org/docs/file-format/data-pages/encodings/",
      "accessed": "2026-10-17"
    },
    {
      "id": "parquet-metadata",
      "title": "Apache Parquet: file, column and page metadata",
      "url": "https://parquet.apache.org/docs/file-format/metadata/",
      "accessed": "2026-10-17"
    },
    {
      "id": "arrow-pruning",
      "title": "Apache Arrow 21.0.0: row-group filtering from statistics expressions",
      "url": "https://github.com/apache/arrow/blob/apache-arrow-21.0.0/cpp/src/arrow/dataset/file_parquet.cc#L838-L899",
      "accessed": "2026-10-17"
    },
    {
      "id": "hesela-statistics",
      "title": "Hesela: controlled Parquet footer statistics and pruning observations",
      "url": "https://hesela.dev/research/parquet-statistics.json",
      "accessed": "2026-10-17"
    }
  ]
}