parquet-page-checksum · shipping

Parquet page checksum

A Parquet page checksum is an optional CRC32 value used to detect changes in the serialized page body, excluding its page header.

Related Observatory article: Parquet page checksum on hesela.com

Scope: the checksum covers page bytes as stored, not the page header; it is not a whole-file or footer integrity certificate. [1]

Version-specific controls: PyArrow 21.0 defaults both write_page_checksum and page_checksum_verification to false. The reader checks an eligible page only if verification is enabled and the CRC field is present. [2] [3] [4]

Interpretation: enabling verification does not require every page to carry a checksum, nor does reading selected columns validate every column. Writer policy, reader policy and read coverage must be recorded separately. [4] [2]

Selection metadata: a page CRC does not validate footer statistics used for row-group pruning. In Hesela's controlled PyArrow 21.0.0 example, narrowing the stored minimum excluded a real matching row while a full CRC-enabled scan returned every original value; a separate statistics-versus-values audit detected the inconsistency. This deliberately invalid footer is not a field-error rate estimate or a claim about every reader. [7] [8]

Relations

related

parquet, checksum, backup-verification, zone-map, parquet-row-group, parquet-column-chunk

Citations

1. Apache Parquet: page checksumming (accessed 2026-10-17)
2. Apache Arrow 21.0: read_table checksum verification and column selection (accessed 2026-10-17)
3. Apache Arrow 21.0: write_table page checksum option (accessed 2026-10-17)
4. Apache Arrow 21.0.0: SerializedPageReader checksum presence guard (accessed 2026-10-17)
7. Apache Arrow 21.0.0: row-group filtering from statistics expressions (accessed 2026-10-17)
8. Hesela: controlled Parquet footer statistics and pruning observations (accessed 2026-10-17)

Further reading

5. Apache Parquet: PLAIN INT64 encoding (accessed 2026-10-17)
6. Apache Parquet: file, column and page metadata (accessed 2026-10-17)

JSON: /data/terms/parquet-page-checksum.json