hesela.devMeasured API behavior / synthetic fault injection
Parquet checksum and read-coverage observations
9 read cases across 3 repetitions show how checksum presence, verification and selected columns or row groups change detection of one deliberate payload bit flip.
Download CSV27 observations / 7 columnsMetadata and hashes / Raw JSON / Full analysis
Method
Write two deterministic fixtures with/without page CRC, flip one payload bit in copies, compare nine read scopes/settings against complete expected selected values; repeat on fresh temporary files. Each fixture has 128 rows in two 64-row groups and two required INT64 columns. Compression and dictionary encoding are disabled. The harness changes exactly one bit in row 17 of the first measurement chunk, from 100017 to 100016; these integers have no physical units.
| Case | Verify | Scope | Outcome |
|---|---|---|---|
| clean-checked | true | all-data | matches-selected-data |
| corrupt-default | default | all-data | wrong-values |
| corrupt-check-off | false | all-data | wrong-values |
| corrupt-check-on | true | all-data | checksum-error |
| projection-check-on | true | id-only | matches-selected-data |
| other-group-check-on | true | row-group-1 | matches-selected-data |
| metadata-only | not-applicable | metadata | metadata-readable |
| absent-crc-check-on | true | all-data | wrong-values |
| clean-no-crc-check-on | true | all-data | matches-selected-data |
Interpretation
The full checked read detects the deliberately altered checksummed payload. Default verification, disabled verification and absent-CRC cases return one incorrect value. Reading only the unmodified column or second row group succeeds without establishing anything about the damaged chunk. All 3 repeats agree; this is not an estimate of real-world error frequency.
Reproduce
Download the harness and pinned requirements together. It refuses existing output paths, creates disposable temporary fixtures only, verifies complete selected values and uses runtime checks that remain enabled with Python optimization. The dataset generator validates the raw evidence before flattening it into CSV.
python3 -m venv .venv
.venv/bin/pip install --only-binary=:all: -r parquet-integrity-requirements.txt
.venv/bin/python parquet-integrity.py --repetitions 3 --output observations.jsonMeasured environment: Python 3.12.3, PyArrow / Arrow C++ 21.0.0, Linux 6.8.0-146-generic, x86_64. File offsets, before/after hashes, mismatch coordinates and error text are retained in raw JSON. The pin identifies the tested version, not a recommendation that it is the latest release.
Limits
- Controlled synthetic one-bit fault on one host and pinned library; not field corruption frequency, power-loss testing, a storage benchmark, hostile-tampering assessment or exhaustive file validation. No compressed, encrypted, nested or dictionary pages tested. Projection and row-group cases compare only selected data.
- The same chosen bit and input are repeated; no population rate, detection probability or confidence interval is estimated.
- A successful checked read does not certify that CRCs existed on every page or that unselected data was validated.
Supplement: footer statistics and query completeness
Seven additional cases, each repeated three times, isolate metadata-driven row-group exclusion. The original 27 payload observations above remain unchanged. Every supplementary read enables page checksum verification.
The generated file contains 128 synthetic rows and four checksummed pages. A structured Thrift edit raises the first measurement group's stored minimum from 100000 to 100032 in both minimum fields, leaving all page bytes unchanged. The predicate measurement = 100017 then returns zero rows. A full checked scan returns all original values, and filtering it in memory recovers id 17. Removing the affected statistics or using wider valid bounds also recovers that row.
| Case | Read | Target |
|---|---|---|
| clean-filtered | 1 | 1 |
| narrow-filtered | 0 | 1 |
| absent-filtered | 1 | 1 |
| wide-filtered | 1 | 1 |
| narrow-full-scan | 128 | 128 |
| narrow-post-scan-filter | 1 | 1 |
| body-fault-positive-control | CRC error | 128 |
The harness verifies byte-identical page regions, independently recomputes all four CRCs, compares complete decoded tables, inspects selected row groups and checks a deliberately changed page as a positive CRC-detection control. A separate bounds comparison detects the invalid minimum. The full scan alone does not report it.
Supplementary CSV: 21 observations / Raw results and byte offsets / Provenance and column definitions / Validator and CSV generator
Download the statistics harness, pinned requirements and official Parquet 2.10.0 Thrift schema. The runner rejects any schema whose SHA-256 differs from its recorded pin. It creates private disposable fixtures and never edits a supplied Parquet file. Existing output paths are refused. Use normal Python or -O; the pinned Thrift parser requires docstrings and does not support -OO.
python3 -m venv .stats-venv
.stats-venv/bin/pip install --only-binary=:all: -r parquet-statistics-requirements.txt
.stats-venv/bin/python parquet-statistics.py --schema parquet.thrift --output observations.jsonTested packages: PyArrow 21.0.0, thriftpy2 0.5.3, ply 3.11 and six 1.17.0. Environment and code hashes are in the supplementary metadata. One implementation and host; deliberately selected, structurally valid but semantically invalid metadata. Not a spontaneous bit-error model, prevalence estimate, security exploit, benchmark or test of encrypted, compressed, nested or dictionary files. Repetitions are not independent reliability trials.
Format and implementation evidence: Statistics and page CRC scope; Arrow 21.0 row-group pruning. CC0-1.0 for Hesela's original code and observations; dependencies and downloaded Apache schema retain their own licenses.
Column definitions
- repetition
- One-based repeat of the same deterministic experiment; not an independent reliability trial.
- case
- Controlled read case identifier.
- verification
- Page checksum verification: true, false, default (false in PyArrow 21.0.0), or not-applicable for metadata-only inspection.
- scope
- Selected data or metadata; success describes only this scope.
- outcome
- Observed return or checksum exception, compared with expected selected payload.
- rows
- Returned rows, or declared rows for metadata; empty when checksum error prevents a result.
- mismatchCount
- Number of differing selected cells; empty for metadata or an error, not a claim of zero errors.
Sources and license
CC0-1.0 for Hesela's original harness and observations; Apache libraries and documentation retain their own licenses.
- Apache Parquet: page checksumming
- Apache Arrow 21.0: read_table checksum verification and column selection
- Apache Arrow 21.0: write_table page checksum option
- Apache Arrow 21.0.0: SerializedPageReader checksum presence guard
- Apache Parquet: PLAIN INT64 encoding
- Apache Parquet: file, column and page metadata
Definitions: Apache Parquet / Parquet page checksum / row group / column chunk / zone map.
Published by Hesela team
Report a correction · Editorial standards · Reuse guidance
Cite this page
Hesela team. Parquet checksum and read-coverage observations. Hesela. https://hesela.dev/datasets/parquet-integrity-boundaries/