# BGZF and tabix: valid files, incomplete indexed answers

Reviewed and replayed 25 October 2026. This package accompanies the existing
Hesela compressed-archive analysis, section bgzf-evidence. It supersedes the
research-only publication status without changing the four original captures.

## Result and limits

Unmodified HTSlib 1.22.1 and 1.24 return no rows with exit status zero for a
query that should find two records when generation B is paired with generation
A's valid TBI or CSI index. Data and index integrity tests pass separately.
Full decompression, sequence-name listing, a broad query and an unchanged-region
query pass. Rebuilding the index from B restores the expected narrow answer.

The fixture has three invented BED records, not biological observations.
Two chr1 coordinate pairs move from around 100000 to around 900000; chr2 stays
unchanged. The CLI query chr1:900001-900030 is 1-based inclusive; the target BED
interval chr1 900000 900030 is 0-based half-open. Both seek the same positions.

Four original captures contain 14 configurations x 9 operations x 4 = 504
observations. Two fresh native captures add 252 observations with the same
semantic outcomes. These repetitions use the same host and build configuration;
they are not independent implementations, field incidence, benchmark results,
a security vulnerability or independent peer review. Other mismatches can fail
loudly. The corrupt-CRC control is rejected; no universal detection claim follows.

An older index mtime warns without failing; equal/newer stale timestamps do not
warn. Sequential -T succeeds for this fixture but the inspected text path still
loads index metadata. It is not general index-free recovery.

## Reproduce

Obtain the official HTSlib release archives linked in provenance.json; verify
their recorded SHA-256. Build each in a disposable directory with:

```sh
./configure --disable-libcurl --disable-s3 --disable-gcs --disable-bz2 --disable-lzma
make -j2 bgzip tabix
python3 probe.py --bin-dir /absolute/path/to/htslib-1.24 --output fresh.json
python3 -m unittest -v test_probe.py
python3 -O -m unittest -v test_probe.py
python3 verify.py
python3 publish.py --check
sha256sum -c SHA256SUMS
```

The Python harness uses only the standard library and disposable generated files.
It accepts no external dataset and refuses an existing output file. Use a new
output name for each capture.

The validator now checks that both executables belong to the same recorded
release and that their hashes agree with provenance, not merely their version
strings. Three negative tests reproduced the prior validation gap before repair.
For a different compiler/build, produce and review a separate build-provenance
record, then call validate(capture, provenance=that_record); do not relabel its
binaries as the archived build. Rebuilding on our original host produced exactly
the recorded executable hashes. The probe and original captures are unchanged.

publication.json is generated from all six captures. Its seven displayed rows
combine TBI and CSI only after checking agreement of every displayed outcome.
For site regeneration, run publish.py --com-root COM_REPOSITORY --dev-root\nDEV_REPOSITORY; --check verifies the same destinations without rewriting them.\n\nSHA256SUMS covers the exported source and evidence; the ZIP contains those exact
files. Hashes detect changes relative to a trusted copy, not authorship.

## Sources and interpretation

Read Heng Li, Tabix, Bioinformatics 27(5), 718-719 (2011),
https://doi.org/10.1093/bioinformatics/btq671
(full text: https://pmc.ncbi.nlm.nih.gov/articles/PMC3042176/).
Sections 2-4 describe retrieval indexing and historical performance; they do not
report our mismatched-generation experiment. No performance number is reused.

sources.json links the pinned SAM/BGZF, TBI and CSI formats, release manuals and
native implementation. The publication review checked HTSlib 1.24 hts.c
hts_itr_query (no selected bins) and hts_idx_load3 (mtime warning), and tabix.c
text index loading; retained source hashes match provenance. Specification
layout does not substitute for execution evidence.

Engineering inference: establish data/index agreement first, then retain the
pair and coordinate conventions as one generation. A trusted manifest can bind
both digests; a manifest built from an already wrong pair freezes the mistake.
Neither successful broad queries nor sampled narrow queries prove completeness
for every possible query.

No third-party paper, dataset or HTSlib source/binary is redistributed here.
All payloads and observations were generated by Hesela. The original HTSlib
archives are MIT-licensed upstream; refer to their included license.