hesela.devMeasured Linux API behavior / synthetic payloads
Linux zero-copy source lifetime: eight file-to-pipe checks
40 file-to-pipe observations, plus 10 separate tee(2) checks, distinguish source stability from enqueue and consumer completion.
Download CSV40 observations / 9 columnsMetadata / Full observations / Harness / Tests / Interpretation and primary-source synthesis
Method
Sequential one-page file-to-pipe tests; source mutation finishes before pipe consumption except the consume-first control. Each file starts with a page of A bytes. B replaces its contents or pathname; C is written through a retained old descriptor after replacement. Hashes, identity comparisons, event order and full byte counts are preserved in JSON. Five repetitions yielded the same results; this is repeatability on one host, not a probability estimate.
| Case | Received | Path | Old inode | Matches initial hash |
|---|---|---|---|---|
| buffered-overwrite | A | B | B | Yes |
| sendfile-overwrite | B | B | B | No |
| splice-overwrite | B | B | B | No |
| sendfile-overwrite-fsync | B | B | B | No |
| sendfile-replace-path | A | B | A | Yes |
| sendfile-replace-then-write-old | C | B | C | No |
| sendfile-close-then-overwrite | B | B | B | No |
| sendfile-consume-then-overwrite | A | B | B | Yes |
Reproduce
The harness requires Linux, Python 3.10 or later and GNU stat. It uses new files in private temporary directories, no raw devices or existing user files. Use --temp-dir to select the filesystem. Pipe capacity and exact transfer counts are checked; API errors abort the run.
python3 zero-copy-lifetime.py --repetitions 5 --output observations.jsonFor regression tests, place the harness, published JSON and test file in one directory, then run python3 -B -m unittest test_zero_copy_lifetime. The frozen-receipt test checks the recorded A/B/C outcomes; live probes test controls without treating mutable-source outcomes as a portable API guarantee.
The dataset generator creates the CSV and metadata from frozen JSON. Keep it in one directory with both harnesses, both observation JSON files, both harness test files, the source list and generator tests. Run python3 -B build-zero-copy-dataset.py; it creates a sibling data/ directory. Run python3 -B -OO -m unittest test_zero_copy_dataset to check rejection of inconsistent receipts and a fresh-directory roundtrip.
Validation checks case coverage, transfer sizes, payload hashes and comparison flags before writing derived files. Different repetition outcomes require review of the summary instead of silently displaying only the first result. SHA-256 checks detect inconsistency; they are not a signature proving where observations originated. The optional --com-root argument mirrors raw evidence into a COM checkout.
Separate tee(2) supplement
The CSV above remains the eight-case experiment. Ten additional observations compare two consumers after tee(2) duplicates a pipe: the first consumer is read and hashed, the file is overwritten, then the second consumer is read and hashed. With sendfile input they observed A then B; with buffered input both observed A. All five repetitions agreed.
Supplement JSON / Supplement harness / Supplement tests. Place the supplement harness beside the main harness and run python3 -B tee-source-lifetime.py --repetitions 5 --output tee-observations.json. This uses libc through Python ctypes and tests the Linux tee system call, not GNU tee. It demonstrates a broken source-stability assumption, not a checksum collision.
Environment
Linux 6.8.0-146-generic, x86_64, Python 3.12.3. GNU stat reported ext2/ext3, a filesystem-magic label that does not distinguish ext-family versions or record mount options. Generated at .
Limits
- One host and filesystem-magic label; not a filesystem comparison.
- Synthetic A/B/C payloads; measured API observations, not storage-fleet statistics.
- No TCP, MSG_ZEROCOPY, crash, power-loss, throughput or cache-eviction test.
- Mutation cases intentionally violate the documented sendfile source-lifetime requirement.
- A different supported path may copy data; post-mutation B/C results are observations, not a universal API guarantee.
Column definitions
- repetition
- Sequential repetition number; not an independent fleet sample.
- case
- Named sequence defined by the Python harness.
- page_bytes
- Requested and received payload length in bytes.
- enqueue_bytes
- Successful file-to-pipe or buffered pipe-write return value in bytes.
- received_byte
- Uniform ASCII byte observed at pipe consumption.
- path_byte
- Uniform ASCII byte read by opening the source pathname after the operation.
- old_inode_byte
- Uniform ASCII byte read through the original writable descriptor.
- path_same_inode
- Whether device and inode identity still match the initial source.
- matches_before
- Whether received SHA-256 equals the hash computed before enqueue.
Sources and reuse
Hesela's original harness, generated observations and derived CSV are dedicated under CC0-1.0. Linked documentation and Linux source retain their own licenses and are not redistributed.
- Linux man-pages 6.19: sendfile(2), return value and source lifetime
- Linux man-pages 6.19: splice(2), pipe buffers and flags
- Linux v6.8 mm/filemap.c: splice_folio_into_pipe and filemap_splice_read
- Linux man-pages 6.19: rename(2), replacement and open descriptors
- Linux man-pages 6.19: fsync(2), synchronization and directory entries
- Linux 6.12 documentation: splice and pipes, copy_splice_read
- Linux kernel documentation: MSG_ZEROCOPY, notifications and deferred copies
- Linux man-pages 6.19: tee(2), duplication without consuming and shared references
- Pai, Druschel and Zwaenepoel: IO-Lite, OSDI 1999, sections 2.1, 2.5, 2.9 and 3
Definitions: zero-copy I/O, sendfile and splice.
Published by Hesela team
Report a correction · Editorial standards · Reuse guidance
Cite this page
Hesela team. Linux zero-copy source lifetime: eight file-to-pipe checks. Hesela. https://hesela.dev/datasets/zero-copy-source-lifetime/