Stream VByte
Stream VByte is a byte-oriented integer codec that stores control bytes separately from the encoded integer data.
Layout: Each control byte describes the widths of up to four unsigned 32-bit values. Separating controls removes the dependency of locating the next control after a variable-sized data block. This changes the format rather than transparently accelerating an existing VByte stream. [1]
API: The ordinary codec uses one to four data bytes per integer. The caller supplies the integer count. The padded decoder may read 16 extra bytes; decode_unpadded avoids that padding requirement but still requires valid input. The 0124 and differential formats have separate contracts. [2]
Validation: validate_stream checks encoded size against count and control widths. It does not authenticate integer values. A same-length payload change can pass this check while changing a decoded value. [3]
Relations
Citations
- Lemire, Kurz and Rupp: Stream VByte, Information Processing Letters 130 (2018), sections 3-5
- StreamVByte public API, pinned revision 089549994faf
- StreamVByte size validation, pinned revision 089549994faf
Sources & caveats
- Paper review and a bounded native codec probe, not a reproduction of the paper's performance benchmark.
- The API details apply to revision 089549994faf3657c13e62f6a1a7b2f544308922.
- Probe and observed results: https://hesela.dev/research/stream-vbyte-probe.py and https://hesela.dev/research/stream-vbyte-probe.json.