Bit packing
Bit packing stores values or fields in a contiguous sequence of bits without requiring a separate whole byte or machine word for each value.
Width: For unsigned values 0 through 7, three bits per value suffice: eight values occupy 24 payload bits. Headers, padding and alignment are separate. Arrow packs Boolean values and validity flags into bits while specifying buffer alignment independently. [2]
Ordering: A packing format must define bit order as well as value width. For 0 through 7 at width 3, Parquet's hybrid payload is hexadecimal 88 c6 fa; deprecated BIT_PACKED produces 05 39 77. These are different wire layouts for the same values. [1]
Context: The hybrid format counts packed runs in groups of eight and assumes the width is known. An encoded byte count alone does not identify the number, signedness or meaning of decoded values. [1]
Relations
- related
- Run-length encoding, Lossless compression, Apache Parquet, Frame-of-reference encoding, Patched frame-of-reference
Citations
- Apache Parquet: RLE/bit-packing hybrid and deprecated BIT_PACKED encodings
- Apache Arrow: Boolean, validity bitmap and run-end encoded layouts
Sources & caveats
- Source synthesis and exact unsigned-integer examples. The executable example implements bounded bit arithmetic, not a complete Parquet reader.