UTF-32
UTF-32 represents each Unicode scalar value using one 32-bit code unit, with a specified byte order when serialized.
Validity: Surrogate values U+D800 through U+DFFF and values above U+10FFFF are invalid UTF-32 units. Unassigned scalar values and noncharacters are not thereby malformed. [1]
Serialization: UTF-32BE and UTF-32LE fix byte order. Generic UTF-32 uses an initial byte-order mark when present and defaults to big-endian without one unless a higher-level protocol specifies otherwise. Under the explicit-endian schemes, an initial U+FEFF is content, not an automatically removed signature. [1]
Counting: Four bytes represent one scalar, not necessarily one displayed character. Storage size relative to UTF-8 depends on the text. Transcoding does not normalize equivalent spellings or establish filename equality. [2] [3]
Recovery: The constructed bytes 00 00 01 00 decode as U+0100 in big-endian and U+10000 in little-endian. Both succeed; decoder acceptance alone cannot recover the intended byte order. Preserve the encoding contract with stored names. [4]
Decoder choice: In CPython 3.12.3, BOM-less bytes.decode('utf-32') uses native byte order, while the generic incremental decoder rejects a complete BOM-less unit. An encoding label alone does not select an API's BOM policy. [5]
Relations
- related
- Character encoding, Unicode scalar value, Byte order, UTF-8, Grapheme cluster, Unicode normalization
Limitations
- Fixed-width scalar units do not provide fixed-width grapheme clusters or validate a filesystem's filename rules.
Citations
- Unicode 17.0: chapter 3, D90 and D99-D101, UTF-32 forms and schemes
- Unicode FAQ: UTF-8, UTF-16, UTF-32 and BOM
- Unicode UAX 15 revision 57: canonical and compatibility normalization
- Hesela: UTF-32 byte order and filename identity
- CPython 3.12.3: UTF-32 stateless and incremental decoder implementation
Sources & caveats
- Source-reviewed terminology and constructed strict-codec examples, not filesystem observations.
- Executable examples and tests: https://hesela.dev/research/utf32-contracts/probe.py and https://hesela.dev/research/utf32-contracts/test_probe.py.