UTF-8
UTF-8 is a Unicode encoding that represents each Unicode scalar value using one to four octets while preserving ASCII byte values.
Validity: Unicode scalar values exclude the surrogate range U+D800 to U+DFFF. Well-formed UTF-8 also excludes overlong encodings and values beyond U+10FFFF. A strict decoder must reject malformed sequences rather than interpret them as valid scalar values. [1]
Size: U+0041 is hexadecimal 41, U+20AC is E2 82 AC, and U+1F600 is F0 9F 98 80. The last example is one scalar value but four UTF-8 bytes and two UTF-16 code units. Count the actual encoded bytes for byte-limited storage fields. [1] [2]
Identity: UTF-8 has no little-endian or big-endian variant. It does not normalize text: U+00E9 and U+0065 U+0301 have different UTF-8 bytes despite canonical equivalence. Specify normalization separately, and preserve original bytes when exact recovery requires them. [3] [4]
Relations
- related
- ASCII, Character encoding, Unicode normalization, Byte, Byte order, Filename collision, Unicode scalar value
Limitations
- Valid UTF-8 alone does not establish byte identity, canonical normalization or filename equivalence.
Citations
- RFC 3629: sections 3 and 4, UTF-8 definition and byte syntax
- Unicode 17.0.0: chapter 3, section 3.9, Unicode encoding forms
- RFC 3629: section 6, byte order mark
- Unicode Standard Annex 15 revision 57: canonical and compatibility equivalence
Further reading
Sources & caveats
- Source-reviewed UTF-8 definition; bounded executable examples: https://hesela.dev/research/text-encoding.mjs and tests: https://hesela.dev/research/text-encoding.test.mjs.