Character encoding
A character encoding specifies how text is represented using coded values and their storage or interchange representation.
Layers: Unicode distinguishes code points, encoding forms that map scalar values to code units, and encoding schemes that serialize code units as bytes. UTF-8, UTF-16 and UTF-32 are different representations of Unicode text. A code point, code unit and byte are not interchangeable counting units. [1]
Counting: UTF-8 stores U+0041 in one byte and U+20AC in three. A user-perceived character can contain several code points, so neither byte count nor code-point count is a universal display-character count. Specify the unit when sizing a text field or comparing names. [2] [3]
Recovery: Record the encoding and decoding-error policy with text-bearing metadata. Replacing undecodable input can discard distinctions in the original bytes; a readable display string is not automatically a reversible filename identifier. Keep the original representation when byte identity is required. [4]
Relations
- related
- ASCII, UTF-32, UTF-8, Unicode normalization, Byte, Byte order, Filename collision, Unicode scalar value, Grapheme cluster, ISO-8859-1 (Latin-1), Windows-1252, Comma-separated values (CSV)
Limitations
- An encoding label does not establish Unicode normalization, case-insensitive equality or a filesystem's filename comparison policy.
Citations
- Unicode 17.0.0: chapter 3, sections 3.9 and 3.10, encoding forms and schemes
- RFC 3629: section 3, UTF-8 definition
- Unicode Standard Annex 29 revision 47: grapheme cluster boundaries
- Hesela: filename identity and backup restores
Sources & caveats
- Source-reviewed terminology with constructed text examples, not storage-device measurements.