{
  "status": "shipping",
  "advantages": [],
  "limitations": [],
  "supersedes": [],
  "supersededBy": [],
  "firstSeen": null,
  "lastSeen": null,
  "updated": "2026-11-19",
  "slug": "entropy-coding",
  "term": "Entropy coding",
  "aliases": [
    "Entropy encoding"
  ],
  "definition": "Entropy coding is a family of lossless coding methods that use a probability model to reduce the expected encoded length of symbols or sequences.",
  "note": "Expectation: Shannon's source-coding limit concerns an average for a specified stochastic source, not a promise that every file becomes shorter. A byte-frequency histogram omits order and higher-order dependencies, so its entropy is not a universal minimum file size. [shannon1948]\n\nModel: Huffman coding, arithmetic coding and asymmetric numeral systems are different constructions. Their decoding rules and probability models must agree with the encoder; the family name does not identify an interchangeable bitstream. [witten1987] [duda-ans]\n\nAccounting: Model descriptions, stopping rules, byte packing and framing can outweigh payload savings on a small object. The linked exact worksheet reports payload bits and packed bytes separately and excludes model and framing costs; it is not a production compressor or benchmark. [witten1987] [hesela-arithmetic]\n\nDenominator: Tunstall coding assigns fixed-width indices to variable-length phrases. Its long-stream payload rate divides bits per phrase by expected phrase length; averaging each phrase's bits-per-symbol ratio uses a different weighting. The worksheet makes this distinction with 128 bits for 148 symbols. [tunstall2006] [hesela-arithmetic]",
  "related": [
    "arithmetic-coding",
    "huffman-coding",
    "asymmetric-numeral-systems",
    "lossless-compression",
    "tunstall-coding"
  ],
  "citations": [
    {
      "id": "shannon1948",
      "title": "Claude E. Shannon: A Mathematical Theory of Communication (1948), sections 6 and 9, Theorem 9",
      "url": "https://people.math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf",
      "accessed": "2026-11-18"
    },
    {
      "id": "witten1987",
      "title": "Ian H. Witten, Radford M. Neal and John G. Cleary: Arithmetic Coding for Data Compression, CACM 30(6), 520-540 (1987), DOI 10.1145/214762.214771",
      "url": "https://web.stanford.edu/class/ee398a/handouts/papers/WittenACM87ArithmCoding.pdf",
      "accessed": "2026-11-18"
    },
    {
      "id": "duda-ans",
      "title": "Jarek Duda: Asymmetric numeral systems, arXiv:1311.2540v2 (2014), sections 1-2",
      "url": "https://arxiv.org/abs/1311.2540v2",
      "accessed": "2026-11-18"
    },
    {
      "id": "hesela-arithmetic",
      "title": "Hesela: exact rational interval worksheet, synthetic two-symbol messages",
      "url": "https://hesela.dev/research/arithmetic-intervals/results.json",
      "accessed": "2026-11-18"
    },
    {
      "id": "tunstall2006",
      "title": "Drmota, Reznik, Savari and Szpankowski: Precise Asymptotic Analysis of the Tunstall Code, ISIT 2006, sections I-II",
      "url": "https://www.cs.purdue.edu/homes/spa/papers/isit06-tunstall.pdf",
      "accessed": "2026-11-18"
    }
  ],
  "provenance": [
    "Primary-source synthesis and original exact-rational teaching arithmetic; no empirical compression ratio or universal file-size bound."
  ]
}
