{
  "slug": "hyperloglog",
  "term": "HyperLogLog",
  "aliases": [
    "HLL"
  ],
  "definition": "HyperLogLog is a compact probabilistic sketch that estimates the number of distinct values from maxima of hash-derived register ranks.",
  "note": "A hash prefix selects one of m registers; remaining bits determine a rank. Repeats leave maxima unchanged. Under independent uniform hash bits, the classic asymptotic relative standard error is about 1.04/sqrt(m): 16384 registers give 0.81%, not a worst-case bound. [flajolet2007]\n\nDirect register-wise maximum constructs the union sketch when both use the same precision, rank range, hash function and seed, input encoding, index/rank conventions and register order. The count remains estimated. HLL++ adds 64-bit hashes, bias correction and sparse storage with different small-cardinality behavior. [heule2013] [ertl2017]\n\nA register-only sketch cannot generally undo an insertion. Distinct assigned hashes 0001 and 0010 select one register with ranks 2 and 1. Sets {a} and {a,b} both store [2,0,0,0], but deleting a requires [0,0,0,0] or [1,0,0,0]. No full-hash collision is involved. [register-example]\n\nNo-op insertion does not prove membership: adding b to {a} leaves the registers unchanged although b was absent. This cannot justify discarding a stored chunk. [register-example] PFADD returning 0 reports no changed registers, not confirmed duplicate input. [redis-pfadd]\n\nRegister-wise minimum does not generally construct an intersection. Joint estimation can estimate overlap size, not recover its items. For A minus B, subtract B's count from the union count; subtracting B's count from A's requires that B is a subset of A. Estimated differences can have large relative errors when the result is small. [ertl2017]",
  "related": [
    "bloom-filter",
    "deduplication"
  ],
  "status": "shipping",
  "advantages": [],
  "limitations": [
    {
      "text": "Redis PFMERGE includes an existing destination. In a pinned Redis 7.2.4 replay, reusing a rollup yields 120 for a requested 20-identity active union; a fresh destination yields 20. Expiring a source does not retract its contribution from the rollup.",
      "citationIds": [
        "redis-pfmerge",
        "native-window"
      ]
    },
    {
      "text": "A missing still-required source is accepted as empty: the same constructed 20-identity window then reports 10. Keep a complete manifest of sealed buckets and distinguish missing from intentionally empty input. These are population-selection failures, not measured estimator error rates.",
      "citationIds": [
        "redis-source",
        "native-window"
      ]
    },
    {
      "text": "The native replay adds a distinct 184th identity while PFADD returns 0, the estimate stays 183 and cached bytes remain identical. The example replacement checks source presence and builds into a fresh key on one server; it does not establish ingestion completeness, cross-server consistency or durability, and makes the destination persistent.",
      "citationIds": [
        "native-window"
      ]
    }
  ],
  "supersedes": [],
  "supersededBy": [],
  "firstSeen": null,
  "lastSeen": null,
  "updated": "2026-11-20",
  "citations": [
    {
      "id": "flajolet2007",
      "title": "Flajolet, Fusy, Gandouet and Meunier. HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm. AofA 2007, sections 1 and 4.",
      "url": "https://monge.univ-eiffel.fr/~fusy/Articles/FlFuGaMe07.pdf",
      "accessed": "2026-10-29"
    },
    {
      "id": "heule2013",
      "title": "Heule, Nunkesser and Hall. HyperLogLog in Practice. EDBT 2013, sections 4-6 and Figure 1.",
      "url": "https://stefanheule.com/papers/edbt13-hyperloglog.pdf",
      "accessed": "2026-10-29"
    },
    {
      "id": "ertl2017",
      "title": "Ertl. New Cardinality Estimation Methods for HyperLogLog Sketches. arXiv:1706.07290v1, sections 2 and 6.",
      "url": "https://arxiv.org/pdf/1706.07290v1",
      "accessed": "2026-10-29"
    },
    {
      "id": "register-example",
      "title": "Hesela: exact register-state example with distinct assigned hashes, executable derivation and tests.",
      "url": "https://hesela.dev/research/hyperloglog-registers.json",
      "accessed": "2026-10-29"
    },
    {
      "id": "redis-pfadd",
      "title": "Redis command reference: PFADD return information and the no-element creation exception.",
      "url": "https://redis.io/docs/latest/commands/pfadd/",
      "accessed": "2026-10-29"
    },
    {
      "id": "redis-pfmerge",
      "title": "Redis PFMERGE reference: an existing destination is an input to the union.",
      "url": "https://redis.io/docs/latest/commands/pfmerge/",
      "accessed": "2026-11-20"
    },
    {
      "id": "redis-source",
      "title": "Redis 7.2.4, pinned hyperloglog.c: pfaddCommand, pfcountCommand and pfmergeCommand.",
      "url": "https://github.com/redis/redis/blob/d2c8a4b91e8c0e6aefd1f5bc0bf582cddbe046b7/src/hyperloglog.c#L1324",
      "accessed": "2026-11-20"
    },
    {
      "id": "native-window",
      "title": "Hesela: native Redis HLL window controls, sparse/dense results, guarded-rebuild example and exact replay package.",
      "url": "https://hesela.dev/research/hll-window/README.md",
      "accessed": "2026-11-20"
    }
  ],
  "provenance": [
    "The three papers were read beyond their abstracts on 2026-10-29; the standard-error statement is asymptotic and assumes the stated hash-randomness model.",
    "The four-bit example is a constructed information-loss witness, not an observed error rate or a reproduction of a paper's experiments.",
    "The deletion result concerns only the register state. Sliding-window or deletion-aware designs can retain additional information.",
    "HLL estimates a count, not variable-sized unique payload bytes, physical capacity savings or safe garbage-collection candidates.",
    "The PFADD contract was read in the official command reference; the four-bit example does not execute Redis or reproduce its hash mapping.",
    "The 2026-11-20 native controls use Redis 7.2.4 commit d2c8a4b91e8c0e6aefd1f5bc0bf582cddbe046b7, isolated Unix sockets and deterministic byte identities. They are not field measurements, an accuracy benchmark, a new Redis defect or a recommendation to deploy that historical version. [native-window]"
  ]
}
