{
  "slug": "content-based-sampling",
  "term": "Content-based sampling (deduplication)",
  "aliases": [
    "Fingerprint-based sampling",
    "Deduplication estimation"
  ],
  "definition": "Content-based sampling selects blocks by a predicate on their content fingerprints so identical blocks share the same sampling decision.",
  "note": "Estimation: Scan every block fingerprint, retain selected distinct identities and scale their total bytes by the inverse selection probability. A small retained sample does not remove the initial full scan. Xie et al. use a modulo filter and adapt its selectivity as the sample grows. [xie2013]\n\nUnits: For a fixed divisor M, unique payload is estimated as M times the selected distinct bytes. Variable-size blocks require byte weighting. The paper calls unique bytes divided by original bytes a deduplication ratio; a reduction factor uses the reciprocal, and saved fraction subtracts that share from one. [xie2013]\n\nUncertainty: Accuracy relies on fingerprint uniformity, negligible collisions and enough selected distinct content. Taking a reciprocal of an unbiased unique-byte estimate does not make the reduction-factor estimate unbiased. A zero sample cannot establish that no unique data exists. [xie2013] [example]\n\nWorked example: For six equal blocks A,A,A,A,B,C, averaging the within-sample saved fraction over every two-position sample gives 20%, while the full population saves 50%. This demonstrates a naive extrapolation error, not a measured backup workload. [example]\n\nPosition sampling has another limit: different full populations can yield identical probability laws for a small sampled frequency histogram, despite different unique-byte totals. This concerns inference from sampled positions, not the full fingerprint scan used above. Hesela provides an exact two-position example and a three-position control. [identifiability]",
  "related": [
    "deduplication-ratio",
    "sparse-indexing",
    "sampling-design",
    "duplication-frequency-histogram"
  ],
  "citations": [
    {
      "id": "xie2013",
      "title": "Xie, Condict and Shete. Estimating Duplication by Content-based Sampling. USENIX ATC 2013, pp. 181-186.",
      "url": "https://www.usenix.org/system/files/conference/atc13/atc13-xie.pdf",
      "accessed": "2026-11-22"
    },
    {
      "id": "example",
      "title": "Hesela: exact occurrence-sampling and fixed-probability content-sampling examples, method and tests.",
      "url": "https://hesela.dev/research/dedup-estimation/README.md",
      "accessed": "2026-11-22"
    },
    {
      "id": "identifiability",
      "title": "Hesela: identical sample histogram laws, different deduplication savings; exact derivation and executable controls.",
      "url": "https://hesela.dev/research/dedup-estimation/IDENTIFIABILITY.md",
      "accessed": "2026-11-22"
    }
  ],
  "status": "research",
  "advantages": [],
  "limitations": [],
  "supersedes": [],
  "supersededBy": [],
  "firstSeen": null,
  "lastSeen": null,
  "updated": "2026-11-22",
  "provenance": [
    "Primary-paper source review; the online system and reported performance were not reproduced. [xie2013]",
    "The exact example uses independent identity selection at fixed probability 1/2, not the paper's adaptive algorithm, a hash implementation or a confidence-interval estimator. [example]"
  ]
}
