{
  "slug": "sector-error-prediction-scrubbing",
  "title": "Sector errors predicted a week ahead let a scrubber find them 1.7 to 1.8 times faster on hard disks, using accelerated scrubs 2% of the time",
  "kind": "dataset",
  "summary": "Mahdisoltani et al., USENIX ATC 2017, predicted one week ahead whether a drive would show sector errors, using Backblaze hard disks and Google SSDs. At a 10% false positive rate random forests caught 90% and 95% of error weeks on two hard disk models and 50% to 70% on three SSD models. A simulated scrubber that sped up after a prediction found errors 1.7 to 1.8 times faster on hard disks. Cross-check rows come from NetApp (SIGMETRICS 2007) and Google (FAST 2007).",
  "license": "Small derived table of figures printed in Mahdisoltani et al. (USENIX ATC 2017), Bairavasundaram et al. (SIGMETRICS 2007), and Pinheiro et al. (FAST 2007), with credit. Not a Creative Commons license. The USENIX proceedings state that rights to individual papers remain with the author or the author's employer. This file is not a copy of the papers and not the data.",
  "licenseUrl": "https://www.usenix.org/system/files/conference/atc17/atc17-mahdisoltani.pdf",
  "sourceUrl": "https://www.usenix.org/system/files/conference/atc17/atc17-mahdisoltani.pdf",
  "accessed": "2026-10-10",
  "rowCount": 14,
  "columns": [
    {
      "name": "measure",
      "type": "string",
      "description": "The quantity as the source names it."
    },
    {
      "name": "value_low",
      "type": "number",
      "description": "Lower end of a printed range. Empty when the source prints a single value."
    },
    {
      "name": "value_high",
      "type": "number",
      "description": "Single printed value, or the upper end of a range. For 'more than half' it is the stated bound."
    },
    {
      "name": "unit",
      "type": "string",
      "description": "Unit of the value columns: disks, blocks, files, percent of disks, percent of mismatches, or probability."
    },
    {
      "name": "scope",
      "type": "string",
      "description": "Population and window the value applies to."
    },
    {
      "name": "note",
      "type": "string",
      "description": "What the value is not, or the source wording behind it."
    },
    {
      "name": "citation_id",
      "type": "string",
      "description": "Id of the opened source in the citations list."
    }
  ],
  "downloadPath": "/data/datasets/sector-error-prediction-scrubbing.csv",
  "humanPage": "https://hesela.com/analyses/sector-error-prediction-scrubbing/",
  "limits": [
    "Two hard disk models were chosen because they are common and have the highest error rates.",
    "The target is a one-week interval with a sector error, not whole-disk failure.",
    "The text read does not say whether the 75/25 train and test split is by time.",
    "The scrubbing result is a simulation with two scrub speeds.",
    "The SSD rows come from drives of Google's own FAST 2016 study and are not independent of it."
  ],
  "citations": [
    {
      "id": "mahdisoltani-atc17-pdf",
      "title": "Improving Storage System Reliability with Proactive Error Prediction, Mahdisoltani, Stefanovici, and Schroeder, USENIX ATC 2017 (USENIX PDF)",
      "url": "https://www.usenix.org/system/files/conference/atc17/atc17-mahdisoltani.pdf",
      "accessed": "2026-10-10"
    },
    {
      "id": "bairavasundaram-sigmetrics07",
      "title": "An Analysis of Latent Sector Errors in Disk Drives, Bairavasundaram, Goodson, Pasupathy, and Schindler, SIGMETRICS 2007 (University of Wisconsin ADSL PDF)",
      "url": "https://research.cs.wisc.edu/adsl/Publications/latent-sigmetrics07.pdf",
      "accessed": "2026-10-10"
    },
    {
      "id": "pinheiro-usenix-html",
      "title": "Failure Trends in a Large Disk Drive Population, Pinheiro, Weber, and Barroso, FAST 2007 (USENIX HTML proceedings)",
      "url": "https://www.usenix.org/legacy/events/fast07/tech/full_papers/pinheiro/pinheiro_html/index.html",
      "accessed": "2026-10-09"
    }
  ]
}