Dataset · sector error prediction · ATC 2017

Sector errors predicted a week ahead let a scrubber find them 1.7 to 1.8 times faster on hard disks, using accelerated scrubs 2% of the time

Mahdisoltani et al., USENIX ATC 2017, predicted one week ahead whether a drive would show sector errors, using Backblaze hard disks and Google SSDs. At a 10% false positive rate random forests caught 90% and 95% of error weeks on two hard disk models and 50% to 70% on three SSD models. A simulated scrubber that sped up after a prediction found errors 1.7 to 1.8 times faster on hard disks. Cross-check rows come from NetApp (SIGMETRICS 2007) and Google (FAST 2007).

Download CSV14 rows · 7 columns · CSV

File /data/datasets/sector-error-prediction-scrubbing.csv · JSON metadata · All datasets · Human-readable note on hesela.com

Method

The CSV transcribes figures printed in Mahdisoltani et al. (USENIX ATC 2017) and two cross-check rows from NetApp (SIGMETRICS 2007) and Google (FAST 2007). It does not read chart values. Prediction rates are per one-week interval. The false positive rate is the share of error-free intervals flagged by mistake. The scrubbing rows come from a simulation.

Limits

Two hard disk models were chosen because they are common and have the highest error rates.
The target is a one-week interval with a sector error, not whole-disk failure.
The text read does not say whether the 75/25 train and test split is by time.
The scrubbing result is a simulation with two scrub speeds.
The SSD rows come from drives of Google's own FAST 2016 study and are not independent of it.

Table

14 rows, the same rows as the CSV. Value low and value high are the printed range ends, or a single printed value in the high column. Units are in the unit column. Sources are the pages opened on 2026-10-10.
Measure	Value low	Value high	Unit	Scope	Note	Citation
hard disks with a reallocated sector lowest model	not stated	0.24	percent of drives	HGST HMS5C4040BLE640 and 9426 drives	SMART 5 and Backblaze data.	mahdisoltani-atc17-pdf
hard disks with a reallocated sector Hitachi HDS722020ALA330	not stated	11.84	percent of drives	4774 drives		mahdisoltani-atc17-pdf
hard disks with a reallocated sector Seagate ST3000DM001	not stated	25.15	percent of drives	4707 drives		mahdisoltani-atc17-pdf
SSDs with uncorrectable errors	37.07	65.56	percent of drives	Three Google MLC models and about 10000 drives each		mahdisoltani-atc17-pdf
SSDs with bad blocks	50.35	82.75	percent of drives	Three Google MLC models and about 10000 drives each		mahdisoltani-atc17-pdf
error weeks caught at 10% false positive rate Hitachi	not stated	90	percent of error intervals	Random forest and one week ahead		mahdisoltani-atc17-pdf
error weeks caught at 10% false positive rate Seagate	not stated	95	percent of error intervals	Random forest and one week ahead		mahdisoltani-atc17-pdf
error weeks caught at 2% false positive rate hard disks	70	90	percent of error intervals	Same models		mahdisoltani-atc17-pdf
error weeks caught at 10% false positive rate SSDs	50	70	percent of error intervals	Three Google MLC models and uncorrectable errors		mahdisoltani-atc17-pdf
faster error detection by accelerated scrubber hard disks	1.7	1.8	times faster than a fixed-rate scrubber	Scrub speed doubled after a prediction	Accelerated mode at most 2% of the time. Simulation.	mahdisoltani-atc17-pdf
faster error detection by accelerated scrubber MLC-A and MLC-D	1.4	1.5	times faster than a fixed-rate scrubber	Same setting	Simulation.	mahdisoltani-atc17-pdf
disks with latent sector errors over 32 months	not stated	3.45	percent of disks	NetApp and 1.53 million disks		bairavasundaram-sigmetrics07
latent sector errors found by scrubbing	60		percent of errors	Same population	Printed as more than 60.	bairavasundaram-sigmetrics07
Google hard disks with a reallocation count above zero	not stated	9	percent of drives	More than 100000 ATA disks from December 2005 to August 2006	Printed as about 9.	pinheiro-usenix-html

Columns

measure (string): The quantity as the source names it.
value_low (number): Lower end of a printed range. Empty when the source prints a single value.
value_high (number): Single printed value, or the upper end of a range. For 'more than half' it is the stated bound.
unit (string): Unit of the value columns: disks, blocks, files, percent of disks, percent of mismatches, or probability.
scope (string): Population and window the value applies to.
note (string): What the value is not, or the source wording behind it.
citation_id (string): Id of the opened source in the citations list.

License

Small derived table of figures printed in Mahdisoltani et al. (USENIX ATC 2017), Bairavasundaram et al. (SIGMETRICS 2007), and Pinheiro et al. (FAST 2007), with credit. Not a Creative Commons license. The USENIX proceedings state that rights to individual papers remain with the author or the author's employer. This file is not a copy of the papers and not the data.