Dataset · FAST 2016 · FAST 2018 · FAST 2023

In 87 days of NetApp RAID data, a disk was 2x slower than its peers in 0.22% of drive hours, but a RAID group had a slow drive in 1.54% of hours

NetApp field data (Hao et al., FAST 2016): disks and SSDs were 2x slower than their RAID group median for 0.22% and 0.58% of drive hours, but groups had a slow drive for 1.54% and 2.23% of hours. Dean and Barroso (CACM 2013) give the fan-out arithmetic; Gunawi et al. (FAST 2018) and Lu et al. (FAST 2023) are cross-checks from other fleets.

Download CSV24 rows · 7 columns · CSV

File /data/datasets/storage-tail-slowdowns-raid-fanout.csv · JSON metadata · All datasets · Human-readable note on hesela.com

Method

The figures are copied from the USENIX PDFs of Hao et al. (FAST 2016), Gunawi et al. (FAST 2018), and Lu et al. (FAST 2023), and from the author PDF of Dean and Barroso (Communications of the ACM, February 2013); no chart was read. Hao et al. compare each drive's hourly average latency to the median of its RAID group and call an hour slow when the ratio is at least 2; a tail hour is a RAID hour with at least one slow drive. Rows marked derived by us are our division of printed counts or our recomputation of the printed formula 1 minus (1 minus p) to the power n. The four sources come from different organizations and fleets: NetApp customers, Google, a set of operator reports, and Alibaba. Hao et al. and Gunawi et al. share an author, so they are not fully independent of each other. Sources opened for this table: The Tail at Store: A Revelation from Millions of Hours of Disk and SSD Deployments, Hao, Soundararajan, Kenchammana-Hosekote, Chien, and Gunawi, USENIX FAST 2016 (USENIX PDF, abstract and sections 1 and 4); The Tail at Scale, Dean and Barroso, Communications of the ACM, February 2013 (author PDF); Fail-Slow at Scale: Evidence of Hardware Performance Faults in Large Production Systems, Gunawi et al., USENIX FAST 2018 (USENIX PDF, sections 1 and 3); Perseus: A Fail-Slow Detection Framework for Cloud Storage Systems, Lu et al., USENIX FAST 2023 (USENIX PDF, abstract).

Limits

The hourly slow rates come from one NetApp study; other sources only cross-check the direction.
Hourly latency against the group median assumes balanced load and fixed 1.5x and 2x thresholds.
Slow drives per tail hour is our division of two table counts.
The fan-out percentages are arithmetic for independent servers, not storage measurements.
The Alibaba rate uses a different definition of slow and is not comparable with the NetApp hours.

Table

24 rows, the same rows as the CSV. Value low and value high are the printed range ends, or a single printed value in the high column. Units are in the unit column. Sources are the pages opened on 2026-10-12.
Measure	Value low	Value high	Unit	Scope	Note	Citation
Data disks studied	not stated	458482	drives	NetApp field RAID groups, 87 days on average	Table 1 of the paper; 38,029 disk RAID groups.	hao-fast16-pdf
Data SSDs studied	not stated	4069	drives	Same study	572 SSD RAID groups.	hao-fast16-pdf
Disk drive hours observed	not stated	857183442	drive hours	Same study	The abstract rounds this to 857 million.	hao-fast16-pdf
SSD drive hours observed	not stated	7481055	drive hours	Same study	The abstract rounds this to 7 million.	hao-fast16-pdf
Disk hours at least 2x slower than the RAID group median	not stated	0.22	percent of drive hours	Disks, slowdown ratio of 2 or more	Hourly average latency against the median of the group.	hao-fast16-pdf
SSD hours at least 2x slower than the RAID group median	not stated	0.58	percent of drive hours	SSDs, same rule		hao-fast16-pdf
Slow hours with a 1.5x threshold	0.69	1.27	percent of drive hours	Disks (0.69) and SSDs (1.27)	A tighter threshold than the 2x used elsewhere.	hao-fast16-pdf
RAID hours with at least one slow disk	not stated	1.54	percent of RAID hours	Disk RAID groups, 2x threshold	Called tail hours by the authors.	hao-fast16-pdf
RAID hours with at least one slow SSD	not stated	2.23	percent of RAID hours	SSD RAID groups, 2x threshold		hao-fast16-pdf
RAID hours with a slow drive, 1.5x threshold	4.6	4.8	percent of RAID hours	Disks (4.6) and SSDs (4.8)		hao-fast16-pdf
Slow disks per RAID hour with a slow disk	not stated	1.70	slow drives per tail hour	Derived by us: 1,885,804 slow disk hours divided by 1,109,514 tail hours	Both counts are in Table 1; more than one slow drive tends to appear together.	hao-fast16-pdf
Slow SSDs per RAID hour with a slow SSD	not stated	1.80	slow drives per tail hour	Derived by us: 43,016 divided by 23,964	Same method for SSDs.	hao-fast16-pdf
Slow drives that stay slow for more than an hour	35	40	percent of slow drives	SSDs (35) and disks (40)		hao-fast16-pdf
Drives with at least one slowdown	26	29	percent of drives	Disks (26) and SSDs (29)	Over the study window, not a yearly rate.	hao-fast16-pdf
Disk tail hours if the longest tail were cut	not stated	0.6	percent of RAID hours	Disk RAID groups, tail-tolerant RAID estimate	Down from 1.5; 0.3 if the two longest were cut. Assumes slowdowns are independent.	hao-fast16-pdf
Requests slow when each server is slow 1 time in 100, fan-out 100	not stated	63	percent of requests	Idealized service, Dean and Barroso	Equals 1 minus 0.99 to the power 100, which we recomputed as 63.4.	dean-barroso-cacm13-pdf
Requests slow when each server is slow 1 time in 10,000, 2,000 servers	not stated	18	percent of requests	Idealized service, Dean and Barroso	The article says almost one in five; we recomputed 18.1.	dean-barroso-cacm13-pdf
99th percentile latency of one random leaf request	not stated	10	milliseconds	A large fan-out Google service, measured at the root		dean-barroso-cacm13-pdf
99th percentile latency until all leaf requests finish	not stated	140	milliseconds	Same service	Waiting for the slowest 5% accounts for about half of this, per the article.	dean-barroso-cacm13-pdf
SSD read latency increase from garbage collection	not stated	100	times	Solid-state storage with modest write activity	Stated in the article as a factor of 100 without a dataset.	dean-barroso-cacm13-pdf
Cut in 99.9th percentile read latency from tied requests	not stated	38	percent	BigTable benchmark on a mostly idle cluster, tied request after 1 ms	Reads sent to a second replica; 32% with a concurrent sort job.	dean-barroso-cacm13-pdf
Fail-slow hardware incident reports studied	not stated	101	reports	Gunawi et al., collected from large-scale clusters	A convenience sample, not a rate.	gunawi-fast18-pdf
Incidents that took months to pinpoint	not stated	17	percent of incidents	Same reports; 45% have an unknown time		gunawi-fast18-pdf
Fail-slow drives found among drives monitored	not stated	0.12	percent of drives	Derived by us: 304 of 248,000 drives in 10 months at Alibaba (Lu et al.)	A sustained fail-slow detector, not an hourly threshold, so not comparable with the NetApp hours.	lu-fast23-pdf

Columns

measure (string): The quantity as the source names it, or our derived quantity.
value_low (number): Lower end of a printed range. Empty when the source prints a single value.
value_high (number): Single printed value, or the upper end of a range.
unit (string): Unit of the value columns: drives, drive hours, percent, milliseconds, times, reports.
scope (string): Population and window the value applies to; 'Derived by us' marks our arithmetic.
note (string): What the value is not, or the source wording behind it.
citation_id (string): Id of the opened source in the citations list.

License

Small derived table of figures printed in Hao et al. (FAST 2016), Dean and Barroso (CACM 2013), Gunawi et al. (FAST 2018), and Lu et al. (FAST 2023), plus our labelled arithmetic, with credit. Not a Creative Commons license. The papers remain with their authors or publishers. This file is not a copy of the papers and not the data.