Dataset · reallocated sectors · FAST 2015
Replacing disks at 200 reallocated sectors removed 88% of triple-disk RAID failures at one vendor
Ma et al., FAST 2015, analysed about 1 million SATA disks in 6 models in one vendor's backup systems (logs of 21 to 60 months from June 2008). Replacing a disk once its reallocated-sector count passed 200 avoided about 88% of triple-disk RAID failures, which were 80% of RAID failures, or about 70% of all disk-related incidents. In simulation, thresholds below 200 caught 52% to 70% of failures within 60 days with 0.8% to 4.5% false positives. Cross-check rows come from Google (FAST 2007) and Backblaze (2016).
Download CSV17 rows · 7 columns · CSVMethod
The CSV transcribes figures printed in Ma et al. (FAST 2015) and two cross-check sets from Google (FAST 2007) and Backblaze (2016). It does not read chart values. Reallocated sectors (RS) are sectors a disk remapped to a spare after a write error. A triple failure is three simultaneous whole-disk or sector failures in one RAID-6 group. The 88% is a vendor's own before and after comparison.
Limits
- The 88% is the vendor's own before and after comparison, normalized to the pre-deployment monthly average. Absolute rates are not published.
- The deployed rule covered 3 of the 6 models. Simulated rates come from one model.
- The paper prints 65% and 2.5% in the introduction and 52% to 70% and 0.8% to 4.5% in the evaluation.
- ARMOR was only simulated.
- The systems are write-heavy backup systems.
- Google and Backblaze rows are different fleets and measures and do not test the 88%.
Table
| Measure | Value low | Value high | Unit | Scope | Note | Citation |
|---|---|---|---|---|---|---|
| triple failures among RAID failures before the rule | not stated | 80 | percent of RAID failures | Vendor backup systems | Another 5% were other hardware faults and 15% user errors and unknown causes. | ma-fast15-pdf |
| triple failures avoided after replacing disks at 200 reallocated sectors | not stated | 88 | percent of triple-failure incidents | Deployed on models A-1 and A-2 and B-1 | Printed as about 88. Normalized to the pre-deployment monthly average. | ma-fast15-pdf |
| all disk-related incidents avoided | not stated | 70 | percent of incidents | Same deployment | Printed as about 70. | ma-fast15-pdf |
| deployed replacement threshold | not stated | 200 | reallocated sectors | Median time to failure fell below 3 days beyond this count | Chosen because replacing a disk can take up to 3 days. | ma-fast15-pdf |
| impending failures caught with thresholds below 200 | 52 | 70 | percent of failures within 60 days | Simulation on 100000 A-2 disks | The introduction prints up to 65% instead. | ma-fast15-pdf |
| false positives with thresholds below 200 | 0.8 | 4.5 | percent of working disks | Simulation on 100000 A-2 disks | The introduction prints up to 2.5% instead. | ma-fast15-pdf |
| failed A-1 disks found in their fourth year | not stated | 63 | percent of failed disks | Model A-1 | Disks fail at similar ages. | ma-fast15-pdf |
| failed A-2 disks found in their fourth year | not stated | 66 | percent of failed disks | Model A-2 | Disks fail at similar ages. | ma-fast15-pdf |
| failed B-1 disks found in their fourth year | not stated | 64 | percent of failed disks | Model B-1 | Disks fail at similar ages. | ma-fast15-pdf |
| growth of sector error counts in the second year | 25 | 300 | percent increase | Disks with at least one sector error; C-2 lowest and A-2 highest | Printed as about 300 for A-2. | ma-fast15-pdf |
| triple failures left after the rule | not stated | 12 | percent of triple failures | Sudden failures or several mildly unreliable disks | Not covered by the single-disk rule. | ma-fast15-pdf |
| vulnerable RAID-6 groups flagged by ARMOR | not stated | 80 | percent of vulnerable groups | Simulation on 5000 healthy and 500 failed groups | Not deployed. | ma-fast15-pdf |
| total triple-failure coverage with ARMOR added | not stated | 98 | percent of triple failures | Simulation only | Not deployed. | ma-fast15-pdf |
| Google drives with a reallocation count above zero | not stated | 9 | percent of drives | More than 100000 ATA disks from December 2005 to August 2006 | Printed as about 9. | pinheiro-usenix-html |
| Google failure likelihood within 60 days after the first reallocation | not stated | 14 | times the rate for drives with none | Same population | Printed as over 14. | pinheiro-usenix-html |
| Backblaze failed drives with one or more of five SMART counts above zero | not stated | 76.7 | percent of failed drives | 67814 drives in 2016 | 23.3% of failed drives showed none. | backblaze-smart-2016 |
| Backblaze operational drives with one or more of five SMART counts above zero | not stated | 4.2 | percent of operational drives | 67814 drives in 2016 | Not a false-positive rate for a replacement rule. | backblaze-smart-2016 |
Columns
- measure
- The quantity as the source names it.
- value_low
- Lower end of a printed range. Empty when the source prints a single value.
- value_high
- Single printed value, or the upper end of a range. For 'more than half' it is the stated bound.
- unit
- Unit of the value columns: disks, blocks, files, percent of disks, percent of mismatches, or probability.
- scope
- Population and window the value applies to.
- note
- What the value is not, or the source wording behind it.
- citation_id
- Id of the opened source in the citations list.
License
Small derived table of figures printed in Ma et al. (FAST 2015), Pinheiro et al. (FAST 2007), and a Backblaze post of 6 October 2016, with credit. Not a Creative Commons license. The USENIX proceedings state that rights to individual papers remain with the author or the author's employer. This file is not a copy of the papers and not the underlying logs.
Sources
- RAIDShield: Characterizing, Monitoring, and Proactively Protecting Against Disk Failures, Ma et al., FAST 2015 (USENIX PDF)
- Failure Trends in a Large Disk Drive Population, Pinheiro, Weber, and Barroso, FAST 2007 (USENIX HTML proceedings)
- What SMART Stats Tell Us About Hard Drives, Backblaze, 6 October 2016