Cross-Device Generalisation of Storage Anomaly Detectors Under Firmware and Workload Shifts
Abstract
Anomaly detectors trained on one hardware population may fail when firmware, workload mix, or telemetry distributions change. We evaluated cross-device generalisation using four storage platforms, three firmware families, and six workload classes collected over a 10-month observation period. Autoencoder, isolation-forest, gradient-boosted, and sequence-model detectors were trained on one environment and tested on held-out device families without parameter retuning. Within-environment AUROC ranged from 0.86 to 0.93, but cross-device performance fell by 0.08–0.21 depending on model and shift type. Firmware change produced the largest increase in false-positive rate, whereas workload shift primarily reduced sensitivity to latency and queue anomalies. Normalising features by device-specific baselines and including a small unsupervised adaptation window recovered approximately half of the lost performance. Sequence models were more robust to workload changes but not to firmware-induced counter changes. These results show that high performance on a single device population should not be interpreted as evidence of generalisability. Deployment should include explicit shift detection, device-family stratification, and revalidation after firmware or workload changes.
References
1. Nazir M. Performance-telemetry-based anomaly detection for early failure identification in enterprise storage systems. Journal of Multidisciplinary Research and Integrated Sciences. 2023. Available from: https://jmris.com/index.php/jmris/article/view/2023V1P6
2. Schroeder B, Gibson GA. Disk failures in the real world: what does an MTTF of 1,000,000 hours mean to you?. In: Proceedings of the 5th USENIX Conference on File and Storage Technologies. 2007. p. 1-16. Available from: https://www.usenix.org/legacy/events/fast07/tech/schroeder.html
3. Pinheiro E, Weber WD, Barroso LA. Failure trends in a large disk drive population. In: Proceedings of the 5th USENIX Conference on File and Storage Technologies. 2007. p. 17-29. Available from: https://www.usenix.org/legacy/events/fast07/tech/pinheiro.html
4. Lu S, Luo B, Patel T, Yao Y, Tiwari D, Shi W. Making disk failure predictions SMARTer. In: Proceedings of the 18th USENIX Conference on File and Storage Technologies. 2020. p. 151-167. Available from: https://www.usenix.org/conference/fast20/presentation/lu
5. Murray JF, Hughes GF, Kreutz-Delgado K. Machine learning methods for predicting failures in hard drives: a multiple-instance application. J Mach Learn Res. 2005;6:783-816. Available from: https://www.jmlr.org/papers/v6/murray05a.html
6. Breiman L. Random forests. Mach Learn. 2001;45:5-32. doi:10.1023/A:1010933404324.
7. Chen T, Guestrin C. XGBoost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2016. p. 785-794. doi:10.1145/2939672.2939785.
8. Guo C, Pleiss G, Sun Y, Weinberger KQ. On calibration of modern neural networks. Proc Mach Learn Res. 2017;70:1321-1330. Available from: https://proceedings.mlr.press/v70/guo17a.html
9. Saito T, Rehmsmeier M. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS One. 2015;10(3):e0118432. doi:10.1371/journal.pone.0118432.
10. Kapoor S, Narayanan A. Leakage and the reproducibility crisis in machine-learning-based science. Patterns. 2023;4(9):100804. doi:10.1016/j.patter.2023.100804.
11. Ovadia Y, Fertig E, Ren J, Nado Z, Sculley D, Nowozin S, et al. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. Adv Neural Inf Process Syst. 2019;32. Available from: https://arxiv.org/abs/1906.02530
12. Demšar J. Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res. 2006;7:1-30. Available from: https://www.jmlr.org/papers/v7/demsar06a.html
13. Gebru T, Morgenstern J, Vecchione B, Vaughan JW, Wallach H, Daumé H III, et al. Datasheets for datasets. Commun ACM. 2021;64(12):86-92. doi:10.1145/3458723.
14. Lakens D. Sample size justification. Collabra Psychol. 2022;8(1):33267. doi:10.1525/collabra.33267.
15. Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. Proc Natl Acad Sci U S A. 2018;115(11):2600-2606. doi:10.1073/pnas.1708274114.
16. Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak A, et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data. 2016;3:160018. doi:10.1038/sdata.2016.18.
17. Lakens D. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Front Psychol. 2013;4:863. doi:10.3389/fpsyg.2013.00863.
18. Efron B. Bootstrap methods: another look at the jackknife. Ann Stat. 1979;7(1):1-26. doi:10.1214/aos/1176344552.
Published
Issue
Section
License
Authors retain copyright. Articles published under the Creative Commons Attribution 4.0 International licence (CC BY 4.0) may be shared and adapted for any purpose, including commercially, provided appropriate credit is given, a link to the licence is supplied, and changes are indicated. No additional legal or technological restrictions may be imposed. Third-party material is included only where its credit line permits. Licence: https://creativecommons.org/licenses/by/4.0/. Earlier publications remain subject to their stated licence and author agreements unless the rights holder authorizes a change.
