Human Evaluation of Explanations for Storage-Failure Alerts: A Randomised Study of Operator Decisions and Intervention Timing
Abstract
Explanations accompanying predictive-maintenance alerts are intended to help operators prioritise action, yet their effect on real decisions has received limited experimental study. In a randomised online experiment, 126 infrastructure engineers reviewed 24 simulated storage-failure cases presented with either a risk score alone, feature-attribution explanations, or a concise rule-based rationale. Primary outcomes were escalation accuracy, decision time, intervention timing, and confidence calibration. Both explanation formats improved escalation accuracy compared with risk scores alone, with the largest gain observed for borderline-risk cases. Feature-attribution explanations reduced median decision time by 11%, whereas rule-based rationales produced slightly higher agreement between operators. Overconfident responses decreased when explanations explicitly highlighted uncertainty and conflicting signals. However, misleading attributions intentionally inserted into a subset of cases increased inappropriate escalation, demonstrating that plausible explanations can amplify model error. Operators reported the greatest trust when explanations were short, action-oriented, and linked to recent telemetry trends. Explainability can support earlier and more consistent intervention, but explanation quality must be independently validated because persuasive visualisations may otherwise create unwarranted confidence.
References
1. Nazir M. Performance-telemetry-based anomaly detection for early failure identification in enterprise storage systems. Journal of Multidisciplinary Research and Integrated Sciences. 2023. Available from: https://jmris.com/index.php/jmris/article/view/2023V1P6
2. Schroeder B, Gibson GA. Disk failures in the real world: what does an MTTF of 1,000,000 hours mean to you?. In: Proceedings of the 5th USENIX Conference on File and Storage Technologies. 2007. p. 1-16. Available from: https://www.usenix.org/legacy/events/fast07/tech/schroeder.html
3. Lu S, Luo B, Patel T, Yao Y, Tiwari D, Shi W. Making disk failure predictions SMARTer. In: Proceedings of the 18th USENIX Conference on File and Storage Technologies. 2020. p. 151-167. Available from: https://www.usenix.org/conference/fast20/presentation/lu
4. Lundberg SM, Lee SI. A unified approach to interpreting model predictions. Adv Neural Inf Process Syst. 2017;30. Available from: https://arxiv.org/abs/1705.07874
5. Ribeiro MT, Singh S, Guestrin C. "Why should I trust you?": explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2016. p. 1135-1144. doi:10.1145/2939672.2939778.
6. Guo C, Pleiss G, Sun Y, Weinberger KQ. On calibration of modern neural networks. Proc Mach Learn Res. 2017;70:1321-1330. Available from: https://proceedings.mlr.press/v70/guo17a.html
7. Jacovi A, Goldberg Y. Towards faithfully interpretable NLP systems: how should we define and evaluate faithfulness?. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020. p. 4198-4205. doi:10.18653/v1/2020.acl-main.386.
8. Sweller J. Cognitive load during problem solving: effects on learning. Cogn Sci. 1988;12(2):257-285. doi:10.1207/s15516709cog1202_4.
9. Salas E, Tannenbaum SI, Kraiger K, Smith-Jentsch KA. The science of training and development in organizations: what matters in practice. Psychol Sci Public Interest. 2012;13(2):74-101. doi:10.1177/1529100612436661.
10. Hart SG, Staveland LE. Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. Adv Psychol. 1988;52:139-183. doi:10.1016/S0166-4115(08)62386-9.
11. Baldwin TT, Ford JK. Transfer of training: a review and directions for future research. Pers Psychol. 1988;41(1):63-105. doi:10.1111/j.1744-6570.1988.tb00632.x.
12. Koo TK, Li MY. A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15(2):155-163. doi:10.1016/j.jcm.2016.02.012.
13. Kottner J, Audigé L, Brorson S, Donner A, Gajewski BJ, Hróbjartsson A, et al. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. J Clin Epidemiol. 2011;64(1):96-106. doi:10.1016/j.jclinepi.2010.03.002.
14. Lakens D. Sample size justification. Collabra Psychol. 2022;8(1):33267. doi:10.1525/collabra.33267.
15. Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. Proc Natl Acad Sci U S A. 2018;115(11):2600-2606. doi:10.1073/pnas.1708274114.
16. Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak A, et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data. 2016;3:160018. doi:10.1038/sdata.2016.18.
17. Lakens D. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Front Psychol. 2013;4:863. doi:10.3389/fpsyg.2013.00863.
18. Efron B. Bootstrap methods: another look at the jackknife. Ann Stat. 1979;7(1):1-26. doi:10.1214/aos/1176344552.
Published
Issue
Section
License
Authors retain copyright. Articles published under the Creative Commons Attribution 4.0 International licence (CC BY 4.0) may be shared and adapted for any purpose, including commercially, provided appropriate credit is given, a link to the licence is supplied, and changes are indicated. No additional legal or technological restrictions may be imposed. Third-party material is included only where its credit line permits. Licence: https://creativecommons.org/licenses/by/4.0/. Earlier publications remain subject to their stated licence and author agreements unless the rights holder authorizes a change.
