Temporal Leakage in Storage-Failure Prediction: A Reproducibility Study of Random and Time-Ordered Evaluation Strategies

Abstract

Random train-test splits can overestimate predictive performance when storage telemetry contains temporal structure and repeated measurements from the same devices. We reproduced a storage-failure prediction pipeline under random, device-grouped, and time-ordered evaluation strategies using 14 months of telemetry from 9,842 devices. Models used SMART attributes, error counters, workload indicators, and recent trend features. Random splitting produced the highest apparent discrimination, with an area under the receiver operating characteristic curve of 0.94, compared with 0.86 under device-grouped validation and 0.79 under strict time-ordered testing. Warning precision at a fixed recall target fell from 0.71 to 0.43 when evaluation moved from random to temporal splitting. Feature analysis showed that several highly ranked variables encoded maintenance-cycle and device-age patterns that were unstable over time. Retraining on rolling windows partially recovered performance but did not eliminate the gap. These findings demonstrate that random evaluation can substantially inflate estimates of deployable performance in predictive maintenance. Time-aware validation, device-level separation, and prospective monitoring should be considered minimum requirements when reporting storage-failure prediction models intended for operational use.