Measuring Storage Automation Quality: A Systematic Mapping Review of Repeatability, Failure Handling, and Operational Evidence
Abstract
Evidence used to justify storage automation commonly emphasises deployment speed, while repeatability, failure handling, and long-term operational quality are reported inconsistently. We conducted a systematic mapping review of empirical studies published between 2014 and 2024 that evaluated automated storage or infrastructure provisioning in enterprise, cloud, or virtualised environments. Searches across major engineering and computing databases identified 1,846 records, of which 93 met predefined inclusion criteria. Studies were coded for automation technology, experimental setting, workload type, comparator, outcome definition, fault model, and reproducibility artefacts. Deployment duration was reported in 71% of studies, but only 32% measured configuration correctness and 21% evaluated recovery from failed automation. Fewer than one quarter provided scripts, infrastructure definitions, or sufficient environment details to support replication. Studies incorporating fault injection reported substantially richer evidence on resilience than benchmark-only evaluations. The literature shows growing interest in infrastructure-as-code, yet operational quality remains under-characterised. Future evaluations should pair speed metrics with repeatability, configuration drift, failure containment, and recovery outcomes, and should publish reusable artefacts wherever security and licensing constraints permit.
References
1. Nazir M. Automating enterprise storage provisioning with PowerShell and Python: effects on deployment time and configuration errors. Journal of Multidisciplinary Research and Integrated Sciences. 2021;1(1). Available from: https://jmris.com/index.php/jmris/article/view/2021P1V6
2. Nazir M. Automated SnapMirror migration from on-premises storage to Cloud Volumes ONTAP: a replication performance study. Journal of Multidisciplinary Research and Integrated Sciences. 2022;2(1). Available from: https://jmris.com/index.php/jmris/article/view/2022V1P6
3. Rahman A, Mahdavi-Hezaveh R, Williams L. A systematic mapping study of infrastructure as code research. Inf Softw Technol. 2019;108:65-77. doi:10.1016/j.infsof.2018.12.004.
4. Sharma T, Fragkoulis M, Spinellis D. Does your configuration code smell?. In: Proceedings of the 13th Working Conference on Mining Software Repositories. 2016. p. 189-200. doi:10.1145/2901739.2901761.
5. Rahman A, Williams L. Source code properties of defective infrastructure as code scripts. Inf Softw Technol. 2019;112:148-163. doi:10.1016/j.infsof.2019.04.013.
6. Hanappi O, Hummer W, Dustdar S. Asserting reliable convergence for configuration management scripts. In: Proceedings of the 2016 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications. 2016. p. 328-343. doi:10.1145/2983990.2984000.
7. Rahman A, Elder S, Shezan FH, Frost V, Stallings J, Williams L. Bugs in infrastructure as code. arXiv [Preprint]. 2018. doi:10.48550/arXiv.1809.07937.
8. Rahman A, Farhana E, Williams L. The 'as code' activities: development anti-patterns for infrastructure as code. arXiv [Preprint]. 2020. doi:10.48550/arXiv.2006.00177.
9. Mytkowicz T, Diwan A, Hauswirth M, Sweeney PF. Producing wrong data without doing anything obviously wrong. In: Proceedings of the 14th International Conference on Architectural Support for Programming Languages and Operating Systems. 2009. p. 265-276. doi:10.1145/1508244.1508275.
10. Kalibera T, Jones R. Rigorous benchmarking in reasonable time. In: Proceedings of the 2013 International Symposium on Memory Management. 2013. p. 63-74. doi:10.1145/2464157.2464160.
11. Dean J, Barroso LA. The tail at scale. Commun ACM. 2013;56(2):74-80. doi:10.1145/2408776.2408794.
12. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. doi:10.1136/bmj.n71.
13. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4:1. doi:10.1186/2046-4053-4-1.
14. Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, et al. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Syst Rev. 2021;10:39. doi:10.1186/s13643-020-01542-z.
15. Campbell M, McKenzie JE, Sowden A, Katikireddi SV, Brennan SE, Ellis S, et al. Synthesis without meta-analysis (SWiM) in systematic reviews: reporting guideline. BMJ. 2020;368:l6890. doi:10.1136/bmj.l6890.
16. Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak A, et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data. 2016;3:160018. doi:10.1038/sdata.2016.18.
17. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77-101. doi:10.1191/1478088706qp063oa.
18. Kottner J, Audigé L, Brorson S, Donner A, Gajewski BJ, Hróbjartsson A, et al. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. J Clin Epidemiol. 2011;64(1):96-106. doi:10.1016/j.jclinepi.2010.03.002.
Published
Issue
Section
License
Authors retain copyright. Articles published under the Creative Commons Attribution 4.0 International licence (CC BY 4.0) may be shared and adapted for any purpose, including commercially, provided appropriate credit is given, a link to the licence is supplied, and changes are indicated. No additional legal or technological restrictions may be imposed. Third-party material is included only where its credit line permits. Licence: https://creativecommons.org/licenses/by/4.0/. Earlier publications remain subject to their stated licence and author agreements unless the rights holder authorizes a change.
