Practical Training for Secure and Recoverable Cloud Automation: A Systematic Review of Learning Outcomes and Assessment Methods
Abstract
Practical cloud training increasingly combines automation, security, and disaster recovery, yet evidence on which instructional approaches produce durable operational skills remains fragmented. We systematically reviewed studies evaluating hands-on training for secure and recoverable cloud or infrastructure automation. Searches identified 1,372 records, with 68 studies meeting inclusion criteria. Interventions included laboratories, cyber ranges, scenario simulations, infrastructure-as-code exercises, and guided production tasks. Most studies reported immediate knowledge or task performance, while only 19% assessed retention beyond four weeks and 16% evaluated transfer to a substantially different environment. Hands-on and scenario-based approaches generally outperformed lecture-only comparison groups for configuration accuracy and troubleshooting performance. Evidence for automated versus instructor feedback was mixed and appeared dependent on task complexity. Reporting of assessment validity and participant experience level was inconsistent. The literature supports active practice for operational cloud skills but provides limited evidence about long-term transfer and real-world safety. Future studies should use validated performance measures, delayed follow-up, and tasks that test adaptation rather than repetition of the training exercise.
References
1. Nazir M. Scenario-based disaster recovery training for improving incident-response and RTO/RPO decision-making among enterprise cloud engineers. Journal of Computing, Intelligence and Information Sciences. 2024. Available from: https://jciis.com/index.php/jciis/article/view/2024-01-05
2. Nazir M. Effectiveness of hands-on lab training for improving zero-trust and identity-access management competencies among hybrid cloud engineers. Journal of Computing, Intelligence and Information Sciences. 2023. Available from: https://jciis.com/index.php/jciis/article/view/2023-01-02
3. Nazir M. Automating enterprise storage provisioning with PowerShell and Python: effects on deployment time and configuration errors. Journal of Multidisciplinary Research and Integrated Sciences. 2021;1(1). Available from: https://jmris.com/index.php/jmris/article/view/2021P1V6
4. Rose S, Borchert O, Mitchell S, Connelly S. Zero trust architecture. Gaithersburg (MD): National Institute of Standards and Technology. 2020;NIST SP 800-207. doi:10.6028/NIST.SP.800-207.
5. Swanson M, Bowen P, Phillips AW, Gallup D, Lynes D. Contingency planning guide for federal information systems. Gaithersburg (MD): National Institute of Standards and Technology. 2010;NIST SP 800-34 Rev. 1. doi:10.6028/NIST.SP.800-34r1.
6. Grance T, Nolan T, Burke K, Dudley R, White G, Good T. Guide to test, training, and exercise programs for IT plans and capabilities. Gaithersburg (MD): National Institute of Standards and Technology. 2006;NIST SP 800-84. doi:10.6028/NIST.SP.800-84.
7. Petersen R, Santos D, Wetzel K, Smith MC, Witte G. Workforce framework for cybersecurity (NICE Framework). Gaithersburg (MD): National Institute of Standards and Technology. 2020;NIST SP 800-181 Rev. 1. doi:10.6028/NIST.SP.800-181r1.
8. Salas E, Tannenbaum SI, Kraiger K, Smith-Jentsch KA. The science of training and development in organizations: what matters in practice. Psychol Sci Public Interest. 2012;13(2):74-101. doi:10.1177/1529100612436661.
9. Sitzmann T. A meta-analytic examination of the instructional effectiveness of computer-based simulation games. Pers Psychol. 2011;64(2):489-528. doi:10.1111/j.1744-6570.2011.01190.x.
10. Hattie J, Timperley H. The power of feedback. Rev Educ Res. 2007;77(1):81-112. doi:10.3102/003465430298487.
11. Freeman S, Eddy SL, McDonough M, Smith MK, Okoroafor N, Jordt H, et al. Active learning increases student performance in science, engineering, and mathematics. Proc Natl Acad Sci U S A. 2014;111(23):8410-8415. doi:10.1073/pnas.1319030111.
12. Salas E, DiazGranados D, Klein C, Burke CS, Stagl KC, Goodwin GF, et al. Does team training improve team performance? A meta-analysis. Hum Factors. 2008;50(6):903-933. doi:10.1518/001872008X375009.
13. Baldwin TT, Ford JK. Transfer of training: a review and directions for future research. Pers Psychol. 1988;41(1):63-105. doi:10.1111/j.1744-6570.1988.tb00632.x.
14. Arthur W Jr, Bennett W Jr, Edens PS, Bell ST. Effectiveness of training in organizations: a meta-analysis of design and evaluation features. J Appl Psychol. 2003;88(2):234-245. doi:10.1037/0021-9010.88.2.234.
15. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71. doi:10.1136/bmj.n71.
16. Moher D, Shamseer L, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4:1. doi:10.1186/2046-4053-4-1.
17. Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, et al. PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews. Syst Rev. 2021;10:39. doi:10.1186/s13643-020-01542-z.
18. Campbell M, McKenzie JE, Sowden A, Katikireddi SV, Brennan SE, Ellis S, et al. Synthesis without meta-analysis (SWiM) in systematic reviews: reporting guideline. BMJ. 2020;368:l6890. doi:10.1136/bmj.l6890.
Published
Issue
Section
License
Authors retain copyright. Articles published under the Creative Commons Attribution 4.0 International licence (CC BY 4.0) may be shared and adapted for any purpose, including commercially, provided appropriate credit is given, a link to the licence is supplied, and changes are indicated. No additional legal or technological restrictions may be imposed. Third-party material is included only where its credit line permits. Licence: https://creativecommons.org/licenses/by/4.0/. Earlier publications remain subject to their stated licence and author agreements unless the rights holder authorizes a change.
