Adaptive Versus Fixed-Sequence Disaster Recovery Simulations: A Randomised Study of Recovery-Objective Decision-Making
Abstract
Disaster recovery simulations are commonly delivered as fixed scenarios, although real incidents evolve unpredictably and require repeated decisions as new information appears. In a randomised study, 132 cloud engineers completed either adaptive simulations that changed in response to participant actions or fixed-sequence exercises with identical initial conditions. Outcomes included recovery-objective prioritisation, decision accuracy, time to stabilisation, and post-exercise transfer to a novel scenario. Adaptive simulations produced higher decision accuracy and better prioritisation of recovery-time and recovery-point constraints, particularly when conflicting business requirements were introduced. Median stabilisation time was 12% longer during training because participants encountered more branches, but performance on the novel transfer scenario was 19% higher. Fixed-sequence participants reported greater perceived fluency despite making more prioritisation errors in transfer. Debriefing quality influenced outcomes in both groups. Adaptive disaster-recovery exercises appear to strengthen decision-making under uncertainty even when they require more training time. Evaluation should therefore distinguish rapid completion of a familiar scenario from the ability to reason through changing recovery conditions.
References
1. Nazir M. Scenario-based disaster recovery training for improving incident-response and RTO/RPO decision-making among enterprise cloud engineers. Journal of Computing, Intelligence and Information Sciences. 2024. Available from: https://jciis.com/index.php/jciis/article/view/2024-01-05
2. Rose S, Borchert O, Mitchell S, Connelly S. Zero trust architecture. Gaithersburg (MD): National Institute of Standards and Technology. 2020;NIST SP 800-207. doi:10.6028/NIST.SP.800-207.
3. Swanson M, Bowen P, Phillips AW, Gallup D, Lynes D. Contingency planning guide for federal information systems. Gaithersburg (MD): National Institute of Standards and Technology. 2010;NIST SP 800-34 Rev. 1. doi:10.6028/NIST.SP.800-34r1.
4. Grance T, Nolan T, Burke K, Dudley R, White G, Good T. Guide to test, training, and exercise programs for IT plans and capabilities. Gaithersburg (MD): National Institute of Standards and Technology. 2006;NIST SP 800-84. doi:10.6028/NIST.SP.800-84.
5. Petersen R, Santos D, Wetzel K, Smith MC, Witte G. Workforce framework for cybersecurity (NICE Framework). Gaithersburg (MD): National Institute of Standards and Technology. 2020;NIST SP 800-181 Rev. 1. doi:10.6028/NIST.SP.800-181r1.
6. Sweller J. Cognitive load during problem solving: effects on learning. Cogn Sci. 1988;12(2):257-285. doi:10.1207/s15516709cog1202_4.
7. Salas E, Tannenbaum SI, Kraiger K, Smith-Jentsch KA. The science of training and development in organizations: what matters in practice. Psychol Sci Public Interest. 2012;13(2):74-101. doi:10.1177/1529100612436661.
8. Sitzmann T. A meta-analytic examination of the instructional effectiveness of computer-based simulation games. Pers Psychol. 2011;64(2):489-528. doi:10.1111/j.1744-6570.2011.01190.x.
9. Hart SG, Staveland LE. Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. Adv Psychol. 1988;52:139-183. doi:10.1016/S0166-4115(08)62386-9.
10. Hattie J, Timperley H. The power of feedback. Rev Educ Res. 2007;77(1):81-112. doi:10.3102/003465430298487.
11. Freeman S, Eddy SL, McDonough M, Smith MK, Okoroafor N, Jordt H, et al. Active learning increases student performance in science, engineering, and mathematics. Proc Natl Acad Sci U S A. 2014;111(23):8410-8415. doi:10.1073/pnas.1319030111.
12. Salas E, DiazGranados D, Klein C, Burke CS, Stagl KC, Goodwin GF, et al. Does team training improve team performance? A meta-analysis. Hum Factors. 2008;50(6):903-933. doi:10.1518/001872008X375009.
13. Baldwin TT, Ford JK. Transfer of training: a review and directions for future research. Pers Psychol. 1988;41(1):63-105. doi:10.1111/j.1744-6570.1988.tb00632.x.
14. Arthur W Jr, Bennett W Jr, Edens PS, Bell ST. Effectiveness of training in organizations: a meta-analysis of design and evaluation features. J Appl Psychol. 2003;88(2):234-245. doi:10.1037/0021-9010.88.2.234.
15. Aguinis H, Kraiger K. Benefits of training and development for individuals and teams, organizations, and society. Annu Rev Psychol. 2009;60:451-474. doi:10.1146/annurev.psych.60.110707.163505.
16. Lakens D. Sample size justification. Collabra Psychol. 2022;8(1):33267. doi:10.1525/collabra.33267.
17. Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. Proc Natl Acad Sci U S A. 2018;115(11):2600-2606. doi:10.1073/pnas.1708274114.
18. Lakens D. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Front Psychol. 2013;4:863. doi:10.3389/fpsyg.2013.00863.
Published
Issue
Section
License
Authors retain copyright. Articles published under the Creative Commons Attribution 4.0 International licence (CC BY 4.0) may be shared and adapted for any purpose, including commercially, provided appropriate credit is given, a link to the licence is supplied, and changes are indicated. No additional legal or technological restrictions may be imposed. Third-party material is included only where its credit line permits. Licence: https://creativecommons.org/licenses/by/4.0/. Earlier publications remain subject to their stated licence and author agreements unless the rights holder authorizes a change.
