Runbook Usability and Cognitive Workload During Storage Recovery: A Randomised Crossover Study

Authors

  • Mateo Costa Department of Information Systems and Human Factors, Middle East Technical University (METU) Author
  • Adam Bennett Department of Information Systems and Human Factors, American University of Beirut Author
  • Amina Qureshi Department of Information Systems and Human Factors, Lahore University of Management Sciences (LUMS) Author

Abstract

Recovery runbooks can reduce omission errors during incidents, but excessive detail or poor information architecture may increase cognitive workload when operators are under time pressure. We used a randomised crossover design in which 64 infrastructure engineers completed two storage-recovery scenarios using a conventional text-heavy runbook and a redesigned task-oriented runbook. Scenario order and runbook sequence were counterbalanced. Primary measures were recovery accuracy, completion time, NASA-TLX workload, navigation events, and skipped steps. The task-oriented runbook reduced median completion time by 14% and lowered overall workload scores, with the largest improvement in temporal demand. Skipped critical steps fell from 8.1% to 3.2%, although the difference was concentrated in complex branching sections. Experienced participants navigated both formats efficiently, whereas less experienced engineers benefited substantially from explicit decision points and status cues. Excessive cross-referencing was the strongest predictor of navigation delay. Runbook usability should therefore be evaluated empirically under realistic time pressure. Clear branching logic, concise action statements, and visible progress markers improved both efficiency and execution quality without removing necessary technical detail.

References

1. Azam MA. Desktop Display Stand for Electronic Tablets and Screens. GB Patent 6,509,489. 2026. Available from: https://scholar.google.com/citations?view_op=view_citation&hl=en&user=Kh7pTrcAAAAJ&citation_for_view=Kh7pTrcAAAAJ:eQOLeE2rZwMC

2. Nazir M. Scenario-based disaster recovery training for improving incident-response and RTO/RPO decision-making among enterprise cloud engineers. Journal of Computing, Intelligence and Information Sciences. 2024. Available from: https://jciis.com/index.php/jciis/article/view/2024-01-05

3. Rose S, Borchert O, Mitchell S, Connelly S. Zero trust architecture. Gaithersburg (MD): National Institute of Standards and Technology. 2020;NIST SP 800-207. doi:10.6028/NIST.SP.800-207.

4. Swanson M, Bowen P, Phillips AW, Gallup D, Lynes D. Contingency planning guide for federal information systems. Gaithersburg (MD): National Institute of Standards and Technology. 2010;NIST SP 800-34 Rev. 1. doi:10.6028/NIST.SP.800-34r1.

5. Grance T, Nolan T, Burke K, Dudley R, White G, Good T. Guide to test, training, and exercise programs for IT plans and capabilities. Gaithersburg (MD): National Institute of Standards and Technology. 2006;NIST SP 800-84. doi:10.6028/NIST.SP.800-84.

6. Petersen R, Santos D, Wetzel K, Smith MC, Witte G. Workforce framework for cybersecurity (NICE Framework). Gaithersburg (MD): National Institute of Standards and Technology. 2020;NIST SP 800-181 Rev. 1. doi:10.6028/NIST.SP.800-181r1.

7. Sweller J. Cognitive load during problem solving: effects on learning. Cogn Sci. 1988;12(2):257-285. doi:10.1207/s15516709cog1202_4.

8. Salas E, Tannenbaum SI, Kraiger K, Smith-Jentsch KA. The science of training and development in organizations: what matters in practice. Psychol Sci Public Interest. 2012;13(2):74-101. doi:10.1177/1529100612436661.

9. Sitzmann T. A meta-analytic examination of the instructional effectiveness of computer-based simulation games. Pers Psychol. 2011;64(2):489-528. doi:10.1111/j.1744-6570.2011.01190.x.

10. Hart SG, Staveland LE. Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. Adv Psychol. 1988;52:139-183. doi:10.1016/S0166-4115(08)62386-9.

11. Hattie J, Timperley H. The power of feedback. Rev Educ Res. 2007;77(1):81-112. doi:10.3102/003465430298487.

12. Freeman S, Eddy SL, McDonough M, Smith MK, Okoroafor N, Jordt H, et al. Active learning increases student performance in science, engineering, and mathematics. Proc Natl Acad Sci U S A. 2014;111(23):8410-8415. doi:10.1073/pnas.1319030111.

13. Salas E, DiazGranados D, Klein C, Burke CS, Stagl KC, Goodwin GF, et al. Does team training improve team performance? A meta-analysis. Hum Factors. 2008;50(6):903-933. doi:10.1518/001872008X375009.

14. Baldwin TT, Ford JK. Transfer of training: a review and directions for future research. Pers Psychol. 1988;41(1):63-105. doi:10.1111/j.1744-6570.1988.tb00632.x.

15. Arthur W Jr, Bennett W Jr, Edens PS, Bell ST. Effectiveness of training in organizations: a meta-analysis of design and evaluation features. J Appl Psychol. 2003;88(2):234-245. doi:10.1037/0021-9010.88.2.234.

16. Aguinis H, Kraiger K. Benefits of training and development for individuals and teams, organizations, and society. Annu Rev Psychol. 2009;60:451-474. doi:10.1146/annurev.psych.60.110707.163505.

17. Lakens D. Sample size justification. Collabra Psychol. 2022;8(1):33267. doi:10.1525/collabra.33267.

18. Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. Proc Natl Acad Sci U S A. 2018;115(11):2600-2606. doi:10.1073/pnas.1708274114.

issue7

Published

2026-06-01

Issue

Vol. 7 No. 01 (2026): Agentic AI, Intelligent Automation and Emerging Computing Frontiers

Section

Articles

License

Authors retain copyright. Articles published under the Creative Commons Attribution 4.0 International licence (CC BY 4.0) may be shared and adapted for any purpose, including commercially, provided appropriate credit is given, a link to the licence is supplied, and changes are indicated. No additional legal or technological restrictions may be imposed. Third-party material is included only where its credit line permits. Licence: https://creativecommons.org/licenses/by/4.0/. Earlier publications remain subject to their stated licence and author agreements unless the rights holder authorizes a change.