Performance-Based Assessment of Cloud Resilience Competence: Instrument Development and Validation
Abstract
Assessing cloud resilience competence requires more than a knowledge test because engineers must interpret evidence, prioritise recovery objectives, and execute technically correct actions under constraints. We developed a performance-based assessment containing eight scenario stations spanning backup validation, failover planning, dependency analysis, identity recovery, monitoring, and post-incident verification. Content validity was established by a 12-member expert panel, followed by pilot testing with 148 cloud and infrastructure professionals. Internal consistency was acceptable for the total score, and inter-rater agreement exceeded 0.82 after rubric calibration. Scores differentiated junior, intermediate, and senior experience groups and correlated moderately with supervisor ratings of operational performance. Knowledge-test scores explained only part of the variance in scenario performance, particularly for prioritisation and troubleshooting stations. A confirmatory factor model supported a three-domain structure covering planning, execution, and verification. The instrument demonstrated promising reliability and construct validity for training evaluation. Further validation in independent organisations is warranted, but performance-based assessment appears to capture aspects of resilience competence that are missed by conventional multiple-choice testing.
References
1. Azam MA. Desktop Display Stand for Electronic Tablets and Screens. GB Patent 6,509,489. 2026. Available from: https://scholar.google.com/citations?view_op=view_citation&hl=en&user=Kh7pTrcAAAAJ&citation_for_view=Kh7pTrcAAAAJ:eQOLeE2rZwMC
2. Nazir M. Scenario-based disaster recovery training for improving incident-response and RTO/RPO decision-making among enterprise cloud engineers. Journal of Computing, Intelligence and Information Sciences. 2024. Available from: https://jciis.com/index.php/jciis/article/view/2024-01-05
3. Nazir M. Effectiveness of hands-on lab training for improving zero-trust and identity-access management competencies among hybrid cloud engineers. Journal of Computing, Intelligence and Information Sciences. 2023. Available from: https://jciis.com/index.php/jciis/article/view/2023-01-02
4. Fornell C, Larcker DF. Evaluating structural equation models with unobservable variables and measurement error. J Mark Res. 1981;18(1):39-50. doi:10.1177/002224378101800104.
5. Rose S, Borchert O, Mitchell S, Connelly S. Zero trust architecture. Gaithersburg (MD): National Institute of Standards and Technology. 2020;NIST SP 800-207. doi:10.6028/NIST.SP.800-207.
6. Swanson M, Bowen P, Phillips AW, Gallup D, Lynes D. Contingency planning guide for federal information systems. Gaithersburg (MD): National Institute of Standards and Technology. 2010;NIST SP 800-34 Rev. 1. doi:10.6028/NIST.SP.800-34r1.
7. Grance T, Nolan T, Burke K, Dudley R, White G, Good T. Guide to test, training, and exercise programs for IT plans and capabilities. Gaithersburg (MD): National Institute of Standards and Technology. 2006;NIST SP 800-84. doi:10.6028/NIST.SP.800-84.
8. Petersen R, Santos D, Wetzel K, Smith MC, Witte G. Workforce framework for cybersecurity (NICE Framework). Gaithersburg (MD): National Institute of Standards and Technology. 2020;NIST SP 800-181 Rev. 1. doi:10.6028/NIST.SP.800-181r1.
9. Sweller J. Cognitive load during problem solving: effects on learning. Cogn Sci. 1988;12(2):257-285. doi:10.1207/s15516709cog1202_4.
10. Salas E, Tannenbaum SI, Kraiger K, Smith-Jentsch KA. The science of training and development in organizations: what matters in practice. Psychol Sci Public Interest. 2012;13(2):74-101. doi:10.1177/1529100612436661.
11. Sitzmann T. A meta-analytic examination of the instructional effectiveness of computer-based simulation games. Pers Psychol. 2011;64(2):489-528. doi:10.1111/j.1744-6570.2011.01190.x.
12. Hart SG, Staveland LE. Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. Adv Psychol. 1988;52:139-183. doi:10.1016/S0166-4115(08)62386-9.
13. Hattie J, Timperley H. The power of feedback. Rev Educ Res. 2007;77(1):81-112. doi:10.3102/003465430298487.
14. Freeman S, Eddy SL, McDonough M, Smith MK, Okoroafor N, Jordt H, et al. Active learning increases student performance in science, engineering, and mathematics. Proc Natl Acad Sci U S A. 2014;111(23):8410-8415. doi:10.1073/pnas.1319030111.
15. Salas E, DiazGranados D, Klein C, Burke CS, Stagl KC, Goodwin GF, et al. Does team training improve team performance? A meta-analysis. Hum Factors. 2008;50(6):903-933. doi:10.1518/001872008X375009.
16. Baldwin TT, Ford JK. Transfer of training: a review and directions for future research. Pers Psychol. 1988;41(1):63-105. doi:10.1111/j.1744-6570.1988.tb00632.x.
17. Kottner J, Audigé L, Brorson S, Donner A, Gajewski BJ, Hróbjartsson A, et al. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. J Clin Epidemiol. 2011;64(1):96-106. doi:10.1016/j.jclinepi.2010.03.002.
18. Lakens D. Sample size justification. Collabra Psychol. 2022;8(1):33267. doi:10.1525/collabra.33267.
Published
Issue
Section
License
Authors retain copyright. Articles published under the Creative Commons Attribution 4.0 International licence (CC BY 4.0) may be shared and adapted for any purpose, including commercially, provided appropriate credit is given, a link to the licence is supplied, and changes are indicated. No additional legal or technological restrictions may be imposed. Third-party material is included only where its credit line permits. Licence: https://creativecommons.org/licenses/by/4.0/. Earlier publications remain subject to their stated licence and author agreements unless the rights holder authorizes a change.
