Dependency-Aware Failover of Stateful Enterprise Applications: A Fault-Injection Evaluation of Service Recovery
Abstract
Stateful applications are difficult to recover because service availability depends on the order in which storage, databases, identity services, networks, and application components are restored. We evaluated a dependency-aware failover controller using 180 fault-injection scenarios across three representative multi-tier enterprise applications. Failures included node loss, network partition, replication lag, authentication unavailability, and partial database recovery. The controller was compared with a conventional priority-based runbook. Dependency-aware failover reduced median service restoration time from 34.8 to 24.1 minutes and decreased failed recovery attempts by 48%. Incorrect startup ordering accounted for most repeat attempts in the runbook condition. The controller provided the largest benefit when identity and data services had interdependent readiness checks, while simple single-tier failures showed little difference between approaches. Recovery success within the target window increased from 78% to 92%. These results indicate that failover automation should model service dependencies and readiness states explicitly rather than rely on static priority lists. Dependency graphs also improved post-incident explainability by recording why each recovery action was delayed or released.
References
1. Nazir M. SVM-based disaster recovery for hybrid enterprise storage: effects on recovery time and service availability. Journal of Computing, Intelligence and Information Sciences. 2022. Available from: https://jciis.com/index.php/jciis/article/view/2022-01-05
2. Nazir M. RTO and RPO optimization using asynchronous replication in multi-petabyte enterprise storage environments. Journal of Computing, Intelligence and Information Sciences. 2021. Available from: https://jciis.com/index.php/jciis/article/view/2020-01-04
3. Patterson RH, Manley S, Federwisch M, Hitz D, Kleiman S, Owara S. SnapMirror: file-system-based asynchronous mirroring for disaster recovery. In: Proceedings of the 1st USENIX Conference on File and Storage Technologies. 2002. p. 117-129. Available from: https://www.usenix.org/conference/fast-02/snapmirror-file-system-based-asynchronous-mirroring-disaster-recovery
4. Cully B, Lefebvre G, Meyer D, Feeley M, Hutchinson N, Warfield A. Remus: high availability via asynchronous virtual machine replication. In: Proceedings of the 5th USENIX Symposium on Networked Systems Design and Implementation. 2008. p. 161-174. Available from: https://www.usenix.org/legacy/events/nsdi08/tech/full_papers/cully/cully.pdf
5. Gray J, Lamport L. Consensus on transaction commit. ACM Trans Database Syst. 2006;31(1):133-160. doi:10.1145/1132863.1132867.
6. Gilbert S, Lynch N. Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services. ACM SIGACT News. 2002;33(2):51-59. doi:10.1145/564585.564601.
7. Ghemawat S, Gobioff H, Leung ST. The Google file system. In: Proceedings of the 19th ACM Symposium on Operating Systems Principles. 2003. p. 29-43. doi:10.1145/945445.945450.
8. DeCandia G, Hastorun D, Jampani M, Kakulapati G, Lakshman A, Pilchin A, et al. Dynamo: Amazon's highly available key-value store. In: Proceedings of the 21st ACM Symposium on Operating Systems Principles. 2007. p. 205-220. doi:10.1145/1294261.1294281.
9. Dean J, Barroso LA. The tail at scale. Commun ACM. 2013;56(2):74-80. doi:10.1145/2408776.2408794.
10. NetApp. Learn about ONTAP SnapMirror asynchronous disaster recovery [Internet]. [cited 2026 Sep 22]. Available from: https://docs.netapp.com/us-en/ontap/data-protection/snapmirror-disaster-recovery-concept.html
11. Swanson M, Bowen P, Phillips AW, Gallup D, Lynes D. Contingency planning guide for federal information systems. Gaithersburg (MD): National Institute of Standards and Technology. 2010;NIST SP 800-34 Rev. 1. doi:10.6028/NIST.SP.800-34r1.
12. Grance T, Nolan T, Burke K, Dudley R, White G, Good T. Guide to test, training, and exercise programs for IT plans and capabilities. Gaithersburg (MD): National Institute of Standards and Technology. 2006;NIST SP 800-84. doi:10.6028/NIST.SP.800-84.
13. Mytkowicz T, Diwan A, Hauswirth M, Sweeney PF. Producing wrong data without doing anything obviously wrong. In: Proceedings of the 14th International Conference on Architectural Support for Programming Languages and Operating Systems. 2009. p. 265-276. doi:10.1145/1508244.1508275.
14. Kalibera T, Jones R. Rigorous benchmarking in reasonable time. In: Proceedings of the 2013 International Symposium on Memory Management. 2013. p. 63-74. doi:10.1145/2464157.2464160.
15. Lakens D. Sample size justification. Collabra Psychol. 2022;8(1):33267. doi:10.1525/collabra.33267.
16. Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. Proc Natl Acad Sci U S A. 2018;115(11):2600-2606. doi:10.1073/pnas.1708274114.
17. Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak A, et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data. 2016;3:160018. doi:10.1038/sdata.2016.18.
18. Lakens D. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Front Psychol. 2013;4:863. doi:10.3389/fpsyg.2013.00863.
Published
Issue
Section
License
Authors retain copyright. Articles published under the Creative Commons Attribution 4.0 International licence (CC BY 4.0) may be shared and adapted for any purpose, including commercially, provided appropriate credit is given, a link to the licence is supplied, and changes are indicated. No additional legal or technological restrictions may be imposed. Third-party material is included only where its credit line permits. Licence: https://creativecommons.org/licenses/by/4.0/. Earlier publications remain subject to their stated licence and author agreements unless the rights holder authorizes a change.
