Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
ISO/IEC 25045:2010 SQuaRE — Evaluation Module for Recoverability
Disturbance injection methodology for measuring resiliency and autonomic recovery in software systems
1. Understanding the Recoverability Evaluation Module
ISO/IEC 25045 is part of the Quality Evaluation Division (ISO/IEC 2504n) within the SQuaRE series. It provides a specialized evaluation module for measuring the recoverability sub-characteristic of software reliability. What makes this standard particularly valuable for practicing engineers is its disturbance injection methodology — a systematic, repeatable approach to quantifying how well a system withstands and recovers from operational faults and unexpected events.
For reliability engineers and DevOps practitioners, ISO/IEC 25045 offers what amounts to a standardized chaos engineering framework, years before the term “chaos engineering” became mainstream. It provides a structured way to inject faults, measure impact, and score autonomic recovery capabilities.
The standard defines two primary quality measures:
- Resiliency (Quantitative) — The ratio of successfully completed transactions under disturbance to those completed in a disturbance-free baseline. This is a direct, objective measure of service degradation.
- Autonomic Recovery Index (Qualitative) — A scored assessment of how well the system detects, analyzes, and resolves disturbances without human intervention, mapped to five levels of autonomic maturity: Basic, Managed, Predictive, Adaptive, and Autonomic.
| Autonomic Level | Score | Description | Detection Example |
|---|---|---|---|
| Basic | 0 | Manual management via reports and product manuals | Help desk calls operators about user complaints |
| Managed | 1 | Management software automates IT tasks | Operators monitor a single management console |
| Predictive | 2 | Tools analyze changes and recommend actions | Autonomic manager notifies operator of a potential problem |
| Adaptive | 3 | Components collectively monitor, analyze, take action with minimal intervention | System detects and analyzes without human involvement, may initiate recovery |
| Autonomic | 4 | Fully automated management by business rules and policies | End-to-end autonomic detection, analysis, and recovery |
A critical engineering insight: The Resiliency measure and Autonomic Recovery Index capture fundamentally different aspects of recoverability. A system might have excellent resiliency (transactions continue during a fault) but poor autonomic index (recovery requires a human operator to restart services). Both measures are needed for a complete picture.
2. The Disturbance Injection Methodology
2.1 Three-Phase Evaluation Process
The evaluation methodology consists of three phases: Baseline, Test, and Check. The Baseline phase establishes normal operational characteristics without disturbances. The Test phase runs the same workload while injecting disturbances. The Check phase verifies system integrity after disturbance testing.
Each disturbance injection is organized into an injection slot with five sub-intervals: Injection Interval (steady state before fault), Detection Interval (time to detect the fault), Recovery Initiation Interval (time to begin recovery), Recovery Interval (time to perform recovery), and Keep Interval (time to re-establish steady state after recovery).
2.2 Disturbance Categories
The standard defines five mandatory disturbance categories for conformance testing:
| Category | Examples | Engineering Relevance |
|---|---|---|
| Unexpected Shutdown | OS shutdown, process termination, network link failure | Simulates operator errors and software crashes — the most common class of production incidents |
| Resource Contention | CPU hog, memory exhaustion, I/O saturation, DBMS deadlock, runaway query, disk full | Simulates noisy neighbor scenarios and resource leaks — increasingly important in multi-tenant cloud environments |
| Loss of Data | Database file deletion, disk loss, table corruption | Simulates storage failures and accidental data deletion — tests backup and recovery mechanisms |
| Load Resolution | 2x and 10x user surge | Simulates traffic spikes (flash crowds, DDoS, viral events) — tests auto-scaling and flow control |
| Restart Failure | Corrupted boot configuration, missing executables | Simulates failures that occur during recovery itself — tests robustness of the recovery mechanism |
A practical consideration: each injection slot should ideally be run in isolation — stop, reset, and restart the system between disturbances. Running disturbances sequentially without resetting can produce misleading results, as residual effects from earlier disturbances compound. However, sequential testing can be useful for evaluating how systems handle cascading failures.
3. Applying ISO/IEC 25045 in Modern Engineering Practice
| Use Case | How ISO/IEC 25045 Applies | Modern Implementation |
|---|---|---|
| Pre-production validation | Run disturbance injection as part of system verification testing | Integrate chaos experiments into CI/CD pipelines |
| Production readiness assessment | Evaluate recoverability of production systems against test environments | Game days and controlled blast-radius experiments |
| Vendor comparison | Compare recoverability of different solutions using common workload | Standardized benchmark suites with fault injection |
| SLA validation | Verify that recovery time objectives (RTO) are met under disturbance | Automated SLA verification with fault injection scenarios |
Start with the “Unexpected Shutdown” category — it is the easiest to implement (kill a process, take down a network interface) and often reveals the most surprising failure modes. Common findings include: no automatic reconnect logic, hardcoded IP addresses that become unreachable, and missing health check endpoints.
4. Frequently Asked Questions
Q1: How many disturbances must be tested for conformance?
The standard requires that all five disturbance categories be used. Within each category, at least one representative disturbance should be tested. The specific disturbances can be selected based on the system architecture and operational context.
The standard requires that all five disturbance categories be used. Within each category, at least one representative disturbance should be tested. The specific disturbances can be selected based on the system architecture and operational context.
Q2: Can the autonomic maturity questionnaire be customized?
Yes. The standard allows the points awarded for each autonomic level to be adjusted based on experience, customer preference, and context. However, the scoring methodology and the three core questions (detection, analysis, action) must remain as defined.
Yes. The standard allows the points awarded for each autonomic level to be adjusted based on experience, customer preference, and context. However, the scoring methodology and the three core questions (detection, analysis, action) must remain as defined.
Q3: How does baseline repeatability affect the validity of results?
The baseline must be run at least three times, and the results must fall within a predefined statistical significance threshold (e.g., throughput variation less than 5%). Without a stable baseline, the resiliency calculation (Pi/Pbase) loses its validity.
The baseline must be run at least three times, and the results must fall within a predefined statistical significance threshold (e.g., throughput variation less than 5%). Without a stable baseline, the resiliency calculation (Pi/Pbase) loses its validity.
Q4: Is ISO/IEC 25045 applicable to microservices architectures?
Absolutely. In fact, microservices architectures benefit particularly well from this evaluation module because individual service failures are expected and the recovery mechanisms (circuit breakers, retries, service mesh failover) can be systematically evaluated using the disturbance injection methodology.
Absolutely. In fact, microservices architectures benefit particularly well from this evaluation module because individual service failures are expected and the recovery mechanisms (circuit breakers, retries, service mesh failover) can be systematically evaluated using the disturbance injection methodology.
