Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
ISO/IEC 25040:2024 — SQuaRE — Quality Evaluation Framework
Framework for quality evaluation of ICT products, data, and IT services with four evaluation types
1. The Quality Evaluation Framework of ISO/IEC 25040
ISO/IEC 25040:2024 provides the comprehensive framework for quality evaluation of ICT products, data, and IT services within the SQuaRE series. This second edition replaces the 2011 version with significant enhancements: alignment with other SQuaRE divisions, expansion of target entities from software to complete ICT products and services, introduction of four distinct types of quality evaluation, and more practical guidance for planning evaluations.
The standard addresses a fundamental question: how does an organization objectively determine whether a system, software product, or data asset meets its quality goals? The answer lies in a structured five-step process: Define the evaluation, Design the evaluation, Plan the evaluation, Execute the evaluation, and Conclude the evaluation. Each step has clearly specified outcomes and activities, creating a repeatable, auditable evaluation methodology.
Before starting any quality evaluation, clearly identify which of the four evaluation types (T1-T4) applies to your context. Using the wrong evaluation model wastes resources and produces meaningless results. For procurement decisions, use T1 (suitability to specific use); for certification, use T2 (qualification to quality standard).
| Evaluation Type | Purpose | Quality Criteria Source | Typical Output | Common Assessment Use |
|---|---|---|---|---|
| T1: Suitability to specific use | Select best-fit component | Requirements specification | Candidates with score | Selection judgment |
| T2: Qualification to quality standard | Certify compliance | Diagnostic model | Pass/Fail per characteristic | Official certification |
| T3: Checking requirements satisfaction | Verify agreed requirements | Requirements specification | Pass/Fail + Weaknesses | Acceptance judgment |
| T4: Suitability to the market | Assess market value | Diagnostic model / Requirements spec | Score + Recommendations | Market value judgment |
2. The Five-Step Quality Evaluation Process Reference Model
The quality evaluation process reference model (Clause 5) is the operational heart of ISO/IEC 25040. Each step transforms inputs into verifiable outputs:
Define the evaluation (5.2): Establish the purpose, identify target entities, define quality evaluation criteria, and set rigor requirements. The evaluation criteria must follow the ISO/IEC 25030 format: target entity, selected characteristic, quality goal with conditions, quality measure, target value, and acceptable range. Four factors influence rigor: coverage of information needs, objectivity of measurement, acceptability of rating levels, and transparency of the process.
Design the evaluation (5.3): Identify components to be evaluated, select and implement quality rating modules, determine quality analysis methods, and define outputs. Quality rating modules are a key innovation — they encapsulate measures, operational environment, and rating methods for a category of target entities. Template rating modules can be parameterized for reuse across projects.
The rigor of evaluation must match the criticality of the decision being supported. Using high-rigor evaluation for preliminary feasibility studies wastes resources; using low-rigor evaluation for safety-critical acceptance testing creates unacceptable risk. Always align evaluation rigor with system integrity level (ISO/IEC/IEEE 15026-3).
Execute the evaluation (5.5): Apply quality rating modules to produce measurement data, then analyze rating results to derive evaluation conclusions for each quality characteristic. Measurement sources can include executable programs, source code, data, user manuals, design specifications, and prototypes — the available artefacts determine which quality measures can be applied.
Conclude the evaluation (5.6): Review results collaboratively, create the evaluation report, review the evaluation process itself for improvement, and manage disposition of evaluation data. The evaluation report must include sufficient detail to enable reproducibility.
3. Engineering Insights for Effective Quality Evaluation
ISO/IEC 25040 recognizes that quality evaluation is not a single event but an ongoing activity that should be integrated into system/software life cycle processes. Annex C maps the evaluation process to ISO/IEC/IEEE 15288 and 12207 processes, showing how quality evaluation activities align with acquisition, development, verification, validation, operation, and maintenance processes.
The most successful quality evaluation programs treat evaluation as a continuous improvement feedback loop. Each evaluation not only assesses the target entity but also improves the evaluation process itself — rating modules are refined, measurement sources are improved, and evaluation criteria are updated based on lessons learned.
A critical concept introduced in this edition is the “quality rating module” — a reusable package containing quality measures, operational environment specifications, measurement methods, and rating levels for a specific category of target entities. Organizations can build libraries of template rating modules for different application domains (web applications, IoT devices, data warehouses), significantly reducing the cost and effort of repeated evaluations.
The standard also addresses measurement sources comprehensively (Annex A), listing which quality characteristics can be measured from each artefact type. For example, executable programs are “fully applicable” for measuring most product quality characteristics through testing, while design specifications are partially applicable through static analysis and inspection. This guidance helps evaluation planners identify missing artefacts early and adjust their evaluation approach accordingly.
Q1: What are the four types of quality evaluation in ISO/IEC 25040?
A: T1 (suitability to specific use), T2 (qualification to quality standard), T3 (checking requirements satisfaction), and T4 (suitability to the market). Each has distinct purposes, quality criteria sources, and typical outputs.
A: T1 (suitability to specific use), T2 (qualification to quality standard), T3 (checking requirements satisfaction), and T4 (suitability to the market). Each has distinct purposes, quality criteria sources, and typical outputs.
Q2: How does ISO/IEC 25040 relate to ISO/IEC 25030?
A: ISO/IEC 25030 defines how to specify quality requirements, while ISO/IEC 25040 defines how to evaluate whether those requirements have been met. Together they form the requirements-to-evaluation lifecycle for quality.
A: ISO/IEC 25030 defines how to specify quality requirements, while ISO/IEC 25040 defines how to evaluate whether those requirements have been met. Together they form the requirements-to-evaluation lifecycle for quality.
Q3: What is a quality rating module?
A> A cohesive collection of quality measures, operational environment specifications, and rating methods for a specific category of target entities. Template modules are parameterized for reuse; implementation modules are directly applicable.
A> A cohesive collection of quality measures, operational environment specifications, and rating methods for a specific category of target entities. Template modules are parameterized for reuse; implementation modules are directly applicable.
Q4: Can quality evaluation be performed early in development?
A> Yes. Using design artefacts (specifications, prototypes, models) as measurement sources, evaluations can be conducted before the target entity is fully built. This enables early detection of quality issues when correction costs are minimal.
A> Yes. Using design artefacts (specifications, prototypes, models) as measurement sources, evaluations can be conducted before the target entity is fully built. This enables early detection of quality issues when correction costs are minimal.
