Understanding CAN/CSA-ISO/IEC TR 9126-4-05: Quality in Use Metrics for Software Product Quality

Software quality measurement is a cornerstone of dependable software engineering. The international community has long relied on the ISO/IEC 9126 series to define and evaluate software product quality. As a Canadian adoption of the international technical report ISO/IEC TR 9126-4:2005, CAN/CSA-ISO/IEC TR 9126-4-05 provides a structured framework for measuring quality in use — the user’s perception and experience of software in a specific context. This article explains the scope, technical content, implementation considerations, and compliance notes for this important document.

Scope and Applicability of CAN/CSA-ISO/IEC TR 9126-4-05

CAN/CSA-ISO/IEC TR 9126-4-05 is a Technical Report (TR) that defines a comprehensive set of metrics for quality in use (QinU). Unlike internal or external quality metrics that focus on code or system behaviour, quality in use metrics measure the outcomes of software when employed by real users in a specified environment.

Purpose and Intended Audience

The primary purpose of this TR is to guide organizations in:

Defining measurable quality requirements for software from a user perspective
Evaluating whether software meets those requirements during development, acquisition, or operation
Comparing alternative software products in terms of user effectiveness, productivity, safety, and satisfaction

The document is aligned with the quality model introduced in ISO/IEC 9126-1 and is intended for software engineers, quality assurance professionals, usability specialists, and procurement officers. Its adoption as a Canadian standard by CSA Group ensures that Canadian organizations have a consistent, internationally recognized basis for usability evaluation.

Relation to the ISO 9126 Series

ISO/IEC 9126 consists of four parts:

Part 1: Quality Model
Part 2: External Metrics
Part 3: Internal Metrics
Part 4: Quality in Use Metrics (this TR)

While Parts 2 and 3 address static code attributes and system behaviour, Part 4 uniquely focuses on the effects of software use. Note that the entire 9126 series has been superseded by the ISO/IEC 25000 SQuaRE series; however, TR 9126-4 remains a valuable reference for legacy systems and organisations that still operate under the original model. The Canadian adoption (2005 edition) is identical to the international version and carries the full authority of a national standard.

Technical Requirements and Quality in Use Metrics Framework

CAN/CSA-ISO/IEC TR 9126-4-05 organizes quality in use around four core characteristics. For each characteristic, the TR provides definitions, examples of metrics, and guidance on measurement methods.

Characteristic	Definition (from the TR)	Example Metrics
Effectiveness	Accuracy and completeness with which users achieve specified goals	Task completion rate, error frequency, task success ratio
Productivity	Resources (time, money, effort) expended relative to the effectiveness achieved	Time to complete a task, cost per task, efficiency ratio (completion time / task complexity)
Safety	Expected consequences of use on users, equipment, and the environment	Number of user errors leading to harm, severity of safety incidents, economic loss
Satisfaction	Users’ subjective reactions to the software, including attitudes and perceptions	User satisfaction score (e.g., SUS), net promoter score (NPS), complaint rate

Tip: When using the TR, always define the context of use (users, tasks, equipment, environment) before selecting metrics. Context influences interpretation and comparability of results.

Measurement Approaches and Data Collection

The TR does not prescribe mandatory metrics but provides a catalogue of candidate metrics. Organisations can select, combine, or adapt them to their specific evaluation goals. Recommended data collection methods include:

Controlled user tests with logging
Field studies and observation
Surveys and questionnaires
Automated usage analytics

Normalisation is a key theme: raw counts (e.g., number of errors) should be normalised by task duration, number of users, or task complexity to enable fair comparisons across contexts.

Implementation Highlights

Implementation best practice: Integrate quality in use metrics into your agile sprints by collecting small, frequent measurements during usability testing. Use the TR’s metrics as a checklist to identify weak areas early.

Define clear goals: Before measuring, articulate what “good” looks like for each characteristic (e.g., “task completion rate ≥ 90 %”).
Use multiple characteristics: Effectiveness alone does not guarantee a good user experience. Always consider productivity and satisfaction together.
Document context: For reproducibility, record user profiles, tasks, environment (hardware, software, physical setting), and any deviations from typical use.

Caution: The TR is a technical report, not a normative standard. It offers guidance rather than requirements. Organisations should not treat it as a compliance checklist without first customising metrics to their specific domain and user base.

Compliance and Certification Notes

Because CAN/CSA-ISO/IEC TR 9126-4-05 is a Technical Report (TR), it does not carry the same normative weight as a full International Standard. However, its adoption as a Canadian standard means that references to quality in use metrics in contracts, procurement documents, or regulatory guidelines should follow the definitions and metrics provided in this TR.

Using the TR for Contractual Compliance

When writing requests for proposals (RFPs) or software quality agreements, organisations can cite CAN/CSA-ISO/IEC TR 9126-4-05 to define:

Minimum acceptable levels for effectiveness, productivity, safety, and satisfaction
Required measurement methods and reporting formats
Boundary conditions (e.g., tasks sampled, user sample size)

Relation to ISO/IEC 25000 (SQuaRE) series

Organisations migrating from ISO/IEC 9126 to the newer ISO/IEC 25000 series (SQuaRE) should note that quality in use is now handled by ISO/IEC 25022 and ISO/IEC 25023. The metrics in TR 9126-4-05 have been largely absorbed into that framework with refined definitions. Therefore, compliance with SQuaRE implicitly covers the intent of this TR, with additional requirements for measurement repeatability and quality measurement planning.

Important: Even if your organization uses the SQuaRE series, CAN/CSA-ISO/IEC TR 9126-4-05 remains a legally applicable Canadian standard for contracts that explicitly reference the 9126 series. Always verify which edition is cited in your agreement.

Certification and Accreditation

There is no specific certification program for compliance with CAN/CSA-ISO/IEC TR 9126-4-05 alone. However, organisations seeking to demonstrate adherence can pursue:

Internal quality assurance programs that incorporate the TR’s metrics
Accreditation for usability testing laboratories (e.g., ISO/IEC 17025) using the TR as a foundation
Third-party evaluation against the SQuaRE series, which references the same quality in use concepts

Auditors evaluating compliance will typically look for evidence that:

Quality in use characteristics were defined before testing
Metrics were selected from or aligned with the TR’s catalogue
The context of use was documented
Measurement results were reported with appropriate normalisation and confidence intervals

Frequently Asked Questions

Q: Is CAN/CSA-ISO/IEC TR 9126-4-05 still current given that the ISO 9126 series has been replaced?
A: Yes, the Canadian adoption (2005 edition) remains active as a reference for legacy systems and contracts that require the original quality model. However, for new projects, organisations are strongly encouraged to adopt the ISO/IEC 25000 SQuaRE series, which consolidates and extends the quality in use metrics from this TR.

Q: What is the difference between external metrics (Part 2) and quality in use metrics (Part 4)?
A: External metrics measure the behaviour of the software when executed (e.g., response time, throughput). Quality in use metrics measure the outcomes of that behaviour for real users (e.g., task success, satisfaction). External metrics help predict quality in use, but the latter is the ultimate measure of software value in operation.

Q: Can I use the metrics from this TR in an agile user-centred design process?
A: Absolutely. The TR’s metrics are context-independent and can be easily integrated into sprint retrospectives and usability test sessions. Short, iterative measurement cycles for effectiveness and satisfaction fit naturally with the agile philosophy.

Q: Does CAN/CSA-ISO/IEC TR 9126-4-05 require specific tools to collect metrics?
A: No. The TR describes what to measure, not how to implement measurement tools. Organisations may use automated logging, video recording, screen capture, surveys, or manual observation depending on resources and context.

📥 Standard Documents Download

🔒

Please wait 10 seconds, the download links will appear after the ad loads