Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
ISO/IEC TR 29119-11: Software Testing — Usability Testing
A comprehensive deployment package for usability evaluation in software testing
ISO/IEC TR 29119-11 extends the ISO 29119 software testing framework with dedicated guidance for usability testing. It defines a structured process for evaluating how well users can interact with a software product in terms of effectiveness, efficiency, and satisfaction — the three pillars of usability as defined in ISO 9241-11.
Usability testing is one of the highest-ROI testing activities. A single usability test with five participants typically uncovers 80% of critical usability issues. The cost of finding these issues during development is a fraction of the cost after release.
The Usability Testing Process
The standard defines a usability testing process comprising five main activities, from planning through reporting. Each activity includes specific tasks, input artifacts, and output deliverables.
| Activity | Input Artifacts | Key Tasks | Output Deliverables |
|---|---|---|---|
| Usability Test Planning | Test objectives, user profiles, system under test | Define test objectives, identify participant profiles, select test methods, prepare test environment, develop test scenarios and tasks | Usability Test Plan |
| Participant Recruiting | User profiles, recruitment criteria | Screen and recruit participants matching target user profiles, schedule sessions, obtain informed consent | Participant Schedule, Consent Forms |
| Test Execution | Test plan, scenarios, test environment | Conduct test sessions (moderated or unmoderated), collect observations, record screen and audio, log issues and deviations | Session Recordings, Raw Observation Log |
| Data Analysis | Observation logs, recordings, metrics data | Calculate usability metrics, identify critical incidents, categorize issues by severity, perform statistical analysis if applicable | Data Analysis Report, Issue List (prioritized) |
| Reporting | Analysis results | Document findings, provide recommendations, present metrics, communicate severity ratings and business impact | Usability Test Report, Executive Summary |
A common mistake in usability testing is testing with the wrong participants. Testing with colleagues or friends who are familiar with the system will yield inflated usability scores and miss real-world issues. Always recruit participants who match the actual target user profile — ideally people who have never seen the product before.
Usability Metrics and Evaluation Techniques
ISO/IEC TR 29119-11 defines a comprehensive set of usability metrics aligned with the ISO 9241-11 quality model. The three primary dimensions are effectiveness (can users complete tasks?), efficiency (how much effort is required?), and satisfaction (do users feel positive about the experience?).
Key metrics specified in the standard include: Task Success Rate (binary completion per task), Time on Task (efficiency), Error Rate (number of errors per task), SUS Score (System Usability Scale, a standardized satisfaction questionnaire), and NASA-TLX (task load index for perceived workload). The standard also covers qualitative techniques such as think-aloud protocol, co-discovery, and retrospective probing.
Accessibility Testing Integration
The Deployment Package explicitly addresses the integration of accessibility testing (WCAG compliance) with usability testing. Accessibility issues affect not only users with disabilities but also users in challenging environments (bright sunlight, noisy surroundings, limited connectivity). The standard recommends incorporating accessibility checks — screen reader compatibility, keyboard navigation, color contrast, and captioning — into the usability test scenarios rather than treating them as a separate activity.
A healthcare software team conducted usability testing following ISO/IEC TR 29119-11 during the design phase of a clinical dashboard. By testing with five clinicians before writing a single line of production code, they discovered that the proposed data visualization was difficult to interpret under time pressure. The redesign — simplifying charts and adding configurable thresholds — reduced the average task completion time from 45 seconds to 12 seconds, directly improving clinical decision speed.
FAQs
Q: How many participants are needed for valid usability testing?
A: Research by Nielsen Norman Group shows that 5 participants per user group uncover approximately 80% of usability issues. For critical systems (medical, aviation, financial), 8-10 participants per group are recommended to achieve higher confidence.
A: Research by Nielsen Norman Group shows that 5 participants per user group uncover approximately 80% of usability issues. For critical systems (medical, aviation, financial), 8-10 participants per group are recommended to achieve higher confidence.
Q: What is the difference between usability testing and user acceptance testing (UAT)?
A: UAT verifies that the system meets contractual or regulatory requirements from the user’s perspective. Usability testing evaluates the quality of user interaction — ease of use, efficiency, satisfaction. UAT asks “does it work?” while usability testing asks “can users work with it effectively?”
A: UAT verifies that the system meets contractual or regulatory requirements from the user’s perspective. Usability testing evaluates the quality of user interaction — ease of use, efficiency, satisfaction. UAT asks “does it work?” while usability testing asks “can users work with it effectively?”
Q: Can usability testing be automated?
A: Some aspects can be partially automated — heatmap tracking, clickstream analysis, A/B testing — but the core of usability testing (observing real users performing real tasks) requires human observation and qualitative analysis. Automated tools complement but cannot replace moderated testing.
A: Some aspects can be partially automated — heatmap tracking, clickstream analysis, A/B testing — but the core of usability testing (observing real users performing real tasks) requires human observation and qualitative analysis. Automated tools complement but cannot replace moderated testing.
Q: How does 29119-11 relate to ISO 9241?
A: ISO 9241 provides the foundational definitions and principles of usability (effectiveness, efficiency, satisfaction). ISO/IEC TR 29119-11 operationalizes these principles into a testing process within the ISO 29119 testing framework. The two standards are complementary.
A: ISO 9241 provides the foundational definitions and principles of usability (effectiveness, efficiency, satisfaction). ISO/IEC TR 29119-11 operationalizes these principles into a testing process within the ISO 29119 testing framework. The two standards are complementary.
