Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
ISO/IEC TR 25005-2:2023 — Systems and Software Engineering — Quality Measurement of Robotic Systems
SQuaRE Series: A Technical Framework for Evaluating Robotic System Quality Characteristics
Introduction to ISO/IEC TR 25005-2:2023
ISO/IEC TR 25005-2:2023 extends the well-established SQuaRE (Systems and Software Quality Requirements and Evaluation) series into the domain of robotic systems. As robots transition from controlled industrial environments to dynamic, human-collaborative settings, the need for a structured quality measurement framework becomes critical. This technical report addresses the unique quality characteristics of robotic systems — including autonomy, adaptability, safety, and human-robot interaction — which traditional software quality models inadequately capture.
Robotic systems differ fundamentally from conventional software: they operate in physical spaces, interact with humans, and must make real-time decisions under uncertainty. The quality model in TR 25005-2 explicitly addresses these dimensions.
The report defines a quality measurement framework built upon the ISO/IEC 25010 product quality model but extended with robotic-specific characteristics. It provides measurable indicators for evaluating robotic system performance across operational scenarios, covering both the software and the integrated cyber-physical system behavior.
Quality Model for Robotic Systems
The adapted quality model introduces robotic-specific sub-characteristics within the existing SQuaRE structure. Below is a summary of the key quality dimensions and their robotic-specific interpretations:
| Quality Characteristic | Robotic-Specific Sub-Characteristics | Example Metrics |
|---|---|---|
| Functional Suitability | Task completion accuracy, mission success rate | Success rate in pick-and-place, navigation error |
| Performance Efficiency | Response time, energy consumption, computational load | Inference latency, battery life under load |
| Compatibility | Interoperability with other systems, communication protocol adherence | ROS2 compatibility score, message loss rate |
| Interaction Capability | Human-robot interaction quality, collaboration safety | Collaborative task fluency, force-limited impact detection |
| Reliability | Mission reliability, fault tolerance, graceful degradation | MTBF, recovery time after failure |
| Security | Resilience to cyber-attacks, physical security, data integrity | Authentication bypass resistance, sensor spoofing detection |
| Maintainability | Module replaceability, software update capability | Hot-swap time, update rollback success rate |
| Portability | Environment adaptability, re-deployment flexibility | Reconfiguration time for new environment |
A key insight from TR 25005-2 is the emphasis on “interaction capability” as a first-class quality characteristic — acknowledging that robotic systems must be evaluated not just on isolated functionality but on how well they operate alongside humans and other autonomous systems.
Measurement Framework and Engineering Insights
TR 25005-2 defines a structured measurement approach consisting of three layers: quality measure elements (base measures), quality measures (derived measures), and quality evaluation. Each layer corresponds to a level of abstraction suitable for different stakeholders — from component engineers to system integrators.
Key Metrics and Their Practical Application
For autonomy assessment, the report introduces the “Autonomy Level Metric” which evaluates a robot’s capability to handle mission variation without human intervention. This metric ranges from teleoperation (level 0) to full autonomy (level 5), analogous to automotive SAE levels but tailored for general robotic systems. Engineers can use this metric to set clear acceptance criteria during system procurement or development.
Safety-related metrics focus on risk mitigation effectiveness. The “Collision Severity Reduction Factor” measures how effectively a robotic system reduces impact forces during unintended contact. This directly informs safety system design — from velocity scaling algorithms to compliant joint control strategies.
When applying these metrics, engineers must be aware of the context-dependency challenge. A metric valid for a warehouse robot may not apply to a surgical robot. TR 25005-2 emphasizes the need to define the “operational design domain” (ODD) before selecting quality measures.
Real-World Application: Industrial vs Service Robotics
The report distinguishes between industrial robotics applications (ISO 10218-compliant, fenced environments) and service robotics (ISO 13482, human-collaborative environments). Quality measurement priorities differ significantly: industrial robots prioritize performance efficiency and reliability, while service robots must emphasize interaction capability and safety. This distinction guides engineers in selecting appropriate sub-characteristics and weights during quality evaluation.
Frequently Asked Questions
Q1: How does TR 25005-2 relate to ISO 10218 and ISO 13482?
A: ISO 10218 and ISO 13482 are safety standards for industrial and service robots respectively. TR 25005-2 complements them by providing a quality measurement framework, whereas the safety standards focus on risk assessment and mitigation requirements. The TR references these safety standards as inputs for the safety quality characteristic evaluation.
A: ISO 10218 and ISO 13482 are safety standards for industrial and service robots respectively. TR 25005-2 complements them by providing a quality measurement framework, whereas the safety standards focus on risk assessment and mitigation requirements. The TR references these safety standards as inputs for the safety quality characteristic evaluation.
Q2: Can TR 25005-2 be used for AI-based robotic systems?
A: Yes, the framework is designed to accommodate AI/ML components. The report suggests additional metrics for AI-specific concerns such as model drift detection, explainability score, and training data quality indicators. However, detailed AI quality measurement is addressed in companion standards like ISO/IEC 25059.
A: Yes, the framework is designed to accommodate AI/ML components. The report suggests additional metrics for AI-specific concerns such as model drift detection, explainability score, and training data quality indicators. However, detailed AI quality measurement is addressed in companion standards like ISO/IEC 25059.
Q3: What tools support the quality measurement framework?
A: The report does not mandate specific tools but provides guidance on automation of quality measurement collection. Many organizations implement the framework using continuous integration pipelines that collect runtime metrics from robotic systems, combined with static analysis for software quality attributes.
A: The report does not mandate specific tools but provides guidance on automation of quality measurement collection. Many organizations implement the framework using continuous integration pipelines that collect runtime metrics from robotic systems, combined with static analysis for software quality attributes.
Q4: How frequently should robotic system quality be re-evaluated?
A: TR 25005-2 recommends quality evaluation at each major system iteration — after software updates, hardware modifications, or deployment environment changes. For continuously learning systems, the report suggests periodic re-evaluation (e.g., monthly) to detect quality degradation from model drift or mechanical wear.
A: TR 25005-2 recommends quality evaluation at each major system iteration — after software updates, hardware modifications, or deployment environment changes. For continuously learning systems, the report suggests periodic re-evaluation (e.g., monthly) to detect quality degradation from model drift or mechanical wear.
