Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
ISO/IEC 26559:2018 — Software Engineering — Product Family — Testing Management
Systematic testing strategies for software product families and shared platforms
Introduction to Testing Management for Product Families
ISO/IEC 26559:2018 addresses one of the most challenging aspects of software product family engineering: testing management. Testing a product family is fundamentally different from testing a single product because testers must contend with the combinatorial explosion of possible product configurations, shared platform assets that evolve over time, and the need to verify both common and variable functionality across all family members.
In a software product family, the testing challenge is not linear — it is combinatorial. A family with 20 independent variation points can generate over one million possible product configurations. Smart test management is not optional; it is essential.
The standard provides a structured approach to test planning, test artifact management, test execution, and test evaluation specifically tailored to the realities of product family development. It covers domain testing (testing the platform and reusable assets) and application testing (testing specific product configurations derived from the platform).
Core Testing Processes
Domain Testing
Domain testing focuses on verifying the shared platform assets — the core architecture, reusable components, and variability mechanisms. Key activities include: test of the commonality model (ensuring shared features work correctly in all contexts), test of variability mechanisms (verifying that variation points and binding mechanisms function correctly), and test of the reference architecture (validating architectural qualities such as performance, security, and scalability across the intended scope of the product family).
Domain testing requires a mindset shift: you are not testing a product; you are testing a production capability. Every test case must be designed with the understanding that it will apply to multiple future products, not just the current release.
Application Testing
Application testing verifies specific product configurations derived from the platform. This includes: configuration validation (ensuring the selected variant combination is valid according to the variability model), product-specific integration testing, acceptance testing for each product variant, and delta testing that focuses only on the parts that differ from previously tested configurations.
| Testing Level | Domain (Platform) Focus | Application (Product) Focus |
|---|---|---|
| Unit Testing | Test reusable components in isolation with stubs for variation points | Test product-specific adaptations and extensions |
| Integration Testing | Verify component interactions through defined variation interfaces | Validate the specific configuration’s component wiring |
| System Testing | Validate platform qualities: performance, security, reliability | End-to-end product validation in target environment |
| Acceptance Testing | Verify platform meets domain requirements | Verify product meets customer-specific requirements |
Variability-Aware Testing Techniques
The standard describes several advanced techniques for handling the combinatorial complexity of product family testing:
Sample-Based Testing
Rather than testing every possible product configuration (which is often infeasible), this approach selects a representative sample of configurations for full testing. Techniques include: pairwise (covering all pairs of variant interactions), t-wise combinatorial testing, and risk-based sampling where configurations with higher risk profiles receive more thorough testing.
Delta Testing
Delta testing exploits the fact that product variants share a common core. By testing the core thoroughly once, subsequent product configurations only need to test the “delta” — the specific variant selections and their interactions. This can dramatically reduce testing effort while maintaining coverage.
Delta testing is the single most effective technique for managing testing cost in a product family. Invest in a comprehensive core test suite, and then test the deltas for each new product configuration.
Regression Testing
Platform evolution introduces the risk of regressions across all products in the family. The standard recommends a risk-based regression strategy: full regression for the platform core, targeted regression for variation points affected by the change, and sampling-based regression for unaffected areas. Automated regression test suites are essential for managing this complexity.
Without automated regression testing, platform evolution becomes prohibitively expensive. Every platform change requires re-validating all active product configurations — automation is the only scalable approach.
Test Artifact Management
A distinctive contribution of ISO/IEC 26559:2018 is its treatment of test artifacts as first-class assets in the product family. Test cases, test data, test scripts, and test results all exhibit variability and must be managed accordingly. The standard prescribes: variability-aware test design (test cases should document which variation points they cover), reusable test assets (test components that can be parameterized for different product configurations), and traceability between test artifacts and the variability model.
Automation Strategies for Product Family Testing
Test automation is not merely beneficial in product family contexts; it is an operational necessity. The combinatorial nature of product configurations makes manual testing infeasible beyond the simplest product families. The standard recommends a layered automation strategy: a core regression suite that runs on every platform commit, a configuration-specific test suite that is parameterized by the variability model, and a smoke test suite that validates each derived product configuration before release. Key automation enablers include test case generation from variability models (automatically producing test configurations that cover pairwise variant interactions), containerized test environments that can be instantiated per configuration variant on demand, and continuous integration pipelines that detect regressions at the platform level before they propagate to product-level releases. Investment in test automation infrastructure yields compounding returns as the product family scales, making it one of the highest-ROI activities in product family engineering.
Frequently Asked Questions
Q: How do we prioritize which product configurations to test?
Prioritization should be risk-based: configurations with high business value, those that exercise high-risk variation points, configurations that combine previously untested variant interactions, and configurations with known customer issues. The standard recommends combining functional risk with market risk in the prioritization model.
Prioritization should be risk-based: configurations with high business value, those that exercise high-risk variation points, configurations that combine previously untested variant interactions, and configurations with known customer issues. The standard recommends combining functional risk with market risk in the prioritization model.
Q: Can ISO/IEC 26559 be applied alongside continuous integration / continuous deployment (CI/CD)?
Yes. In a CI/CD context, the domain testing suite runs on every platform commit, while application testing is triggered selectively based on which product configurations are affected by the change. Feature toggles and configuration matrices integrate naturally with deployment pipelines.
Yes. In a CI/CD context, the domain testing suite runs on every platform commit, while application testing is triggered selectively based on which product configurations are affected by the change. Feature toggles and configuration matrices integrate naturally with deployment pipelines.
Q: What is the recommended approach for testing variability mechanisms themselves?
Variability mechanisms (e.g., plugin loaders, feature toggle infrastructure, preprocessor logic) should be tested independently of the features they enable. This includes testing mechanism correctness (does the right variant get selected?), performance (what is the overhead of the variability mechanism?), and failure modes (what happens when a variant is missing or invalid?).
Variability mechanisms (e.g., plugin loaders, feature toggle infrastructure, preprocessor logic) should be tested independently of the features they enable. This includes testing mechanism correctness (does the right variant get selected?), performance (what is the overhead of the variability mechanism?), and failure modes (what happens when a variant is missing or invalid?).
Q: How do we handle testing of third-party components used in the product family?
Third-party components introduce variability constraints that must be captured in the variability model. Testing should verify boundary conditions at the integration points, and the standard recommends establishing a certification process for third-party components that defines the required test coverage before they can be incorporated into the platform.
Third-party components introduce variability constraints that must be captured in the variability model. Testing should verify boundary conditions at the integration points, and the standard recommends establishing a certification process for third-party components that defines the required test coverage before they can be incorporated into the platform.
