System Qualification Test (SYS.5)

Process ID: SYS.5
Process group: System Engineering
Automotive SPICE® is a trademark of VDA QMC.

The System Qualification Test process in Automotive SPICE® (also known as SYS.5) helps your organization establish that the integrated system meets the system requirements and can be delivered to the customer. The expectation is that system requirements are already established, so the goal is to check against these requirements and determine if they are fully met and correctly implemented.

This process is the final stage of testing and should demonstrate that the product works as intended by the developing organization and as expected by the customer. If the System Qualification Test does not work well, errors can go undetected.

System tests usually include environmental, performance and endurance tests. The test environment depends on the product, but examples include hardware-in-the-loop (HIL), vehicle tests and climate chambers.

This process is carried out shortly before delivery, and there are close relationships to processes such as Project Management (MAN.3), Configuration Management (SUP.8), Product Release (SPL.2) and System Requirements Analysis (SYS.2).

The following are the most important aspects of System Qualification Test in Automotive SPICE®:

How to design a system qualification test strategy

Like all testing and supporting processes, system qualification testing requires the development and definition of a testing strategy. A separate test strategy may exist for each test level, but it is better to develop and coordinate the test strategy across all test levels. This helps establish that all requirements are covered and redundancies are avoided.

The test strategy should cover the following topics:

• The test object in question.
• The methods for developing test cases and test data (for example, development of positive/negative tests, equivalence partitioning).
• A regression test strategy, which in Automotive SPICE terminology means defining how to retest after a bug fix or change request.
• The test environment.
• Test coverage in relation to the project plan and release plan.
• Entry and exit criteria and test interruption criteria.

This process has a strong connection to Problem Resolution Management (SUP.9), allowing the use of either the test strategy or the defect management strategy for navigating failed tests. Part of the strategy is also to define coverage targets based on the type of delivery, so there is a strong link to the project plan and the demand plan.

When documenting a test strategy, parts of it can be generic and covered in the internal process description and guidelines. Again, Automotive SPICE® does not define how and where test strategies are documented, meaning they can be distributed over several documents.

The test strategy should also cover the approach to test automation. Generally, it can be said that test automation is always better than manual testing, but there may be tests that are too expensive to automate or the test environment may not allow automation, such as vehicle maneuvers.

Test case selection for system qualification

This process also expects a corresponding selection of test cases to be performed for the various tests. The expectation is that, for the different deliveries, the system is properly tested on the basis of the previously mentioned test strategy.

The idea is that it is possible to have deliveries with different expectations. A possible strategy could be that you have complete coverage of all implemented systems requirements for the important deliveries. 

For smaller deliveries, only the delta of implemented requirements since the last delivery is tested. For this approach, the correct test cases must be selected.

Another possible situation for test case selection would be the regression test, which covers change requests and/or bug fixes. Here, test cases that cover the change request or bug, and the effects they can have, are selected. This means dependencies on requirements that can be affected by the change request or bug fix are also tested.

Traceability and consistency in system test cases

It is essential to establish traceability between system test cases and system requirements. Traceability can be established through traceability matrices or through other manageable means supported by an organization’s tool landscape.

The purpose of traceability is that it:

• Supports consistency checks, such as checking the completeness and accuracy of the coverage of system requirements.
• Supports the impact assessment in case of change requests or bugs.
• Supports the reporting of stakeholder expectations and identifies which requirements have been tested, often referred to as coverage reports.

Traceability also supports consistency

Consistency can only be proven in a review where the correct and complete coverage of system requirements can be demonstrated.

Skipping this review may lead to incomplete or faulty system test cases. Additionally, defects in the system qualification test may not be detected because this test is performed against those requirements. If the requirements are faulty, the test may not show false behavior.

Following these key points related to the System Qualification Test can help you:

• Implement a powerful system qualification strategy.
• Maintain control of the release.
• Minimize customer dissatisfaction.

Why choose UL Solutions Software Intensive Systems for Automotive SPICE® support? 

We can support automotive original equipment manufacturers (OEMs) and suppliers in: 

• Achieving the required capability levels within key development processes.
• Systematically improving existing workflows and methods.
• Evaluating the status of process improvements through formal assessments and gap analysis.
• Fulfilling the requirements of Automotive SPICE® in harmony with security, functional safety and agile methods.
• Training staff and assessors.

Learn more about the Automotive SPICE® for Machine Learning Engineering

To further your understanding of the Automotive SPICE® System Qualification Test (SYS.5), watch our video.