What is the Software Qualification Test? The expectation is that you already have software requirements, so the goal is to check against these requirements and determine if they are fully met and correctly implemented.
Since this process is carried out shortly before delivery of the software, either directly to the customer or as a basis for the system, there are close relationships to processes such as Project Management (MAN.3), Configuration Management (SUP.8), Product Release (SPL.2) and Software Requirements Analysis (SWE.1).
If the software qualification test does not work well, errors can go undetected and customer satisfaction is going to drop. The test environment depends on the product. Examples are SIL, PIL or HIL.
The following are the most important aspects of the Software Qualification Test process in Automotive SPICE®:
-
You need a clear test strategy. Like all testing and supporting processes, software qualification testing requires the development and definition of a testing strategy. You may have a separate test strategy for each test level, but it is better to develop and coordinate the test strategy across all test levels so that all requirements are covered, and redundancies are avoided.
The test strategy should cover the following topics:
-
The test object in question
-
The methods for developing test cases and test data (e.g., development of positive/negative tests, equivalence partitioning)
-
A regression test strategy, which in Automotive SPICE® terminology means that you define how you want to retest after a bug fix or change request
-
The test environment
-
Test coverage in relation to the project plan and release plan
-
Entry and exit criteria and test interruption criteria
This process has a strong connection to Problem Resolution Management, (SUP.9), so you can use either the test strategy or the defect management strategy to document how to deal with failed tests.
-
-
Select the right test cases. The process expects a selection of test cases to be performed for the various tests, depending on goal of the test and the coverage. The aim and expectation are that for the different deliveries the software is properly tested on the basis of the afore mentioned test strategy.
The idea is that you can have deliveries with different expectations. An example for a strategy could be that you have complete coverage of all implemented software requirements for the important deliveries.
For minor deliveries, only the delta of implemented requirements since the last delivery is tested.
For this approach, of course, the right test cases must be selected.
Another possible situation for test case selection would be the regression test, which covers change requests and/or bug fixes. Here, test cases are selected that cover the change request or bug and the effects that they can have. The latter means that dependencies on requirements that can be affected by the change request or bug fix are also tested.
-
Establish traceability and consistency. This process also requires that you establish traceability between your software test cases and software requirements
Traceability can be established through hyperlinks like in DOORS, through specific traceability tools like Rectify, through traceability matrices or through other manageable means that are supported by your tool landscape.
The purpose of traceability is that it:
-
Supports consistency checks — i.e., checking the completeness and accuracy of the coverage of software requirements.
-
Supports the impact assessment in case of change requests or bugs.
-
Supports the reporting of stakeholder expectations and identify which requirements have been tested, often referred to as coverage reports.
The second part of this aspect is the consistency, in this case between software requirements and the software test cases. Checks for consistency are typically if the coverage is complete and correct. This can only be performed by a review.
If you cannot prove that your software is fully and correctly covered, you may release an insufficiently tested software, so it is in your best interest to demonstrate consistency and perform this review properly.
-
Advanced tutorial about the Software Qualification Test
What is the benefit of the Software Qualification Test?
These tests are carried out during the development of a system before the entire software is linked with the hardware and the mechanical components. This is to establish that the software works as expected and meets the software requirements.
What is the content of the Software Qualification Test process?
- The software test strategy for the tests of the overall software is defined (BP1).
- This supports all parties involved (testers, test managers, requirements engineers, project management) to develop a common understanding of how to accomplish the work (BP1).
- It must be prevented that changes to the software have negative side effects on already successfully tested requirements. For this purpose, a regression strategy needs to be provided (BP1).
- The tests that demonstrate the implementation of the software requirements are to be specified in detail (BP2).
- Test cases are selected for execution in line with the test strategy, the regression strategy, and the release plan (BP3). Tests are performed and results are recorded (BP4).
- There needs to be bidirectional traceability between software requirements, test cases and test results (BP5). The software requirements and the test cases need to be consistent (BP6). Consistency requires also that the traceability links are correct and complete. This is typically checked through reviews.
- A test summary report is produced and disseminated to the relevant people, such as project managers and customers (BP7).
Bidirectional traceability between software requirements and software qualification tests
Experiences, problems and hints
- It is very beneficial (but not mandatory) to have one common test strategy for all test processes to avoid gaps and to create synergies.
- The software qualification test is closely connected to the software requirements activities (SWE.1). The requirements engineers develop the verification criteria which comprise their expertise and hints on what should be observed when developing the tests.
- Traceability between requirements and tests is essential to be able to demonstrate coverage of requirements by tests and to monitor the test progress. This requires tool-based traceability using a good integration of the requirements tool and the test tool.
- Consistency checking is often underestimated and poorly implemented. This requires reviews which check whether the tests are correct, and the requirements are covered completely.
- Software qualification tests can be combined with any other type of test as long as one can demonstrate that the software requirements have been tested. Often the software qualification test is performed on the target hardware in conjunction with the system qualification test.
- If the project uses platform software, the tests against platform requirements and those against the project-specific requirements should be seamlessly adjusted.
Deepen your knowledge of Automotive SPICE®
If you’re serious about learning Automotive SPICE®, participate in one of our Automotive SPICE® trainings.
About the author
Bhaskar Vanamali has been working on process improvement for nearly 20 years and was secretary of working group 13 of VDA QMC.
He is a principal assessor and trainer for Automotive SPICE® and has performed more than 140 assessments and has trained more than 250 assessors.
To access the rest of this engaging content, please fill out the form below:
Please wait…