V&V deep dive Part 3: Software testing
...and its unique implications for traceability
In Part 2 of this series on verification and validation, we discussed the concept of design controls traceability. When getting into the medical device software realm, verification testing gets a little more complicated, and this has implications for traceability.
IEC 62304’s testing requirements
IEC 62304 is the main standard related to the medical device software development process. While its general structure mirrors that of ISO 13485, it has some software-specific requirements that warrant attention, especially in the area of verification testing. IEC 62304 requires three different types of verification testing:
Unit testing
Integration testing
System testing
In order to understand these types of testing, we should define some important terms. A software item is defined by IEC 62304 as “any identifiable part of a computer program”. A software unit is defined as a "SOFTWARE ITEM that is not subdivided into other items". Finally, a software system is defined as an "integrated collection of SOFTWARE ITEMS organized to accomplish a specific function or set of functions".
Unit testing
Unit testing is covered in clause 5.5 of IEC 62304. It involves testing the basic functional elements of a software system - the software unit. Since the definition of a software unit offered by the IEC standard is somewhat vague, it will be up to the development team to decide the appropriate level of granularity. There have been some good discussions online about how to make this choice. There are unit tests out there that go down to the level of individual functions, and there are unit tests that cover an entire suite of functions that accomplish a discrete task. Your programming language, system architecture, and overall design can all influence how granular your unit tests are, so remember that you have some flexibility in this area.
Unit testing is a critical part of the “shift-left” testing philosophy. Testing individual software units as soon as they are ready has a few advantages. Bugs found early in the development process (before the software units have been integrated) are generally easier and cheaper to diagnose and fix. The integration of software units also might mask the presence of software unit bugs due to system conditions. Testing early contributes to the quality and safety of the medical device software.
Integration testing
In clause 5.6, the standard also requires integration testing after the various software units have been combined. It requires two types of integration testing: testing that the integration operation itself was successful, and testing that the integrated software units perform as intended. Clause 5.6.4 details exactly what “performs as intended” means, listing a few considerations such as required functionality, implementation of risk controls, and technical considerations such as correct timing, interfacing, and edge case handling.
System testing
System testing is covered in clause 5.7, and is defined as testing that the software system meets the general system requirements specified for the overall performance and safety of the software device. This is the type of testing that matches most closely to the concept of verification in ISO 13485 that would be expected of any medical device, software or not. Keep in mind that clauses 5.6.4 and 5.7.1 note that this type of testing has some overlap with the “performs as intended” integration testing, and one set of tests can satisfy both of these verification types.
Implications for traceability
While reviewing clauses 5.5 and 5.7 in IEC 62304, you might notice that for all but the lowest risk software, the standard requires design input requirements to be specified for both unit tests and system tests. This means that both of these types of testing should be performed against specific documented requirements. The standard calls these “acceptance criteria” for the unit tests, and “software requirements” for the system tests. But the idea is the same: tests should be written against design inputs that were specified before development started, not improvised or written ad-hoc.
Part 2 of this series discussed that the concept of traceability involves the documented linking of V&V tests to their corresponding design inputs and outputs. Does this mean that, for medical device software, traceability is required for both unit testing and system testing? After all, both involve testing design outputs against design inputs, and both are considered forms of verification. This is a question that has come up often when I’ve worked with SaMD products because documenting traceability adds some time and therefore cost to the development process.
Let’s look at where IEC 62304 addresses traceability specifically. Like ISO 13485, IEC 62304 only directly addresses the concept of design controls traceability when discussing design and development planning. Clause 5.1.1 states:
The (software development) plan shall address the following: TRACEABILITY between SYSTEM requirements, software requirements, SOFTWARE SYSTEM test, and RISK CONTROL measures implemented in software.
Why and how IEC 62304 differentiates between “system requirements” and “software requirements” is a topic for another post. However, we have sufficient evidence here to see that IEC 62304 is only concerned with tracing system tests to system-level requirements. The clause does not mention unit tests or unit acceptance criteria, and therefore, I believe, does not strictly require traceability to be documented between unit tests and their acceptance criteria.
Why would the authors of IEC 62304 only require documented traceability for system tests and requirements? I believe that this was probably a risk-based decision. While documenting traceability of unit tests probably has some value for retrospection, ultimately it adds quite a burden for the manufacturer. Once system-level traceability has shown that the software has met its ultimate goals related to intended use, performance, safety, usability, regulatory compliance, marketability, and so-forth, demonstrating unit-level traceability offers little value in proving that the device is ready to be put on the market.
See the other parts of this V&V series:
