Software as a Medical Device (SaMD) Audit Challenges

Software as a Medical Device (SaMD) is no longer an emerging vocabulary in 2022. With the rapid development of medical digitalization, medical devices are no longer just physical “devices”, but more in the form of pure software; performing clinical evaluation, diagnosis, surgical planning, and even navigation during the operation and follow-up care after the operation. Among the medical devices that integrate software and hardware, the role of software typically is to drive main functions of the hardware (i.e., firmware). However, with the gradual acceptance of information and communication technology in the medical industry, the role of software has gradually expanded to enhance connectivity and the exchange of medical data across medical device and information technology platforms. And efficiency requirements are relatively improved, so the design control level is also extended to the complete software development life cycle.

The mode of software development has also shifted with the integration of digital transformation into the development of medical devices. From the early waterfall model to the popular agile development, many development models, frameworks, and practical methods have been introduced, including Test-Driven Development and iterative incremental software development framework (Scrum). Although current regulations and guidelines do not stipulate that medical device manufacturers must develop medical devices in a specific model, their spirit is to ensure that the deliverables can meet the requirements of stakeholders; that is, safety, effectiveness, and quality. However, in actual implementation, we found that there is a slight gap between practice and regulatory requirements. The following is a summary of the challenges that we’ve faced in assisting software medical device manufacturers with establishing quality systems, and how we worked with their development teams to find appropriate solutions.

Control of technical documents

Since the development cycle of software as medical devices (SaMD) is very fast, the process will go through many iterations to integrate software modules one by one. However, there are two important regulatory requirements that will increase the cost of implementation or make compliance difficult. The first is that design control requires that every design change needs to be controlled, which may reduce flexibility in software development. Regarding this issue, we usually discuss with the development team what level of change needs to be controlled; the second is that risk management usually conducts risk assessment when there is a more complete concept (or requirement specification) , And then takes the high-risk part into development considerations. However, the requirements and specifications of the software development process will change with iterations. Therefore, how to integrate the concept of risk management into the development process will also be adjusted as the development team adopts different development models. 

Automated tools

There are many automated software in the software development process, which provide developers with automatic testing, version control or even electronic quality management systems (eQMS). The following two challenges will be faced during the delivery process: The first challenge is what information the development team should extract. We usually discuss with the team and filter the content that needs to be included in the quality management system or the delivery document. It is imperative for the quality team to get alignment with the development team on (1) how much engagement the development team should be expecting for documentation efforts, and (2) the exact work model between development and quality team. The second challenge is how to verify the effectiveness and compliance of these automated tools. This part combines the concepts of Qualification and Software Validation. We will work with the development team on each automated tool to evaluate the impact on the quality system and whether it may derive related risks, and further plans for suitable software verification methods.

Software deployment and release

The requirements of the regulations for software deployment and release are similar to those of general medical devices that require inspection reports to be attached when they leave the factory. The software covers deployment plans, test verification reports, records of unresolved anomalies, and designs. Review records and release approval records, etc. The widespread use of automated tools has followed the concept of Continuous integration/Continuous deployment (CI/CD), which can reduce human error and improve software service quality. Its automated functions include version control, software build, and deployment. We will also discuss with the development and maintenance personnel (DevOps) how to extract and organize the required information into a compliant record.

Software verification and qualification

The purpose of the verification and validation of the aforementioned automated tools is to ensure that the software can achieve the required functions and will not derive related risks. Software tools and functions are changing with each passing day. In addition to the software of the medical product itself, the regulations require non-product software used in the development process such as the above-mentioned automation tools, electronic quality management system (eQMS), and even products. The software used in the verification and manufacturing stages may initially be designed for a wide range of applications, and may not fully meet the requirements of medical device regulations. Therefore, it must be validated for functions that may potentially affect product quality or quality systems.

Clinical validation planning

The performance test of software or medical devices is similar to that of traditional medical devices, with different challenges depending on the level of risk. However, in the planning phase of clinical trials, software often faces the following scenarios, which may affect the safety and effectiveness of the evaluation, and the cost of the trial.

• The software is used on different platforms, such as mobile phones, web pages, or desktop computers. The differences in these operations may cause the results of the Usability Test to be different.
• Other medical equipment or equipment docked with the digital medical device may be of different manufacturers or different models in each hospital, and the difference in the docking method may cause the compatibility of the medical device to decrease.
• The aforementioned differences in platforms and docking medical devices, as well as the operating habits of different physicians, patients, caregivers, or different environments may lead to biases in the verification process data, so the experimental design and recruitment of usability tests are required.
• The above uncertainties may affect the final clinical indicators and the establishment of acceptance criteria. Therefore, it is not only necessary to refer to the existing standards and guidelines, but it is also necessary to be supported by relevant academic research.

Cybersecurity

Software developers often look for some commercial/licensed existing software or technologies (collectively referred to as off-the-shelf software [OTS] or software of unknown provenance [SOUP]) for functions that have been widely circulated in the market and apply them to the software itself, such as data chart generation, graphical user interface (Graphical User Interface, GUI), database management system, or machine learning library. When there are information security vulnerabilities in products or these existing software, it is usually difficult for manufacturers to confirm and inform users (doctors, caregivers, patients, etc.) of the affected medical devices, and how to mitigate the impact of the vulnerability. Therefore, the current regulatory guidelines suggest that threat modeling analysis should be carried out in the product development process to identify important assets, potential threats, potential loopholes, and control methods to mitigate the impact. Conduct information security risk analysis for the above OTS/SOUP and software functions, and list all the OTS/SOUP information used, such as use environment, version, manufacturer/developer, purpose of use, release notes and errors report (Bug Report) information.

The drastic changes in the software industry drive the evolution of regulatory agencies in terms of regulations; we are honored to be able to establish a quality system that complies with regulations, and is closest to the daily operations of the development team in this rapidly changing environment. Rook’s team of consultants focus on assisting medical device companies with establishing and maintaining their quality systems, tailoring the most suitable consulting services according to the size of the client, and the stage of product development.

If you have any questions about medical device regulations or quality systems, please feel free to reach out and schedule a complimentary consult.