QA/RA Implications of the New ‘Predetermined Change Control Plans’ Guidance

CDRH finally issued Draft Guidance on Pre-determined Change Control Plans for AI/ML-enabled medical devices or AI/ML-enabled device software functions, including software functions that are part of or control hardware medical devices, after authorizing 500+ AI/ML-enabled medical devices to date. Rook Quality Systems submitted feedback on the proposed regulatory framework to AI/ML-based SaMD back in 2019, and some of our feedback was reflected in the 2021 Action Plan.

This much-anticipated guidance aims to guide manufacturers of AI/ML-enabled medical devices to safely, effectively, and rapidly modify, update, and improve in response to new data, re-training practices, algorithm update procedures, and performance criteria improvement throughout the total product lifecycle (TPLC). Please stay tuned to our upcoming AI/ML webinar series (subscribe to our newsletter here) to learn more about the practical considerations of implementing changes to AI/ML modules during the TPLC, based on our experience working with the FDA and other regulatory authorities.

What is a Predetermined Change Control Plan (PCCP)?

The Predetermined Change Control Plan (PCCP) refers to a plan that includes device modifications that FDA allows in the initial marketing submission to provide a means to implement modifications and to avoid a new marketing submission (e.g., premarket approval supplement, De Novo submission, or a new premarket notification). Please note that FDA defined certain minor modifications that would not require a new submission in the previous guidance, “Decide when to submit a 510(k) for a Software Change to an Existing Device.”

Scope of PCCP Guidance

The scope of the draft guidance applies to machine learning-enabled device software functions (ML-DSFs) whose modifications to the ML model are implemented automatically and manually. For instance, the scope of your PCCP might only cover a function that is enabled by ML, but not the rest of the software modules. Refer to this guidance “Multiple Function Device Products: Policy and Considerations” to understand how to define a medical device function. In addition, from our experience working with the FDA, the FDA has always been very conservative on authorizing adaptive algorithm which automatically implements modifications on their own. This is a particularly intriguing area we need to pay close attention to on the specific boundaries FDA allows manufacturers to draw in authorizing this type of adaptive algorithm when this guidance is finalized.

Application of the Draft Guidance

In the proposed regulatory framework for AI/ML SaMD back in 2019, PCCP should constitute SaMD Pre-Specification (SPS) and algorithm change protocol (ACP). Instead, this draft guidance specifically states the following three elements need to be included in a PCCP:

1. Description of Modification

Define the ‘range’ of FDA-authorized specifications for the characteristics and performance of the planned modifications of the device, around the initial characteristics and performance of the device. All the planned modifications must be specific, and that can be verified and validated, and each modification should be linked to a specific performance evaluation activity within the modification protocol.

The Description of Modifications should also clearly specify if the proposed modifications will be implemented in a uniform manner across all devices on the market (sometimes referred to as homogenous or global changes, or global adaptations) or implemented differently on different devices on the market based on, for example, the unique characteristics of a specific clinical site or individual patients (sometimes referred to as heterogeneous or local changes, or local adaptations). For local adaptations, the Description of Modifications should include describing what local factors or conditions warrant a local change.

2. Modification Protocol

The Modification Protocol describes the methods that will be followed when developing, validating, and implementing those modifications, to ensure the device remains safe and effective. The methods described in the Modification Protocol should be consistent with and support the modifications outlined in the Description of Modifications. For each planned modification provided in the Description of Modifications, FDA recommends that manufacturers should follow their risk management processes to develop a Modification Protocol that considers each modification from four aspects:

• Data Management Practices
• Re-training Practices
• Performance Evaluation Protocols
• Update Procedures

3. Impact Assessment

Assessment of the benefits and risks of implementing a proposed PCCP. It is important for manufacturers to determine whether submission of a new 510(k) is required depending on whether the change could significantly affect the safety or effectiveness of the device.

For software, failures tend to be systematic in nature, and therefore, the probability of occurrence of a software failure cannot be determined using traditional statistical methods. While it may be possible to estimate the probability for other events in the sequence, if the overall probability of occurrence of harm cannot be estimated, the estimation of risk should be based on the severity of harm alone. Stay tuned for our upcoming blog post about ISO 34971 risk assessment on AI/ML devices.

Risk Assessment in AI/ML Devices

We consider this draft guidance a huge step forward for regulating AI/ML-enabled software as medical devices. The speed of adaptation of ML modifications is well recognized by the regulator, and it was a groundbreaking attempt by the FDA to establish this framework. We anticipate that the initiatives such as good learning practice for medical device development and Digital Health Software Precertification (Pre-Cert) Program will become complementary components of the overhaul paradigm shift in the digital health landscape.

What Does This Mean for AI/ML Enabled SaMD?

Top 10 Quality and Regulatory Considerations and Implications:

1. FDA considers the PCCP to be part of the technological characteristics of the device. That being said, the modifications proposed within the PCCP would only be authorized if they don’t violate the substantial equivalence to the predicate device. Hence, in making a determination of substantial equivalence where the predicate device was authorized with a PCCP, the proposed PCCP of the subject device must be compared to the originally authorized PCCP of the predicate device cleared or approved prior to changes made under the PCCP. Given that PCCP covers various changes (e.g., new data, re-training practice, algorithm update procedure, performance criteria improvement) to the technological characteristics of the subject device, we would imagine the industry asks for more clarification and rationale behind this proposal by the FDA.
2. Manufacturers can make multiple changes without having to submit a new 510(k) under the authorized PCCP, but each time they make a change, the modified device should be compared to the original device (i.e., the device described in their most recently cleared 510(k) for the device, to their legally marketed preamendment device, or to their device that was granted marketing authorization via the De Novo classification process.) When the authorized device is significantly modified other than as specified in the authorized PCCP, the manufacturer should submit a new 510(k).
3. FDA already stated that FDA may determine that a Modification protocol supports some but not all modifications identified in a PCCP; in such cases, only those modifications that are appropriate in the FDA’s findings of substantial equivalence or reasonable assurance of safety and effectiveness would be included in the authorized PCCP.
4. We also want to remind our audience that the visibility of proposed changes covered by labeling information (e.g., 510K summary, indications for use, device description) would become critically important. If the visibility is very low, manufacturers can embed as many modifications as possible within the PCCP without their competitors knowing. It Is safe to assume that any manufacturer can maintain a certain competitive edge by not disclosing as much information as required for a new marketing submission (e.g., additional justification or additional testing is needed if technological characteristics differences are identified because they can only predicate off of the original product). We presume that the FDA will be more clear about this in the final guidance.
5. Following the above comment, it was made clear by the FDA that identifying a PCCP in a marketing submission is very critical, and FDA has specified the approach. PCCP should be addressed in the:
   • Device description
   • Labeling required for safe and effective use of the device as such device changes pursuant to such plan (e.g., explain to the end user that the device incorporates ML and has a PCCP so that the users are aware that the device may require the user to perform software updates). The end user should be fully aware of the modifications made to the device (e.g., performance, inputs, or use), and
   • Relevant sections (e.g., 510K summary, De Novo decision summary, PMA summary, and approval order) used for determining SE or reasonable assurance of safety and effectiveness.
6. FDA recommends not to use the term ‘validation data set,’ which is typically used for algorithm tuning (e.g., architecture design decisions and hyperparameter selections). Instead, it should be called ‘tuning data set.’
7. Deviations from the authorized PCCP reviewed in the initial marketing submission could significantly affect the safety and effectiveness of the device. In such circumstances, continued distribution of the ML-DSF without submitting a new marketing submission would constitute adulteration and misbranding under sections 501(f)(1)(B) and 502(o) of the FD&C Act, respectively.
8. Q-submission is highly encouraged by the FDA to obtain FDA feedback on the proposed PCCP prior to submitting a marketing submission.
9. FDA also defined the process for implementing device modifications utilizing an authorized PCCP:
   • Determine whether the modification is consistent (i.e., modification has been specified in the Description of Modifications and has been implemented in accordance with the modification protocol) with the authorized PCCP or not.
   • If the modification is not consistent with the originally authorized PCCP, the manufacturer can submit a marketing submission to request authorization for (1) A device modification effected through a change to the authorized PCCP, (2) a device modification not implemented through a PCCP, or (3) Both.
10. The PCCP demonstrates a simple traceability table showing the mapping between modifications and the different modifications’ components (data management practices, re-training practices, etc). However, the description is very high-level and lacks detail. For manufacturers who might not have enough resources to prepare detailed planning and to develop a solid post-market MLOps pipeline, the cost for proposing and complying a PCCP might be much higher than that during evaluation in the early product planning stage.

In conclusion, we consider this guidance a huge step forward for regulating AI/ML-enabled software as medical devices. The speed of adaptation of ML modifications is recognized by the regulator, and it was a groundbreaking attempt by the FDA to establish this framework. We anticipate that the initiatives such as good learning practice for medical device development and Digital Health Software Precertification (Pre-Cert) Program will become complementary components of the overhaul paradigm shift in the digital health landscape. We foresee that there will be an increasing need for quality and regulatory experts with enough technical knowledge in the Machine Learning space to contribute to the heavy requirements in documentation and product planning.

Look to Rook for 510(k) Clearances for AI/ML SaMD Devices

RookQS’ experienced Software Quality Engineers and Software Engineers can be plugged into your organization to support these types of activities; we have successfully secured 510(k) clearances for multiple AI/ML-enabled SaMD clients. Please reach out to us to learn more about our service offerings.