Breaking Down the FDA Draft Guidance on Lifecycle Management and Marketing Recommendations for AI-Enabled Devices

The FDA’s draft guidance, “Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations,” supports a Total Product Lifecycle (TPLC) approach to overseeing AI-driven medical devices. It outlines recommendations for marketing submissions, documentation requirements, and TPLC considerations for AI-enabled devices, emphasizing transparency and bias reduction throughout the lifecycle.

AI-Enabled Devices Software Functions: Defined

AI-enabled devices incorporate one or more AI-enabled device software functions (AI-DSFs), which are software functions using AI models to fulfill the device’s intended purpose. These devices are subject to FDA’s regulations on device software functions defined in the FD&C Act.

General Principles of a TPLC Approach

The FDA’s TPLC approach ensures safety, effectiveness, and compliance. Key principles include:

• Comprehensive Risk Management: Covers the entire lifecycle, with a focus on risk management from design to maintenance.
• Transparency and Bias Mitigation: Ensures accessible and understandable information, with efforts to control bias through representative data and subgroup evaluations.
• Performance Management: Monitors changes in data and system performance, with a recommended Predetermined Change Control Plan (PCCP) to streamline updates.

Marketing Submission Content Recommendations

The following provides a high-level breakdown of FDA recommendations on the information sponsors should include in each section of an AI-DSF marketing submission. It is important to note that these suggestions, outlined in the draft guidance, are not legally binding at this time.

Device Description

The Device Description section provides critical details to help the FDA evaluate the device’s safety, effectiveness, and operation.

Key Elements to Include:

• General Info: Confirm AI use, describe inputs/outputs, and explain how AI achieves the intended use.
• Users & Environment: Define user roles, training, and use environments (e.g., clinical or home).
• Workflow: Detail workflow, automation level, and clinical integration of outputs.
• Maintenance: Include installation, calibration, and maintenance procedures.
• Configurable Features: Explain configurable elements (e.g., visualizations, thresholds) and their impact on decisions.
• Multi-Interface Devices: Describe all connected applications and their integration.

The above information should be included in the “Device Description” section of a marketing submission.

User Interface

The User Interface section of a marketing submission helps the FDA assess device functionality, clinical workflow integration, and risk management.

Sponsors Should Provide the Following Information:

• Graphical representations (e.g., diagrams or screenshots) of the device and its user interface.
• Written description of the device user interface.
• Overview of the operational sequence of the device and the user’s expected interactions with the user interface.
• Descriptions of workflow and user interactions.
• Examples of outputs (e.g., reports) and recorded demonstration of the device.

The above information should be included in the “Software Description” section of a marketing submission.

Labeling

The Labeling section of a marketing submission requires the inclusion of labeling information in sufficient detail in order for the FDA to determine that the proposed labeling satisfies the requirements for the type of marketing submission.

Content to Include in Submission:

• AI Use Summary: State that AI is used and explain its role, including interactions between AI and non-AI functions in multifunction devices.
• Model Inputs: Summarize inputs (e.g., signals, images) and include compatible devices, protocols, system needs, and the impact of lost inputs. Provide user instructions for preparing data aligned with validation.
• Model Outputs: Define the output and its intended use.
• Automation and Model Development: Describe automation level and provide a high-level overview of the model architecture, data sources, demographics, and validation standards.
• Performance and Monitoring: Highlight performance metrics (e.g., AUROC, sensitivity) with confidence intervals, limitations, and tools for performance monitoring.
• Installation, Customization, and Use: Offer installation instructions, system integration guidance, and details on customizable features like operating points.
• User Information: Include visualizations to contextualize outputs and patient-friendly labeling with instructions, risks, and limitations.

The above information should be included in the “Labeling” section of a marketing submission.

Risk Assessment

The Risk Assessment section of a marketing submission allows the FDA to grasp whether appropriate risks have been identified and how they are controlled, in order to ensure the device is effective and safe for use.

What to Include in Your Submission:

• Include a Risk Management File with a detailed plan and risk assessment covering the entire device lifecycle, including installation, performance maintenance, and user result interpretation.
• Address risks related to user understanding and provide explanations of risk controls, including user interface elements like labeling.
• Refer to FDA-recognized standards and guidance documents and consider usability evaluation aspects outlined in Appendix D for managing identified risks.

The above information should be included in the “Risk Management File” in the “Software Documentation” section of a marketing submission.

Data Management

The Data Management section of a marketing submission provides the FDA with a clear understanding of how the device was developed and validating, allowing the agency to evaluate the AI-enabled device’s safety and effectiveness.

Key Elements to Include:

• Describe data collection methods, including study protocols, site details, acquisition timelines, dataset limitations, and quality controls. Address pre-existing databases, real-world data (RWD) with assessments, and synthetic data with justifications.
• Explain efforts to enhance diversity and ensure study generalizability across populations and sites.
• Report data cleaning and preprocessing for training data, including quality factors, inclusion/exclusion criteria, and handling missing data, while ensuring testing data processing reflects real-world data (RWD) and aligns with the final AI-DSF.
• Explain the reference standard’s establishment, uncertainties, and handling of equivocal or missing results, including details on clinician evaluations, grading protocols, data provided, evaluation methods, blinding, qualifications, and variability assessments.
• Provide details on the data annotation process, including annotator expertise, training/guidelines, quality evaluation methods, and a plan for addressing incorrect annotations.
• Explain the data’s representativeness, covering population characteristics, test data collection sites, and any subgroup analyses, as well as how the data aligns with the intended use population and indications for use.

Data Used in Model Development: The above information regarding data used in the development of the model should be included in the “Software Documentation” section of a marketing submission.

Data Used in Performance Validation: The above information regarding data used in the performance validation should be included in the “Performance Validation” section of a marketing submission.

Model Description and Development

The section on Model and Device Design, which includes information on biases and limitations, enables the FDA to evaluate the safety and effectiveness of the AI-enabled device while determining the specifications for its performance testing.

Submissions Should Include the Following:

• Include a diagram showing how multiple model outputs combine to form the device’s final output, alongside detailed textual descriptions to help an AI practitioner replicate the model.
• Provide a comprehensive model description, covering inputs, outputs, architecture, features, feature selection, loss functions, model parameters, and customization options, if applicable.
• Explain quality control criteria for input data and any methods applied to data (pre-processing, post-processing, data augmentation), ensuring alignment with the device’s intended use.
• Detail the model development process, including training methods, optimization techniques, performance metrics, pre-trained models, ensemble methods, and calibration of output.

The above information should be included in the “Software Description” in the “Software Documentation” section of a marketing submission.

Performance Validation

The section on Performance Validation provides objective evidence that the device performs reliably and predictably for its intended use, enabling the FDA to assess both the device’s performance and its safety and effectiveness.

Your Submission Should Include:

• Validation testing should objectively assess model performance using independent datasets and evaluate the model’s robustness to anticipated changes in input data and conditions of use.
• Different validation methods are required depending on device type, such as precision studies for measurement devices, stability studies for time-series monitors, and survival analysis for prognostic decision support devices.
• Include comprehensive study protocols detailing study design, statistical analysis plans, and controls to mitigate risks to patients or users during the study.
• Study results should include pre-specified outcomes, subgroup analyses, and evaluations of repeatability and reproducibility, with justifications for any missing demographic data or deviations from the original protocol.

The above information should be included in the “Software Testing as Part of Verification and Validation” in the “Software Documentation” section of a marketing submission.

Device Performance Monitoring

The section on Device Performance Monitoring outlines your plan for detecting and addressing performance changes in the device, helping to mitigate uncertainty and support the FDA’s assessment of risk controls.

A Sponsor Should Include the Following in Their Submission:

• Sponsors should include a performance monitoring plan in their premarket submission, especially for De Novo or PMA devices, to manage risks and ensure safety and effectiveness.
• The performance monitoring plan should outline data collection and analysis methods to detect and address performance changes, including monitoring shifts in input data, patient demographics, and user behavior.
• The plan should describe lifecycle processes for monitoring, deploying updates, and taking corrective actions in response to performance changes, as well as how results and mitigations will be communicated to device users.

The above information should be included in the “Risk Management File” in the “Software Documentation” section of a marketing submission.

Cybersecurity

The Cybersecurity section outlines the security measures implemented to ensure the integrity, availability, and confidentiality of your AI-enabled device, enabling the FDA to assess cybersecurity risks and the effectiveness of your preventive measures.

Information to Include in Your Submission:

• Cybersecurity Risk Management: Include unique considerations related to AI cybersecurity in the risk management report, threat modeling, risk assessment, labeling, and other deliverables.
• Testing to Mitigate Risks: Provide evidence of appropriate cybersecurity testing for AI-specific risks, including malformed input (fuzz) testing and penetration testing.
• Data Vulnerability Controls: Describe measures to address data vulnerabilities and prevent data leakage, such as access controls, encryption, and data anonymization or de-identification.
• Threat-Specific Countermeasures: Detail methods to address AI-specific cyber threats, such as validating and cleansing data, employing adversarial training, implementing differential privacy, and adopting continuous model performance monitoring.

The above information should be included in the “Cybersecurity/Interoperability” section of a marketing submission.

Public Submission Summary

The Public Submission Summary section provides detailed information on the characteristics of your device, supporting transparency. This information is made available on the FDA website, offering the public insight into the safety, effectiveness, and intended use of the device.

What to Include in Your Submission:

• Include a clear declaration that AI is used in the device and explain its role as part of the device’s intended use, including interactions with other device functions if applicable.
• Describe the class of AI model (e.g., convolutional neural network) and its limitations within the device, alongside a comparison of training, validation datasets, and model data inputs relevant to the intended use.
• Provide statistical confidence levels for predictions, including metrics describing statistical confidence and uncertainty where applicable.
• Outline plans for updating and maintaining the AI model over time and consider using a model card to organize this and related information for clarity.

The above information should be included in the “Administrative Documentation” section of a marketing submission.

Ensure AI-DSF Compliance with RookQS

Rook Quality Systems (RookQS) supports AI-enabled medical devices by leveraging its deep expertise in Quality Management System (QMS) procedures for Software as a Medical Device (SaMD) projects. With the evolving landscape of AI and Machine Learning (AI/ML) regulations, RookQS offers regulatory guidance for AI-driven innovations, whether in development or for updating existing devices. Our approach ensures that your AI-enabled devices meet regulatory expectations while prioritizing patient safety and product effectiveness.

Let RookQS help navigate the complexities of AI medical device regulations and streamline your journey to compliance. Contact us to get started!

SCHEDULE CONSULT