What Happens When AI Tools in Your QMS Go Unvalidated
FAI has already shown up in your quality system, and nobody had to approve a purchase order for it. Your eQMS vendor added a summarization assistant. Your PLM platform now suggests impact assessments. Engineers draft SOPs and DHF content with a generative AI tool. Someone is testing a chatbot that triages complaints. A coding assistant is writing unit tests inside your IEC 62304 lifecycle.
None of these tools is a medical device. All of them are non-product software (NPS): software used in production, the quality system, or the device development lifecycle. Regulators have expected manufacturers to validate that kind of software for intended use for decades. AI didn't remove that expectation. It made it harder to meet, because AI output is non-deterministic, vendors change models often, and AI features can show up in a platform without anyone in Quality signing off.
This post explains what the regulations require, what can go wrong when assurance is skipped, and how to handle it in a practical, risk-based way.
FFDA: QMSR and ISO 13485. The Quality Management System Regulation (QMSR), the amended 21 CFR Part 820, took effect on February 2, 2026, and incorporates ISO 13485:2016 by reference. Three clauses matter most here:
§4.1.6 requires you to document procedures for validating computer software used in the QMS, to validate that software before first use and after changes, and to scale the approach to the risk of using it.
§7.5.6 requires validation of processes, including software used in production and service provision, when you can't fully verify their output.
§7.6 requires that software used to monitor and measure requirements be confirmed for its intended application.
A generative AI tool drafting controlled documents, an assistant classifying complaints, and an AI-enabled visual inspection system all fall within these clauses. The fact that it's AI changes how you validate it. It doesn't change whether you have to.
FDA Computer Software Assurance (CSA) guidance. FDA's final guidance, Computer Software Assurance for Production and Quality Management System Software (February 2026), which supersedes the September 2025 version and aligns with the amended 21 CFR Part 820 (the QMSR), describes a risk-based approach: define intended use, determine whether a failure could create a high process risk that affects product quality or patient safety, then choose assurance activities that match the risk. It tells manufacturers to spend their effort on critical thinking rather than paperwork. It doesn't say low-risk tools need no assurance. It says the assurance should fit the risk, and that you should document the decision.
21 CFR Part 11. If an AI tool creates, modifies, or maintains electronic records you rely on for regulated purposes (an AI-drafted CAPA investigation, a complaint record, a reviewed batch dataset), Part 11 controls apply. That includes system validation (§11.10(a)), audit trails, and access controls. "The AI wrote it and someone pasted it in" isn't a record-integrity strategy.
EU MDR and ISO 13485. Under the EU MDR, Article 10 requires manufacturers to maintain a quality management system, and EN ISO 13485:2016 is the standard most organizations use to demonstrate it. Notified body auditors assess the same §4.1.6 software validation requirement, which means your AI tools will come up in audits whether or not they're on your validation list.
EU AI Act (Regulation (EU) 2024/1689). Most internal QMS AI tools aren't high-risk AI systems under the Act, but that doesn't take them out of scope. Since February 2, 2025, Article 4 has required providers and deployers to ensure their staff have a sufficient level of AI literacy. Obligations for general-purpose AI model providers have applied since August 2, 2025, and they affect what documentation you can expect from your vendors. ISO/IEC 42001, the AI management system standard, gives you a framework for governing all of this.
Skipped assurance tends to cause trouble in two ways: through findings from regulators and auditors, and through quality failures.
Inspection and audit findings. Software validation gaps are an established category of FDA observations and notified body nonconformities. If an investigator asks how you validated the AI that drafted your CAPA root-cause analyses and you have no answer, that's a finding. A device made under a quality system that doesn't conform to CGMP requirements can be deemed adulterated under section 501(h) of the FD&C Act. The possible escalation runs from Form 483 observations to warning letters, and in serious cases to import alerts or consent decrees. Pre-approval and certification audits can also slip while you fix gaps that should have been closed before go-live.
Unreliable quality decisions. AI tools can be confidently wrong. A triage assistant that misclassifies a reportable complaint as non-reportable creates a medical device reporting (MDR) problem. A summarizer that leaves out a failure mode from a design review weakens your risk file. Because outputs vary, a single successful demo tells you very little. You need acceptance criteria based on output-quality rates and task-level rubrics, not exact-match expected results.
Silent change. Vendors update models, system prompts, and default configurations, sometimes without notice. A tool that performed well at go-live may not perform the same way next quarter. Without change detection and re-assurance triggers, your "validated state" goes out of date without anyone noticing, and you lose the "after changes" control that §4.1.6 explicitly requires.
Data integrity and supplier exposure. If you don't know where prompts and outputs are stored, who can change configurations, or whether your data is used to train the vendor's model, you have open Part 11 and supplier-control questions (ISO 13485 §7.4). These are the gaps that turn up during an inspection, not before one.
Remediation costs more later. Retroactively validating a tool after months of regulated use often means reviewing the records it already touched. Doing the assurance up front almost always costs less.
Applying CSA to AI tools doesn't require writing a 300-page validation package for a meeting-notes assistant. It requires you to:
Inventory your AI tools, including AI features switched on inside existing platforms.
Write an intended-use statement for each tool and use case.
Classify risk based on how a wrong output could affect product quality, patient safety, or record integrity.
Assess the supplier: security attestations (SOC 2, ISO 27001, ISO/IEC 42001), data-use terms, and model change-notification practices.
Specify the configuration you're validating: model, version, prompts, and settings.
Run assurance activities sized to the risk, with acceptance criteria that account for non-deterministic output.
Keep the tool in a validated state through monitoring, regression evaluation, and defined re-assurance triggers.
Rook Quality Systems provides risk-based Computer System Assurance for AI tooling used in regulated medtech quality systems, covering software used in your QMS, production, development lifecycle, or regulatory submissions, but not as or in the device. Our quality engineers have a decade of computer system validation experience and use our proprietary CSA validation instrument, which streamlines the CSA process and measures the uncertainty of AI output for each use case. The work is aligned to FDA's CSA guidance, 21 CFR Part 11, the QMSR and ISO 13485, ISPE GAMP 5 (2nd ed.), ISO/IEC 42001, and the EU AI Act.
Tier 1: AI Tool Readiness Assessment. Discovery workshops, a full AI tool inventory, intended-use statements, risk classification, a review of your CSV/CSA, supplier, and Part 11 procedures, and a prioritized remediation roadmap. You'll know what you have, what's risky, and what each tool needs.
Tier 2: CSA Validation Package (per tool). An inspection-ready assurance record for one AI tool and one intended use: supplier assessment, configuration specification, a CSA plan and execution records, and a summary report with a release decision and re-assurance triggers.
Tier 3: Continuous Computer System Assurance. Telemetry-based monitoring, regression evaluations, vendor model and configuration change detection, impact assessments, and periodic reports, so your AI tools stay validated as they change.
Not sure where to start? Book an AI Tool Readiness Assessment with Rook Quality Systems. We'll help you find out which of your AI tools carry regulatory risk before an investigator or auditor does.