Assuming we solve the “alignment problem” and humanity is not turned into paperclips, artificial intelligence will, likely, take a central role in the development of medical devices in the coming years. As technology advances at full pelt, FDA has tried in earnest to keep up. However, in doing so, it has created a cloud of related documents that would require, ironically, a fairly high-powered artificial intelligence system to comprehend. Since 2021, FDA has issued or held Guidance Documents, Guiding Principles, Advisory Committees, Discussion Papers, and other resources. The purpose of this and subsequent posts is to hack through the weeds and get to a deeper understanding of how FDA views this rapidly changing landscape.
Instead of deconstructing FDA’s communications chronologically, I thought it best to divide them by communication type, such as Guidance Documents, Discussion Papers, etc., starting with FDA’s January 2025 Guidance Document, Artificial Intelligence Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations. This guidance document was a joint effort between CDRH, CBER, CDER, and OCP.
I understand that this is a massive post, so for those not interested in nerding out on FDA’s requirements for AI enabled devices, here is an AI generated TLDR, or, for my Israeli audience, AMLK:
· FDA’s January 2025 guidance makes clear that AI enabled medical devices will be regulated throughout their entire product lifecycle, not simply at the point of market authorization. For innovators, this means thinking about FDA requirements from the earliest stages of product development.
· The guidance places particular emphasis on clearly describing how AI is used, what data are used to develop and validate the model, how the model was developed and tested, how users interact with the device, and how the manufacturer will monitor performance after deployment.
· FDA also expects sponsors to address issues that are particularly important for AI, including data representativeness, demographic performance, model limitations, human factors, cybersecurity, model performance changes, and the circumstances under which modifications to a device may require a new or amended submission.
· Perhaps most importantly, FDA encourages innovators to engage with the agency early through the Q Submission Program. For companies developing AI enabled devices, early regulatory planning may save significant time and money later in the development process.
· The practical takeaway is simple: AI should not be treated as an isolated feature that can be bolted onto an otherwise conventional medical device regulatory strategy. FDA expects sponsors to understand and manage the implications of AI throughout the entire product lifecycle.
Source Note – This article summarizes the FDA guidance discussed above on a section by section basis. For readability, individual citations to specific pages or sections of the guidance have not been included throughout the article. Readers should refer to the original FDA guidance for the complete text, context, and authoritative information, available here: FDA guidance document.
The guidance begins by explaining that FDA has always promoted the Total Product Lifecycle approach to overseeing the development of medical devices. To that end, this guidance intends to provide valuable information about product lifecycle management approaches along with marketing submission recommendations for AI enabled devices. Additionally, the guidance provides specific recommendations for the types of information FDA wants to see in marketing submissions, which will assist it during the review process. Finally, the guidance discusses how to address transparency and bias in AI enabled devices through the collection of relevant data on all demographic groups.
More on each of these below. However, we will start with FDA’s intended “scope” for the guidance document. The scope section includes mostly unremarkable definitions that are used throughout the guidance. However, there are some important issues to note.
First, the definition of “AI enabled device software function (AI DSF)” should be taken into consideration. FDA defines an AI DSF as a “device software function that implements one or more ‘AI models’ … to achieve its intended purpose. A model is a mathematical construct that generates an inference or prediction based on new input data.” Though not surprising, this definition casts a wide net. Market considerations almost demand that new medical devices have some AI element, making this guidance immediately relevant to many innovators.
Second, FDA provides specific definitions of different types of marketing submission types that are outside the scope of this post. However, it is important to note that FDA states that if an AI enabled device is being reviewed under a 510(k) submission, the AI enabled device “can be found substantially equivalent to a non AI enabled device with the same intended use provided, among other things, the AI enabled device does not introduce different questions of safety and effectiveness compared to the non AI enabled device.” Therefore, innovators contemplating AI enabled devices need not fear that their product’s AI function automatically takes it out of the 510(k) predicate analysis. Rather, they should focus on whether new risks are involved as compared to the predicate product.
Third, and importantly, FDA wants you to communicate with them early and often through their Q Submission Program. Many young companies “don’t want to put themselves on FDA’s radar.” However, in this case, early communication will likely save time and money.
Next, the guidance emphasizes FDA’s Total Product Lifecycle, or TPLC, approach to regulation. In broad strokes, TPLC refers to the notion that FDA does not simply review premarket submissions and then cease involvement. Rather, FDA is critically involved in the process from product ideation, bench testing and clinical studies, premarket submissions, and post market monitoring. Thus, this guidance provides recommendations for every step in the product development lifecycle.
With this preliminary information out of the way, the guidance gets into the meat of its recommendations, beginning with specific information related to a product’s Device Description that should be included in marketing submissions.
A Device Description “supports FDA’s understanding of the intended use, expected operational sequence of the device, such as clinical workflow of the device, use environment, features of the model, and design of the AI enabled device.” In sum, the Device Description is a critical element of a marketing submission because it provides FDA with key information about the context in which a given device is to be used.
The information in the “device description” section of a given marketing submission should include a clear statement that AI is used, the device’s inputs and outputs and how data are acquired, how AI contributes to the intended use and interacts with other device functions, the intended users and their required qualifications or training, the environments in which the device will be used, and the expected clinical workflow. It should also explain the device’s level of automation compared with current standard of care practices, the circumstances in which it would be used, and how its outputs support clinical decision making. Finally, the description should cover installation, maintenance, and any required calibration or configuration procedures, including when they must be performed and how users can determine whether recalibration or correction is needed. For a full description of the required submission elements, please see page 8 of the guidance.
If the user can configure elements in the device, the Device Description should also include a description of all configurable elements of the AI enabled device, such as visualizations, software inputs, model parameters, and alert thresholds, as well as how these settings can be configured and by whom. It should identify the users responsible for configuration and any required qualifications or training, explain how users can confirm which settings are selected, specify the level at which configurations apply, such as patient, clinical site, or hospital network, and describe any predefined operating points, including their outputs, performance ranges, and how they were selected based on the device’s intended use. It should also explain how different configuration choices may affect user interpretation, clinical decision making, or use of the device. For a full description of the required submission elements, please see page 9 of the guidance.
The guidance then discusses a product’s user interface design and labelling together. A user interface includes “all points of interaction between the user and the device, including all elements of the device with which the user interacts…. It also includes all sources of information transmitted by the device… training, and all physical controls and display elements… as applicable.” A full description of what constitutes labelling is beyond the scope of this post. However, section 201(m) of the Food, Drug, and Cosmetic Act defines labelling as “all labels and other written, printed, or graphic matter (1) upon any article or any of its containers or wrappers, or (2) accompanying such article.” 21 U.S.C. § 321(m).
With regard to the user interface, FDA has requested that sponsors include a comprehensive visual and functional description of the AI enabled device and its user interface, including graphical representations of the device, its components, and all screens, controls, alerts, and other elements with which users interact. It should also include a written explanation of the interface, an overview of the device’s operational sequence and expected user interactions, including user actions and corresponding device responses, as well as representative examples of the device’s outputs and reports covering a range of expected results. Where applicable, a demonstration, such as a recorded video, should also be provided to illustrate how the device operates and how users interact with it. For a full description of the required submission elements, please see page 12 of the guidance.
The Labelling section of a marketing submission for an AI enabled device should include a comprehensive description of the AI enabled device, including how AI contributes to its intended use, the model inputs and outputs, the required input preparation and compatible devices or acquisition protocols, the intended level of automation, and a high level description of the model architecture. It should also document the data used to develop and validate the model, including data sources, study sites, sample sizes, demographics, reference standards, study designs, endpoints, and performance criteria.
The documentation should report device performance metrics with confidence intervals and assess performance across relevant patient, geographic, equipment, and operating point subgroups. It should describe methods for ongoing performance monitoring, all known device and model limitations, installation and integration requirements, input data compatibility, customizable features and operating points, and any additional metrics or visualizations that provide context for the model’s output. For a full description of the required submission elements, please see pages 13 through 15 of the guidance.
The guidance then moves on to discuss unique information related to risk management activities that should be included in marketing submissions. Specifically, FDA recommends that sponsors include a complete risk management file, including a risk management plan and risk assessment, in the Risk Management File section of the marketing submission. The risk management requirements described in the guidance are surprisingly sparse. I wouldn’t take this to mean FDA isn’t fundamentally interested in risk management information. Rather, ISO 62366, ISO 14971, and other risk management principles are broadly applicable to AI enabled devices. FDA then describes required data management information to be included in the “Software Description” section of a premarket submission. This is the most expansive description of required information in the guidance, and innovators should take note.
Generally, sponsors are expected to provide information on data collection, data cleaning and processing, reference standards, data annotation, data storage and management, data independence, and the representativeness of the data used to develop and validate the AI enabled device. A useful description of each of these elements is obviously outside the scope of this post. Therefore, I highly recommend that readers refer to pages 18 through 24 of the guidance for a more complete discussion.
The guidance then takes up model description and development. Here, FDA is interested in learning about the specific model and its implicit biases in order to more deeply assess the safety and efficacy of a given AI enabled device. Generally, FDA requires that sponsors provide a detailed description of each model used in the AI enabled device, including its inputs and outputs, architecture, features, feature selection process, loss functions, and model parameters, as well as the technical elements that support any customizable model features or operating points. It should also describe quality control methods for input data and any preprocessing, post processing, data augmentation, or data synthesis techniques applied. In addition, sponsors should explain how the model was trained, including the training approach, optimization methods, regularization techniques, hyperparameters, training and tuning performance, use of pretrained models or ensemble methods, and the datasets involved. The documentation should also explain how operating thresholds were determined and how model outputs were calibrated. For a full description of the required submission elements, please see page 25 of the guidance.
The guidance then launches into a deep discussion about validation. By “validation,” FDA means “ensuring that the device, as utilized by users, will perform its intended use safely and effectively, as well as establishing that the relevant performance specifications of the device can be consistently met.” Importantly, FDA yet again shines a spotlight on human factors. Anyone paying attention to FDA’s priorities over the last several years would see that FDA views a robust human factors plan as absolutely fundamental to product development, and AI enabled devices are no exception. FDA requires validation information to be included in the Software Testing section as part of the Verification and Validation section of a marketing submission, with the specific validation methods depending on the intended use of the device.
For example, devices that estimate defined measurements typically require precision studies to assess repeatability and reproducibility. Devices that monitor time series patient data and require periodic recalibration may require stability and change tracking studies. Devices measuring less well defined patient parameters may require evidence of construct validity. Prognostic clinical decision support devices may require longitudinal data and analyses such as survival, calibration, and discrimination analyses.
The submission should include the applicable study protocols and study results supporting the device’s validation. For a full description of the required submission elements, please see pages 30 through 31 of the guidance.
Returning to the TPLC approach, the guidance provides recommendations for device monitoring as well. To that end, “[a]s part of their ongoing management of AI enabled devices manufacturers should proactively monitor, identify, and address device performance changes, as well as changes to device inputs and the context in which the device is used that could lead to changes in device performance.”
FDA requires the risk management file in marketing submissions to address how sponsors will monitor and manage changes in the performance of AI enabled devices, including methods for identifying and assessing performance changes and their potential impact on safety and effectiveness. This should include monitoring potential causes of performance changes, such as shifts in patient demographics or disease prevalence, changes in input data, data integrity issues, and changes in user behavior or demographics.
Sponsors should also describe robust software lifecycle processes that support monitoring in the deployment environment and provide a timely plan for implementing updates, mitigations, and corrective actions when performance changes occur.
The risk management approach should also address when changes may require a new or amended marketing submission, including consideration of a Predetermined Change Control Plan, or PCCP, while recognizing that applicable requirements for reporting adverse events, corrections, and removals continue to apply. For a full description of the required submission elements, please see page 34 of the guidance.
Unsurprisingly, the guidance also requires that sponsors provide robust cybersecurity related documentation. FDA requires sponsors to include, as part of the Cybersecurity and Interoperability section of marketing submissions, information addressing cybersecurity considerations unique to AI enabled devices, including relevant elements of the cybersecurity risk management report, threat modeling, risk assessment, labelling, and other supporting documentation. Sponsors should explain how cybersecurity testing addresses AI related risks, including, at minimum, malformed input, or fuzz, testing and penetration testing, and provide Security Use Case Views covering AI specific considerations. Submissions should also describe controls for protecting data and preventing leakage, including access controls, encryption, and anonymization or de identification of sensitive data. Sponsors should further address appropriate mitigations for AI specific threats such as data poisoning, forged or manipulated data, and model evasion. These may include data validation and integrity checks, anomaly detection, adversarial training, differential privacy, secure multi party computation, watermarking, continuous model performance monitoring, and strict input verification. Sponsors should also explain any relevant tradeoffs between cybersecurity protections, privacy, accuracy, utility, and model performance. For a full description of the required submission elements, please see pages 35 through 36 of the guidance.
Finally, the guidance touches on issues related to transparency of AI enabled devices in response to patient concerns related to the use of AI in their care. To that end, FDA requires sponsors to include, within the Administrative Documentation of marketing submissions, a clear statement that AI is used in the device and an explanation of how AI contributes to the device’s intended use, including how AI and non AI device functions interact where applicable. Sponsors should also describe the class and limitations of the model, provide information on the development and validation datasets, including dataset size, data sources, demographic characteristics, and comparisons between training, validation, and intended use populations, and explain how independence between training and test data was ensured. In addition, sponsors should describe the statistical confidence and uncertainty associated with model predictions and, where applicable, explain how the model will be updated and maintained over time.
In sum, the guidance is an extraordinary resource for innovators of AI enabled devices. It is highly recommended that sponsors of AI enabled devices deeply understand the contents of this guidance and refer to it in the preparation of marketing submissions. Failure to do so could result in time delays and financial loss.