This week, the FDA announced six initiatives to address various aspects of medical devices based on artificial intelligence (AI) and machine learning (ML). The programs will address major issues that should be discussed with the FDA when developing and using AI/ML technologies in medical devices.
The first initiative discusses the limitations of medical data in AI, specifically simulations used to supplement medical patient datasets with simulated or in silico data generated partially or fully using computational techniques. In silico data can be used to fill gaps in the data from various patient population distributions and imaging conditions when such data from patients is hard to obtain due to high acquisition costs, low disease prevalence rates, and other reasons. FDA researchers will conduct four projects to better understand this issue.
The second program intends to identify and measure AI bias specifically to analyze training and test methods to understand, measure, and minimize bias, and characterize performance for subpopulations. There will be three projects under this program. The Third program intends to evaluate methods for performance assessment and uncertainty quantification. This program will develop tools for the selection of appropriate metrics in the assessment of AI-enabled device performance to address the often high uncertainty or variability of the reference standard used for assessment. Further, this program will develop methods and tools to quantify this uncertainty, and, if applicable, convey it in the device output to users, and measure its effect on users. Another program will develop statistical methods and tools that can be used to design studies that will continually measure the performance of evolving algorithms while addressing issues with the re-use of evaluation datasets.
The last two newly announced programs will run projects to improve regulatory evaluation of new AI technologies and post-market monitoring. Regulatory applications for new AI/ML-enabled devices frequently use a combination of multiple types of data sources leading to questions about data harmonization and missingness. This program will run four projects to understand and create tools that the FDA will use for the regulatory assessment of such devices. Finally, three projects will develop tools for post-market monitoring of AI-enabled devices to address changes in the data acquisition systems and out-of-distribution data that a model has not encountered during model development.
Although these new programs are intended to educate FDA researchers and reviewers to assess common and newly discovered issues with AI/ML-enabled devices, developers of such medical devices can learn a lot by following the FDA’s progress in these programs to preempt future regulatory issues with the approval of their devices.
