While there have been numerous discussions by regulators worldwide, particularly the FDA and European Medicines Agency (EMA), about the regulation of devices based on Artificial Intelligence (AI), there have not been any specific rules, till this week. The EU published the first formal regulation for AI this week to lay down a uniform legal framework for the development, commercialization, and use of AI systems, including for healthcare. The regulation details the expectations and requirements that are based on about a decade of discussion and experience.
There are two primary concerns for AI-based systems: “trustworthiness” or reliability to produce expected outcomes in a fair and balanced manner, and “human centricity” to ensure that the AI system is trained to give human-like and human-favored outcomes characterized by similarity to outcomes from a human person. These aspects of AI have been discussed extensively, so these should not be onerous to follow.
The key element of this regulation is the list of prohibited practices. The prohibited practices that might impact medical devices are that (1) AI systems must not deploy subliminal techniques beyond a person’s consciousness or purposefully manipulative or deceptive techniques to distort the ability of a person to make an informed decision thereby causing them to make a decision that they may otherwise not make leading to harmful outcomes. (2) AI should not exploit any vulnerabilities of a person based on age, disability, or specific social or economic situation. (3) Based on the use of biometric data to create discriminatory or harmful situations or practices.
AI-based medical devices and systems are limited by the data they are trained on which in turn depends on the availability of diverse data. Due to the competitiveness of the industry, developers conduct limited pre-launch training and rely on ongoing training with real-world data. The EU regulations don’t explicitly prohibit this practice but do create a basis for legal action should there be an accusation or incidence of prohibited practices.
