In early 2020 over 100 startups and 18 groups were developing artificial intelligence systems for the medical sector in France. It is estimated that the annual growth rate will be between +30% and +50% by 2025. The AI Act, the European regulations governing the security of data used by AI systems, will also enter into force at this time, with the same enforceability as the GDPR, the general European regulations on data protection. French and European health actors must therefore prepare themselves now.
The new laws, at both the French and European level, are based on the principle of Human Oversight. The aim is to make sure that behind the algorithms and machines, human beings – in this case committees comprising medical professionals and patient representatives – must verify, at each stage of development, that the integrity of the users whose data have been collected is respected, in order to ensure that the self-learning AI systems adhere to their guidelines.
In the medical field, these regulations concern public authorities, agencies, businesses involved in the design of AI solutions or medical devices that incorporate AI.
This principle was evoked for the first time in 2021 during the revision of Article 17 of the French Bioethics Law. “This principle imposes a duty to inform patients about the use of AI in their treatment, and also requires human oversight – with health care professionals and patients’ representatives – in the design phase and the real-life application of the solution,” said David Gruson, Founder of Ethik-IA. Potential corrective actions may thus be adopted. The WHO and the FDA also recommend such human oversight.
Non-European actors working inside Europe must also comply with the new legislation.
“Respecting the principle of Human Oversight of AI is now essential for French health actors, not only because they need to plan for the European regulatory framework that will apply to them within two years, but also so that they can improve their AI solutions, based on feedback received from health care professionals and patient representatives. This logic of co-construction improves the product design and its real-life application,” David Gruson, Founder of Ethik-IA.
Ethik-IA is an academic and civic initiative that was started in 2017. It first fueled public debate as a result of over 800 contributions from professionals, before being established as a company in 2019, which allowed it to implement this approach at the operational level. Since then, eleven Human Oversight ecosystems have been developed, in sectors such as oral health, radiology, ophthalmology, genomics or pharmacovigilance.
This new certificate, which was designed in collaboration with the Digital Medical Hub (DMH), a spin-off from the AP-HP, is now targeting the hospital sector. It was launched on February 1, 2023, at the PariSanté Campus.
This certificate will apply to all health care innovations being developed by the DMH within the AP-HP, which, David Gruson explained, is “a major force for distribution.” Indeed it will serve as benchmark and basis for all public or private hospital sector professionals to designate which model is best suited to their solutions. “Ethik-IA does not replicate the device but retains the method and the process,” said David Gruson.
Such compliance is not expected to be very restrictive for the vast majority of French operators. “We are trying to leverage what we have already achieved,” said David Gruson.“Generally speaking, oversight mechanisms are in place; they simply need to be reinforced, formalized and made traceable, so that the regulator can conduct an audit.”
In February, a project was also initiated with AFNOR (Association Française de Normalisation) designed to assess compliance with the criteria of the future European regulations, before the CNIL (Commission Nationale de l’Informatique et des Libertés) assumes responsibility for the final inspection, once the text has entered into force.
