Two years ago, the , funded by the Federal Ministry of Health and headed by TU computer scientist joint project EVA-KI, was launched. What makes it unique is that for the first time, computer scientists, doctors from two German university hospitals, experts from the medical software industry and experts on the approval of medical products are working hand in hand. Together they want to establish a platform for the development and validation of AI solutions in clinical routine. Dr. Anirban Mukhopadhyay
From static to continuous learning
When translating promising research results into clinical practice, an important hurdle must be overcome: the transition from static to continuous learning, i.e. training models throughout their entire life cycle. Continuous learning continues to gain momentum, but still has a long way to go in the health sector. Both the methods and the legal regulation need to be rethought and tested for this.
Dr. Anirban Mukhopadhyay,,
Leader of MEC-Lab at the Department of Computer Science
Medical knowledge is dynamic, always adjusting to newer knowledge, updated medical practices or when new diseases arise (e.g. the pandemic). Yet the only type of learning models that are currently allowed to help patients in US or EU are the static learning models. We created the EVA-KI consortia to take a holistic approach of bringing continual learning in healthcare, from the technical, clinical and regulatory perspectives.
Through the joint project, the researchers have very well-functioning collaborations between the technical, clinical and regulatory working worlds. They can train the algorithms with optimally prepared data and then have them tested under realistic conditions in the clinics. In return, the medical professionals receive an AI solution that is constantly improved and adapted to their needs.
Cooperation between the technical, clinical and regulatory working worlds
Mukhopadhyay and his team receive anonymised medical image data from the university hospitals, which experts have already uniformly structured with additional information using special software. For each CT scan of a pulmonary embolism, for example, they receive certain basic data as well as an evaluation of the lung lesions to be detected. All information has a predefined structure. The AI learns from this data which structures it should classify and how. Using trained AI image recognition, clinicians can also analyse new images more quickly and accurately. In turn, user statistics and user feedback help the researchers to improve the algorithms. An ideal cycle.
To ensure that the AI solutions developed can be used outside of research, the scientists are also working closely with a medical regulation company. This is to ensure that the requirements for approval in various countries are already taken into account during development. The hurdles that have to be overcome are described by the project participants on the basis of the . fictional company TuMEDa
Standardised framework Lifelong nnU-Net
In future, other researchers and clinicians should also benefit from the results of the joint project EVA-KI: With their publication , Mukhopadhyay's team presented a library of code and trained models created in the project. It was published in June 2023 in the renowned open access journal Nature Scientific Reports. “Lifelong nnU-Net: a framework for standardised medical continual learning”
The enables researchers and clinicians to perform sequential training and testing of models for continuous segmentation. By releasing their code and trained models to open-source datasets on GitHub, the Darmstadt researchers are creating a benchmark for evaluating future continuous learning methods for segmentation models. The goal of Lifelong nnU-Net is to ensure high technical standards and reproducible results. Building on the proven segmentation method nnU-Net it ensures broad applicability and lower the hurdle for the continuous evaluation of new methods. standardised framework Lifelong nnU-Net
Currently, research is very focused on improving the performance of AI models in synthetically designed test scenarios. However, these models are often unable to meet the dynamic requirements of the real world. We want to find a middle ground that allows us to safely test and develop innovative AI solutions in real clinical scenarios
Camila González, Alumna of MEC-Lab and Post-doc at Stanford University
Recently, the US Food and Drug Administration (FDA) published in place for regulating Continually learning AI as a Medical Device. It aims to support the iterative improvement of machine learning-enabled device software functions while assuring their safety and effectiveness. The EVA-KI partners will therefore continue the project with a focus on bringing together the silos of technical and regulatory affairs and will prepare several white papers. This effort will be critical to align EU AI-act with the EU Medical Device Regulations. a draft on Predetermined Change Control Plan
Joint project EVA-KI - Establishment of a platform for the development and validation of AI solutions in clinical routine
The consortium of researchers from Darmstadt, Cologne, Essen, Göttingen and Munich is led by , head of the Dr Anirban Mukhopadhyay at the Department of Computer Science at TU Darmstadt. independent research group MEC-Lab
The Federal Ministry of Health has been funding the project since the end of 2020 with 1.7 million euros as part of the digital innovations for improving patient-centred care in the healthcare system.
González, C., Gotkowski, K., Fuchs, M., Bucher, A., Dadras, A., Fischbach, R., … & Mukhopadhyay, A. (2022). Distance-based detection of out-of-distribution silent failures for Covid-19 lung lesion segmentation. Medical Image Analysis, 82, 102596.