Vivaldy

Vivaldy: VerIfication and Validation of Ai-enabLeD sYstems. The Vivaldy project addresses efficient re-verification and re-validation of (AI enhanced) healthcare systems. [Vivaldy]

An incremental upgrade cycle for healthcare systems delivers faster innovations, a better response to public health challenges, and improves the quality of care. An agile paradigm to validate safety and efficiency is still lacking to accommodate the faster rate of new product updates. Vivaldy project performs research on generic components that will enable efficient verification and validation.

Background, objectives of the project and challenges

Artificial intelligence (AI) can improve its performance when trained with more data. This new improved product would trigger more market adoption and more users, who would provide more data, which can be used to create another update. This however assumes that an update can be put on the market in a fast and efficient way, which is more challenging in heavily regulated domains such as healthcare where any significant update needs to prove its safety and effectiveness. In the Vivaldy project, researched generic components have been researched that could facilitate and accelerate this crucial validation and verification step. And not only for AI-enhanced products, but also by leveraging AI and other intelligent algorithms to evaluate updates of complex healthcare systems such as image-guided therapy systems. The first challenge is that AI can be used in many different applications and forms and an efficient solution should cater to the full range of (potential) healthcare products. To cover the broad range of challenges that relate to this topic, the consortium started with 4 complementary industry use cases, both to ensure the applicability of the developed methodologies on real products and processes and also to benchmark the resulting tools. The second challenge is that this research is not constrained to the classical technical innovations, instead diverse internal and external stakeholders have an important role, e.g. regulatory officers, testers, engineers, data privacy officers, process owners, agencies responsible for certifications, etc… Any (technical) solution needs to be accepted by all stakeholders before they can create any actual impact in the product development cycle. Based on the interactions with these stakeholders the consortium developed a set of regulatory guidelines for efficient validation and verification of updates in a healthcare system, which have been disseminated to various agencies responsible for certifications.

Technological achievements

In Philips’ Azurion systems PC components must be replaced several times during the platform’s lifetime. After such a replacement, the performance of the system must remain the same. Until now this had to be tested extensively using physical test systems. In Vivaldy, an AI-based algorithm was developped to predict the performance based on the component’s characteristics, helping first-time-right design and saving months of the testing effort.

There are hundreds of configurations of Azurion systems with different tables, arcs, software, etc. When updating applications, the impact of the update on all those configurations needs to be carefully assessed and tested. Vivaldy provided a solution to assess the impact using models and automatically run the relevant tests on a virtual system that dynamically switches to the right configuration. This reduces the need for real test systems and enables shorter update cycles.

The new monitoring tool has the capability to:

• 1. identify user groups for which the AI component underperforms,
• 2. do this based on real-world usage data where the pathology-proven diagnosis is not always available,
• 3. forecast the performance of future updates if these would be trained with more data,
• 4. compare the performance of an update with the performance of an already deployed AI.

icobrain ms harbors several machine learning (ML) models that perform pattern recognition tasks on brain MRI scans, and whose training necessitates substantial amounts of high-quality “ground truth” labels as examples. The Minimally Interactive Adjustments (MIA) framework enables rapid acquisition of additional or higher quality labels that are needed for tasks such
as brain tissue and lesion classification by allowing human experts to refine and correct the predicted labels of the most recent version of each model. As such, MIA accelerates the iterative cycle of ML model improvements.
The main achievements involve different complementary methodologies, supporting tools, and accompanying practices that reduce system verification and reverification time, effort, and cost by determining and reducing verification scope and verification automation through model-based testing. The different approaches seamlessly connect to existing development and
verification practices and processes

Market Potential

The image-guided therapy systems market that Philips targets with the innovations in Vivaldy is expected to grow from USD 4.41 billion in 2021 at a compound annual growth rate (CAGR) of 5.2% . Philips is the market leader in this domain. One of the biggest challenges in the medical domain is the long time to market for new product introductions. By creating AI-based algorithms for change impact analysis and developing a virtual test environment, Vivaldy contributed to a shorter design phase, and is expected to lead to faster validation and verification, reducing time to market by months or even over one year.

Barco is active in medical visualization solutions for radiology, mammography, surgical, dental, pathology, etc. Barco has a market share of 30-50% in those markets. The Vivaldy project is strategic because the methods and tools created in the project will allow
Barco to broaden its solutions toward more complex image processing & image analysis functionality. Moreover, the results of the Vivaldy project will also help achieve a shorter design phase and potentially shorter regulatory certification cycles, especially for more complex AI functionality which represents a fastgrowing market (CAGR of 38.1%) [2] . Finally, it is expected that this will increase competitive differentiation and help to maintain or even further extend the market share.

Societal & Economic Impact

The Vivaldy project creates impact in 3 phases.
Phase 1 already has proven economic impact today. The partners can now identify subpopulations where an AI could/should be improved and how much additional data is required to achieve a certain performance gain. This makes the development planning and related business case more reliable. Furthermore, due to the Vivaldy project a faster time to market can be achieved. For example, by selecting better potential subcomponents for updates of a complex healthcare system, the testing throughput time can be reduced by 3 months compared to the iterative trial and error approach, or by using the automated data quality inspection tools 87% fewer images need a manual inspection.
Phase 2 required more integration into the existing development ecosystem, and therefore was implemented at the end of the project. Nevertheless, these components already resulted in the early detection of 39 bugs during the project, which has not only improved the quality of the product but also resulted in a significant cost reduction as early detection is typically 10 to 100 times
cheaper than resolving these in a later stage. These software-based tests do not require scarce physical test systems, thus further reducing the time to market.

The biggest gain will come in phase 3 when the proposed regulatory guidelines will be accepted by legislation, as this would reduce the time to market from several quarters to even a year. Note that these gains are not for a single product, but for each update and as such will accumulate over many iterative updates during a product life cycle.

Patents, Standardisation, Publications

The results of the project have been disseminated to a broad audience through 4 workshops, 7 presentations, 6 publications, 5 new processes, 2 conference presentations, one press release, and the publication of an opensource software package.

Future Developments

The work initiated in the VIVALDY project will continue in a number of new projects e.g., VITO will create training for notified bodies in Europe in the context of the NoBoCAP (Notified Body Increased Capacity) project. This will enable notified bodies to better evaluate AI-based medical devices. Research activities will continue in the context of the Horizon Europe REALM (Real-worlddata Enabled Assessment for health regulatory decision-Making) project, while model evaluation and model performance monitoring services will be offered to the industry in the context of the European TEF-Health.

1 Source: https://www.grandviewresearch.com/industry-analysis/image-guided-therapy-systems-market
2 Source: https://www.globenewswire.com/en/news-release/2022/04/06/2417486/0/en/Artificial-Intelligence-MarketSize-2022-2029-Worth-USD-1394-30-Billion-Exhibiting-a-CAGR-of-20-1.html