CAVIAR

The CAVIAR project developed and demonstrated technologies and building blocks for CMOS image sensors, video processing hardware and algorithms, video data reduction and transmission. [CAVIAR]

In all vision based professional applications the trend in image capture is towards higher spatial and temporal resolutions, wider colour gamut, higher dynamic range and improved image quality. In CAVIAR, 7 partners cooperated to develop and optimize the next generation of technologies for professional products for use in Medical, Television Broadcast, Precision Agriculture, Mobile and (in cabin) Automotive markets.

Background, objectives of the project and challenges

Image capture and transmission are generic technologies used in a multitude of business applications, including Healthcare, Precision Agriculture, Security, Automotive and Digital Entertainment. In all vision based professional applications, the trend in image capture is towards higher spatial and temporal resolutions, wider colour gamut including the use of non-visible spectral domains, higher dynamic range and improved image quality. The goal of the CAVIAR project was to collaborate on these technologies to address needs in several markets.

Given the challenges of providing affordable and accessible healthcare as the world population ages and chronic diseases increase, healthcare was a key focus for the project. Improved imaging in diagnostics was researched to improve reliability (fewer false negatives/positives) and support quicker, more effective and efficient patient-focused care.

CAVIAR also improved sensor technologies in multispectral imaging. Multi-spectral imaging is central to Precision Agriculture to support sustainable practices by providing farmers with detailed information about plants, weeds and conditions at a highly local scale. This can result in reduced inputs (water, chemicals), better resource protection and increased traceability, and can help the expansion of organic agriculture. In the fiercely competitive digital entertainment market, CAVIAR’s innovations support the move to higher-resolution formats for cinema and television. The project developed image capture and replay for slow motion Ultra-High Definition TV and optimizations from image capture to transmission. These will be particularly relevant as demand for live events continues to grow worldwide. The project moved from market specific interfaces to open standards IP transmission enabling remote productions, increasing cost efficiency and reducing the carbon footprint of live television productions, as less people and equipment have to travel to the live event. For in cabin security applications in automotive the NIR efficiency of global shutter CMOS image sensors was extended.

Technological achievements

CMOS Image Sensor achievements

The project worked on next generation CMOS image sensors. For broadcast a 4K UHDTV sensor with 2.5μm global shutter pixels was designed to provide 4K streams in full resolution at 180 frames per second to create, when played back, slow motion in 4K detail. For Precision Agriculture, an existing monochrome CMOS image sensor with a pixel pitch of 4.5μm has been equipped with a 4×4 filter array to create 16 bands ranging from 400nm to 940 nm. For face recognition and authentication in mobile, as well as in cabin safety features in automotive, a 2M pixel global shutter CMOS image sensor was developed with improved near infrared sensitivity for better operation under bad lighting conditions.

Advances in image processing

To process multi-spectral images in Precision Agriculture, a deep learning algorithm was developed. A colour image containing a crop plant and a weed is given to the algorithm which returns two separate images: one of the crop plant and one of the weed. These images are then ready to be used in agricultural applications or by agricultural robots. To benefit from the full 4K UHDTV specification, High Dynamic Range processing that can handle up to 15 F-stops of video, and conversion towards the smaller dynamic range in HDTV have been researched. Also an improved focus assist algorithm was developed to help the camera operator properly focus on a display with a lower spatial resolution than that of the camera.

Video data reduction and transmission

An automated ROI segmentation of Electron Microscopy images with annotation effort of less than 5 minutes was developed, saving hours in training a model to automatically segment cellular structures of interest, speeding up the digital pathology processing pipeline. For security an edge processing pipeline was developed for high resolution, high framerate imagery in aerial surveillance on a drone with limited processing power and low transmission bandwidth. A data reduction of 99% was achieved.
To enable transmission of triple speed 4K video streams next to normal 4K and HDTV (2K) outputs on the camera towards the replay server, 100G Ethernet transmission using open IP standards was developed.

Hardware demonstrators

For all four applications hardware demonstrators have been developed that will result in commercial products or a demonstration kit for potential customers (automotive).

Market Potential

With the 4K slow motion image capture system EVS and Grass Valley have completed the basic 4K toolset for content creators to secure their service available market. The 21M pixel camera reduces the scan time of histology slides in the P1000 scanner. Allowing new and existing scanners to increase the scan capabilities in digital pathology labs. With introduction planned in 2022 by 3DHistech, the new scan capability will find its way to hundreds of systems on yearly basis. The multispectral camera from ImViA and MsEyeTech provides VIS and NIR imaging for several agricultural applications. Monitoring plants growth, optimization of growth process, and detection of weeds and diseases are the potential usages of the system. The MS camera can be mounted on a tractor to take photos of the field while tractor is being used for other agricultural operations and provide information about the situation of the field and newly-growing weeds. The NIR-enhanced global shutter image sensor with high QE at NIR allows AMS to penetrate consumer and automotive markets while keeping the lead in the industrial global shutter market. High QE at NIR allows high image quality with 5 to 10x less light power compared to existing solutions. This is key for batterypowered applications such as consumer/mobile 3D authentication, AR/VR, in- cabin driver monitor. Most industrial applications make use of global shutter.

Societal & Economic Impact

To improve Health & Well Being, CAVIAR improved diagnostic capabilities in digital pathology to support our society’s increasing need for more, better and faster
patient diagnostics at lower cost per diagnosis. The agricultural industry faces worldwide demands for higher yields and greater environmental protection. The CAVIAR technology for precision agriculture can enable production of more crop with less water and fewer chemicals. It can be used for image-guided operations in the field to help feed the world in a more economic and environment-friendly way. Broadcast television moves to ever higher resolutions and frame rates. The technology developed in CAVIAR makes this possible. Addition of slow motion to ultra HDTV will make it possible to place audiences in the middle of the action. Furthermore, the IP transmission
developed to enable remote productions will result in less staff and equipment traveling, thus lowering the carbon foot print of live television productions, and improving the social life of content creators who can be home more often.

For a secure society, CAVIAR demonstrated technology which allows to apply high-resolution cameras in drone applications. This allows for a less operator-intensive monitoring solution, leading to a more secure society. NIR-enhanced cameras for in-cabin monitoring are essential tools to detect the status of the driver and other car occupants. NIR detection is necessary especially during night driving when indoor light is off. The car processor determines if the driver is in good driving shape. If not, alert messages are sent to the driver and ultimately the car stops. Global shutter and high quantum efficiency at NIR are required, as they allow the illuminator to be active for much shorter time (hence sparing considerable power and illuminator lifetime) compared to traditional rolling shutter sensors with low NIR QE. Apart from the benefits identified above, the project has strenghtened the position of Europe in the field of CMOS image sensors, cameras and image processing. Through the successful cooperation, important knowledge has been shared to improve the competitiveness of all. Collaboration on individual technical issues enabled partners to mature their designs faster, and find root causes for potential problems in an early stage.

Patents/Standardisation/ Publications

Protection of the IP developed in the project is of key importance to the partners; in total 6 patents have been filled. Partners have been active in the following standardisation groups either contributing or actively following: SMPTE, AIMS, CoaXPress, GenICam and MIPI. Grass Valley chaired AIMS. To disseminate the results partners organised a promotion campaign in 2021 together with the Aeneas office, published and presented several papers during conferences, exhibited during trade shows, and performed product demonstrations and field tests at many potential customers world-wide, as well as online during covid-19 restrictions.

Future Developments

The R&D performed in the CAVIAR project is an excellent basis for new products that will be launched by the commercial partners to fully harvest the market potential, and for follow-up research in the years to come. The good results of the project and the excellent cooperation made part of the partners decide to collaborate in a next PENTA project called Mantis Vision, which already started mid 2021.