SENSATION

The SENSATION project developed and demonstrated technologies and building blocks for CMOS image sensors, image sensor interface standards, video processing, video compression and transmission.

In all vision based professional applications the trend in image capture is towards higher spatial and temporal resolutions, wider colour gamut, higher dynamic range and improved image quality.

In SENSATION, 6 world-class partners cooperated to develop and optimize the next generation of technologies for professional products for use in Television Broadcast, Machine Vision and Automotive markets.

Image capture and transmission are generic technologies used in a multitude of business applications, including Production Technologies, Healthcare, Security, Automotive and Digital Entertainment (TV Broadcast, Digital Cinema, Gaming). In all vision based professional applications the trend in image capture is towards higher spatial and temporal resolutions, wider colour gamut including the use of non-visible spectral domains, higher dynamic range and improved image quality.

The overall goal of the SENSATION project was to develop technologies and improved building blocks necessary for CMOS image sensors, image sensor interface standards, video processing, and video compression and transmission. These technologies and building blocks are required to address the needs of the next generation imaging systems for the Television Broadcast, Machine Vision and Automotive markets.

To develop and optimize this next generation of technologies for professional products, a consortium with a critical mass of world class engineering and research skills is required. Neither of the partners is large enough and has the skills available to develop these technologies alone.

The SENSATION project was formed around partners that are leaders in the world-wide markets in which they operate, without being competitors. Through cooperation in this project, technologies and building blocks have been developed that would not be possible without cooperation. This allows all partners to stay in the forefront of their markets and further improve their market position.

The five elements of increased image quality. Broadly speaking, increased image quality involves five factors: resolution, bit depth, frame rate, colour gamut, and brightness. Of these, the first three are all related to pixel density, and they have all gradually evolved over time. For a long time, colour gamut has been at a standstill, but a few years ago the Rec. 2020 wide colour gamut was established. That means that the only factor left is brightness, and HDR represents an evolution over the existing technology.

Source: https://www.eizo.lv/knowledge/monitor-expertise/age-of-hdr/

Technological achievements

The figure below shows a diagram of the technological building blocks of the project and developments.

CMOS Image Sensor achievements

The project worked on next generation CMOS image sensors, including a 4K UHDTV sensor with 2.5µm Global Shutter pixels for broadcast applications, a new image sensor platform with sub-3 µm Global Shutter pixels for machine vision applications, and new High Dynamic Range pixel architectures for e.g. automotive and scientific applications. New high speed serial interfaces (up to 7.2Gbps; the fastest in the world today) have been integrated to deal with the very high data rate of the image sensors while limiting the number of I/O pins and power dissipation.

Furthermore, research has been performed on new ADC concepts, a pixel test container to allow for easier testing of new pixel concepts, and a new method of Region of Interest management for significantly faster sensor readout.

Advances in image compression

To efficiently deal with the ever-increasing pixel data rates, work has been performed on the development, evaluation, and standardisation of a new (semi-) lossless image compression method: Tico-RAW. This compression algorithm has been optimized for applications where low image latency, high image quality and low power operation are required.

New camera platform for Machine Vision

A new camera platform has been developed for use in high-end Machine Vision applications. A first camera prototype uses a 65 Mpixel CMOS imager running at a sustained speed of 70 frames per second – an industry first. The camera can accept various image sensors, supports the new high-speed sensor interface standard SLVS-EC, performs real-time pixel non-uniformity corrections, and is interfaced using the new CoaXPress V2.0 video standard.

4K HDTV Broadcast camera with Global Shutter (GS) sensor

The 2/3″, GS image sensor developed during this project has been integrated in a 4K UHDTV broadcast camera. This sensor will allow the market introduction of the first UHDTV camera for the 2/3″ optical format, offering GS in triple frame speed and offering all signals over IP.  This will allow for remote productions and distortion-free, slow-motion playback of footage shot at for example sports events.

Two Caeleste test pixel container ICs assembled into a DIL package and mounted onto an interface board. Besides of a faster turn-around time between idea and silicon implementation, this setup allows easy and flexible testing of new pixel architectures and topologies

Copyright: Caeleste

Adimec’s S-65A70 camera delivers 65 Mpixel images at a sustained speed of 70 frames per second. The new CXP-12 video interface uses 4 coaxial cables for power, communication, triggering, and video data transfer at 5 Gpixel/second.

Copyright: Adimec

Grass Valley’s highest performing live broadcast camera. LDX 150 NativeIP cameras use a new 2/3-inch Xenios imager – developed in SENSATION – to capture Ultra High Definition (UHD) High Dynamic Range (HDR) images at 3X high speed for slow motion replay.

Copyright: Grass Valley

Market Potential:

Grass Valley will launch a new 4K UHDTV camera for 2/3″optics during the second half of 2021. The image sensor with global shutter pixels and the triple speed acquisition capability makes the camera a perfect choice for e.g. sports events. Field tests are done with multiple key customers worldwide, and negotiations are ongoing for the use of this new camera at different big sports events.

At the end of 2020, Adimec introduced a new camera platform for high-speed, high-resolution Machine Vision applications like semiconductor manufacturing and display inspection. The first version of this product uses a 65 Mpixel sensor, while versions using 21 and 150 Mpixel sensors are under development. This new line of products will further strengthen Adimec market leader position and result in a yearly double-digit revenue growth.

For Caeleste the new, fast ADC has in the meantime become a standard IP block that is being included in new chip designs for various customers. The pixel test container is being rolled out as general monitoring tool and being added to practically every IC tape-out, allowing a continuous evaluation of pixel modifications and new architectures.  It also allows Caeleste to test much faster innovative pixel concepts; this approach is being used in 2 other pixel validation concepts.

intoPIX has launched the new lightweight TICO-RAW codec. TICO-RAW IP blocks enables to directly compress the RAW Bayer data coming from the sensor. It allows to efficiently capture, transmit, store, and analyse the RAW Bayer data with a much smaller bandwidth and file size than regular RAW while preserving its quality and flexibility. TICO-RAW uses a lightweight and low power processing that needs only few micro-seconds of latency. It can be integrated in any type of cameras for human or vision applications:  Live IP production, DSLRs, Advanced driver-assistance systems (ADAS), machine vision, medical systems (microscopes, endoscopy cameras, etc.).

(source : https://www.intopix.com/tico-raw)

For onsemi the SENSATION project developed image sensor architecture is being used in multiple image sensor products under development. Those will be used to strengthen the portfolio, targeting various Machine Vision markets. In addition, the developed Global Shutter HDR 4.2um pixel is being used to expand into markets like high-end security and automotive that require the combination of Global Shutter and High Dynamic Range operation to allow for artifact-free acquisition of fast moving objects under difficult lighting conditions.

Societal & Economic Impact:

Machine Vision is an important technology for modern production lines. To keep up with the ever-increasing demands for higher image resolution and speed, the project worked on all aspects of the image acquisition chain: CMOS image sensors, video processing, image compression and video transmission, to make the next generation production lines more flexible, more accurate and faster than before. The technologies developed have been integrated in a new camera platform for high-end Machine Vision applications and were successfully integrated and tested by customers in the field of manufacturing of high value consumer goods like computer displays, tablets and mobile phones. Although the production of these consumer goods mostly take place in the far east, critical sub-systems of the production equipment are developed and manufactured in Europe!

Apart from the benefits identified for each of the partners, the project strengthened the position of Europe in the field of development and production of CMOS image sensors. Through the successful cooperation between the project’s partners, important knowledge has been shared which improved the competitiveness of all.

Collaboration on individual technical issues has enabled partners to mature their designs faster, and to find root causes for / potential problems in an early stage.

Patents/Standardisation/ Publications:

Protection of the IP developed during the project is of key importance to the partners. During the project in total 14 patents have been granted, filed or are still under preparation.

Partners have been active in many standardisation groups: SMPTE, AIMS, EMVA1288, CoaXPress, emVision, GenICam and SLVS-EC. Partner intoPIX co-chairs the JPEG XS work group and works on the addition of TICO-RAW to the JPEG-XS standard.

To disseminate the results of the project the partners organised 2 workshops, published and presented several papers during conferences, exhibited during trade shows, and performed product demonstrations and field tests at many potential customers world-wide.

Finally, research of Delft University of Technology will result in a PhD thesis planned for 2023.

Future Developments:

R&D performed in the SENSATION project is an excellent basis for new products that will be launched by the commercial partners to fully harvest the market potential, and follow-up research in the years to come.  For example, TUD will continue with the project, and at the end of 2021 a new tape-out is planned for the sensor with a high-speed region-of-interest readout.

The good results of the project and the excellent cooperation made the partners decide to submit a follow-up PENTA project proposal: IMAGINATION. For this project the existing team will be extended with three new partners extending the machine vision value chain.