IMAGINATION

IMAGINg And TransmissION
[IMAGINATION]

The IMAGINATION project developed innovations in image sensors, optical and camera architectures, and advanced video processing for Live Television Broadcast and Machine Vision Semiconductor inspection), with a European consortium optimizing these technologies.

Image capture capabilities need constant improvement to help content creators in Broadcast support the growing media consumption, and to drive improved production quality and increased productivity in Machine Vision. Project goals were:

• Design of a new two-image sensor architecture for research on higher frame rates for slow motion and wider dynamic range for better pictures, and compression to cloud or edge processing for Broadcast.
• Research high resolutions with high quality image performance and speed capabilities in cameras and frame grabbing devices for component / advanced packaging inspection in Machine Vision.
• Research applicability of 3D sensors for machine vision inspection.
• Improved CMOS image sensors: higher infrared sensitivity, lower pixel noise, and photon counting.
• Standardization.

Achievements and results of the project

Main achievements for Broadcast are:

— Doubled frame rate (from maximum 6-speed to 12-speed) for slow motion applications and improved dynamic range with at least 4 times bigger range between black and top white (2 F-stops);
— Creation of depth maps from single frames to reduce depth of field in video processing, and a control loop to keep the iris opening fixed under changing light conditions have been demonstrated;
— ticoRAW compression has been demonstrated in a broadcast camera.

In Machine Vision the results of the project will enable faster inspection of more complex components in advanced packaging, bring 3D sensors to the solution field for inspection equipment manufacturers and open up new ways to deal with data management in imaging system architectures. Main achievements include:

— Doubling data rates in vision systems from 5 to 10Gpixel/s, through the realisation of

• New camera platform with 100Gbps processing pipeline and CoaXPress interface;
• New frame grabber platform that handles 100Gbps CoaXPress connectivity;
• New 100Gbps vision platform for use in Component / Advanced Packaging Inspection.

— New image signal processing solution enabling 3D cameras for Machine Vision inspections;
— New video pipeline techniques enabling smarter data management, via

• TICO compression for resolutions up to 100 Mpixels, ultra-high-speed acquisition, grayscale, raw Bayer, colour coding as well as Region of interest coding to reduce bandwidth needed in the transmission of higher video data rates;

• World’s first implementation of new Generic Data Container (GenDC) concept within CoaXPress (proven transmission & receive link).

In image sensors technology research further closes the gap between concept and product, making more cost efficient, low light sensitive sensing (SPAD, SARACD) and increased broadband CMOS imagers (layer exchange) realistic within the coming years.
— For amongst others space applications improved specifications have been demonstrated for the design of a new Single-Photon Avalanche Diode (SPAD), a detector capable of detecting extremely faint light signals, down to individual photons, and for the design of a new Successive-Approximation Register Analogue-to-Digital Converter (SARADC) that digitizes analogue video signals using a binary search algorithm.
— First proof of concepts have proven the feasibility of the layer exchange technology to make CMOS image sensors more sensitive in infrared in a low cost process.

Background, objectives of the project and challenges

All partners are active in the image acquisition & processing value chain, delivering innovations to support market & user needs of image-based devices.
Objectives:

1. In Broadcast improve image capturing capabilities for content creators to support the growth in media consumption.
2. In Machine Vision systems help further drive improved production quality, and increased productivity.
3. In sensor technology, extend spectral sensitivity of CMOS image sensors to infrared, photon counting.

Addressed challenges

Broadcast:

1. New two-sensor camera architecture using optical white light splitter.
2. New pulse patterns and video processing to double frame rate and extend dynamic range by at least 2 F-stops.
3. Compression for cloud processing and next generation of broadcast cameras.
4. Stable monocular depth map generation for reduced depth of field.

Machine Vision:

5. High resolution cameras with temperature stabilization for absolute measurements, real-time camera architecture with sustained data rates of 10 Gpixel/s, reliable 100 Gbit/s data rate transmission. Implementation of a Standardized, transport layer neutral, self-described, Generic Data Container (GenDC) for various kinds of data, including image, meta and description data.
6. Development of a 100 Gbps frame grabber for Machine Vision targeting CoaXPress-over-Fiber camera interface supporting up to 12,5 GB/s of camera bandwidth. Extension of the CoaXPress-over- Fiber international standard to support up to 25 Gbps per connection.
7. A new vision board with an increased process capability of 25% was developed, enabling support for advanced algorithms for advanced packaging applications. This vision board creates the compatibility with the new frame grabber and the support of 100Mp camera with framerate of 100fps. It runs advanced algorithms to enable micro crack detection in die and high resolution 2D imaging with stitching capabilities to create a complete picture out of multiple field of views.
8. Generation of accurate 3D data used to be cumbersome. It is critical to address these challenges to improve overall camera speed performance, whilst improving the robustness of the data. The combination of many different system contributors and their individual dimensions poses complex trade-offs that carefully had to be analysed and creatively solved,
   whilst keeping different market requirements in focus.
9. Compression with better efficiency adapted to Machine Vision workflows. (Compression valid for AI models + implementation on embedded devices (Lattice Semiconductor FPGA & ARM CPU)).

Sensor Technology:

10. Low temperature integration of an infrared-sensitive material (Ge) onto Si photodiodes for visible and Short Wave InfraRed wavelength detection, as well as demonstrating the potential to transfer the layer exchange technology into standard CMOS image sensors.
11. Design of pixel architectures for high-speed, low-noise and High Dynamic Range applications, including avalanche Photodiode (SPADs) and below 0.5e- RMS noise integrating pixels. To enable testing of sensors at high output data rates, ultralow noise test platforms capable of handling data rates over 1Gpixels/s and on chip high-speed ADC architectures have been developed.

Technological achievements

Key technology breakthroughs realised are:
— From 5Gpixel/s to 10 Gpixel/s camera sustained data rate and from 50Gigabit/s to 100 Gigabit/s transmission via CoaXPress, GenDC on CoaXPress.
— Frame grabber capable to sustain up to 12.5 GB/s of camera bandwidth (100 Gbps from 50Gbps) based on CoaXPress-over-Fiber.
— New vision board with Bandwidth increased to 10.5GB/s and vision inspection capability of 2,5μm mould crack and 0,2μm die crack.
— Performant 3D TOF demonstrator platform including fully embedded image signal processing pipeline.
— Frame rate out of broadcast camera can be twice the maximum frame rate of the sensor, enabling up to 12-speed (SotA is 6-speed).
— Dynamic range out of the broadcast camera from 14 F-stops (SotA) to 16 F-stops.
— TicoRAW compression reaching visually lossless quality at 1.5 bit per pixel compared to 3 bit per pixel at the beginning of the project => gain of factor of 2 in compression efficiency.
— Fabrication of Ge-on-Si photodiodes using a CMOS- compatible layer exchange technology for visible and Short Wave InfraRed detection.
— A Floating Transfer Gate (FTG) pixel capable of resolving collected electrons with an accuracy ≤ 0.25 e-rms

Demonstrators’ exploitation

Broadcast:

1. TicoRAW implementation in Grass Valley camera and software decoding shows a complete broadcast workflow.
2. Further increase in dynamic range and single sensor processing will be exploited in broadcast cameras.

Machine Vision:

1. The combined demonstration of camera, frame grabber and vision system, showing the feasibility of a 10 Gpixel/s, 100 Gbps ecosystem.
2. Demo of a new vision board capable of processing 10,5GB/s receiving data from Adimec camera and Euresys frame grabber.

Sensor Technology:

1. The experimental results demonstrated that the layer-exchange technology has strong potential as a post-processing step for CMOS image sensors. Using this approach, Ge can be integrated onto Sibased CMOS image sensors both the visible and SWIR wavelength detection.
2. The Pike system is rolled out as the standard test system in Caeleste for new image sensor developments.
   Fast Pike test system will be used in future high speed image sensor developments.
   The SPAD and FTG need further tuning before valorisation is possible.

Market Potential

Broadcast:

1. Grass Valley has used the de-mosaicking and other colour filter related processing in the new single sensor camera.
2. Grass Valley will introduce constant depth of field solutions in existing products and port extended HDR results to existing single and three-sensor camera architectures. High Dynamic Range to Standard Dynamic Range (known from past HDTV) Look-Up-Table conversion has already been introduced.
3. Temporal consistency of monocular depth map generation is not yet good enough and takes too much time to process (6 seconds) to start product development for this functionality.
4. intoPIX will introduce the latest TicoRAW version to its customers in the Broadcast industry. This technology can be used for production and post-production workflows especially where AI models and Cloud processing are used.

Machine Vision:

1. The new 10 Gpixel platform drives product developments at Teledyne Adimec, supporting design in cycles in 2026+, doubling data rate speeds in vision applications. Gradually replacing current installed base solutions in various industries.
2. The new 100G frame grabber addresses new high-end inspection applications specially in Asia where the performance requirements are the most challenging with the emerging of AI, 3D packaging, and highend display technologies. The 100G frame grabber technology developed during the IMAGINATION project also allows Euresys to create a new series of
   high-performance frame grabbers.
3. As investments in AI and HPC market continues to grow with additional metrology and inspection needs: increasing package complexity with additional process steps, increased inspection and higher resolution needs, and the rapid increase in AI packaging sizes. The developments within Imagination are positioning KLA well in the growing AI market. The new vision board enables the support of new complex algorithms to detect smaller defects with limited effects on throughput which is crucial in the component inspector market that requires 100% inspection. The higher resolution module creates the opportunity to provide solutions for the upcoming CPO packages.
4. The Time Of Flight 3D demonstrator platform with embedded image processing pipeline is targeted to defend market leadership in our core automotive industry, whilst expanding the potential of this innovative LIDAR technology in other markets like Industry 4.0, robotics and machine vision markets where absolute 3D distance data availability & reliability are key differentiators. The embedded nature of the data processing significantly reduces time to market for camera makers and reduces the threshold for non-trivial players to enter the market. These achievements will help us to capitalize the investments in these growing markets.

Image Sensors:

1. The Pike and Fast Pike test systems are in use for new image sensor characterization. The advantage is that the Pike platform replaces commercial programmable test systems, costing 2 orders of magnitude more than Pike. This ameliorates considerably the test cost for small production volumes. The SPAD and the Floating Transfer Gate (FTG) pixels
   require further tuning. Caeleste expects a lot of the FTG pixels for ground- and space-based astronomy. During Space Tech Expo, Caeleste was pitching this concept. Also, in photon staved medical and life science applications, such as Fluorescence, Raman and 2-photon imaging the FTG pixel can revolutionize low-light detection. On longer term, it opens also
   opens opportunities for lab-on-a-chip solutions, where faint signals need to be detected.
2. Research on broadband CMOS image sensors is still at a low TRL level. The next advancements of the technology are planned in XECS project ENTERTAIN.

Societal & Economic Impact

IMAGINATION empowers broadcasters with advanced imaging for today’s demands – immediacy, interaction, and immersion – while IP interfacing and compression to the cloud supports remote work, cutting event-related travel emissions.
Achieved Machine Vision innovations help further drive improved production quality, and increased productivity. pipeline aid to provide easy, but accurate and reliable 3D absolute distance data to create solutions that improve people’s safety and daily comfort.
Following a successful demonstration on CMOS image sensors, the layerexchange technology can enable a CMOScompatible, cost-effective, single-chip solution for broadband CMOS image sensors.
The research on improvements for CMOS image sensors can be used for enhanced earth observation and support scientific advancement.

Patents, Standardisation, Publications

Work in the project resulted in 6 (5x GV, 1x Caeleste) patents and 1 trade secret. Teledyne Adimec, Euresys, intoPIX and Grass Valley are involved in standardisation work, including
CoaXPress, SMPTE, and AMWA-NMOS.
The project organised three ePictureThis events. Work from partners was presented on several congresses. See website https://project-imagination.eu for details.

Future Developments

Part of the consortium continues collaborating in new Xecs projects:

1. In ELEVATION Teledyne Adimec, Grass Valley, and intoPIX collaborate with new partners on transmission technologies.
2. In ENTERTAIN Grass Valley, Teledyne Adimec, TU Delft, and intoPIX collaborate on image sensor technologies, -design, and applications.
3. A new project on AI processing in Xecs call 5 is in the making.