MANTIS vision

Enhance image capture systems for broadcast, and security & safety applications, by introducing cost-efficient technology advances with ease-of-use for (end) users. [MANTIS]

Image capture system capabilities need constant improvement to help content creators support the growing need for media consumption; provide security & safety systems/operators with better information; support useability of upcoming AR/VR. Project goals:

• Develop CMOS sensor (technology) for eye-tracking in AR/VR glasses; study spectral sensitivity enhancement; and a develop performance model for electro-optical (EO) systems with the human-in-the-loop.

• Introduce single sensor broadcast camera (2/3 inch) with slow-motion capabilities to lower market.

• Improve detection & recognition in security/safety through sensing & embedded AI inference.

Achievements and results of the project

In Broadcast:

1. fast sensor operation for slow-motion recording during live television;
2. new single sensor architecture for an entry level broadcast camera;
3. generation of depth maps from single video frames.

In Security & Safety:

1. new camera with 4k image at 60 frames per second (fps) and new digital interface to replace existing HD solutions;
2. thermal imager based on new micro-bolometer sensor and specific read-out IC to enhance detection in situational awareness systems;
3. Edge AI inference to detect low resolution persons in high resolution image streams in real-time.

On sensing technology:

1. 3D stacked sensor for AR/VR;
2. new performance measurement on electro-optical systems;
3. a novel process to enhance wave-length sensitivity on CMOS imagers.

Background, objectives of the project and challenges

All partners are active in the image acquisition & processing value chain, delivering innovations to support market & user needs of image-based devices.
Objectives:

1. In Broadcast improve image capturing capabilities for content creators to support the growth in media consumption.
2. In security/safety systems, deliver operators information to do a better job and keep humans safe.
3. In Sensor technology, support useability of upcoming AR/VR applications and enable cost effective broadband imaging.

Addressed challenges:
Broadcast

1. Maximum frame rate of the 4K CMOS image sensor developed in SENSATION was increased from 3 to 6 times for better slow-motion replay.
   Challenge: changing the read-out mechanism cleverly by using a feature in the pixel architecture
2. The broadcast camera architecture has been adapted to enable use of a single 4K CMOS image sensor with color filters to create either HDTV or 4K depending on the internal processing. This enables a cheaper camera for lower market segments
3. The creation of depth maps from single frames has been researched, to enable focusing and shallower depth of field (drama shots) without the need for large format sensors (other cameras)

Security & Safety

4. The 4K60 camera required a new video pipeline and camera design allowing it to operate at twice the speed of the existing camera within the same outline and power usage
5. The thermal imager required a new readout IC to support non-cooled imaging in compact format
6. For detecting small objects (e.g. persons) in outdoor image streams based on minimal set of pixels with edge AI, new labelled datasets were needed, a multi-image (temporal) algorithm was introduced and performance evaluation in four seasons was conducted

Sensor Technology

7. To fit a small camera (1.8mm x 1.8mm) with sufficient resolution into the mount of AR/VR glasses, a 3D-stacked, miniaturized imager was successfully designed and tested
8. To extend wave-length sensitivity of CMOS sensors beyond the limit of silicon, a study was started to add SWIR sensitive materials to CMOS pixels based on the TU Delft proprietary method of ‘layer exchange’
9. To judge the performance of imaging systems as used by humans (DRI), a human performance model was extended to improve capabilities of the ‘triangle orientation detection’ model.

Technological achievements

Key technology breakthroughs realized are
— 6-speed 4K on sensor level in broadcast allowing better slow motion replay
— Very small 3D stacked global shutter pixel and sensor with high contrast and Near-infrared (NIR) sensitivity
— First enabling results on ‘layer exchange’ process research
— Low power highly compact rugged high speed security camera on CoaXPress
— Sensor agnostics Analog Signal Processor for uncooled infrared/thermal imaging
— Proven inference accuracy on small object detection through multi-image detection model

Top-5 capabilities
— Single 4K resolution sensor camera for live television broadcast
— 4K at 60fps rugged security/safety camera
— High performance miniaturized image sensor for eye-tracking
— Real-time multi camera stream edge inference for small object detection
— Proof of layer-exchange process feasibility

Demonstrators’ exploitation
Broadcast:

1. 6-speed capability will be commercialized for 6-speed HDTV in combination with a 4K output in normal frame rate.
2. Single sensor camera architecture will be commercialized in HDTV (Q1 ’25).
3. The depth map generation requires more research.

Security&Safety:

1. 4k60 rugged camera for security and spin-off products for industrial applications are introduced in 2024, design-in & production will start in 2025
2. Thermal camera with ASPIC waits for business case improvement
3. Small Object Detection labelled dataset to be extended in follow-up research

Sensor Technology:

1. Spin off sensor with pixel from CAVIAR project and MANTIS project design blocks is introduced in the market (Mira016)
2. Layer exchange process is followed up in IMAGINATION and ENTERTAIN. Exploitation expected in ENTERTAIN (2027-2028)
3. System optical performance model will be explored with 2 leads in 2025

Market Potential

Broadcast:

1. Competitors cannot output 6-speed HDTV slow motion in combination with a native 4K live output, as they read HDTV resolution out of their image sensors for 6-speed.
2. With the new single sensor camera architecture Grass Valley can address lower market segments.

Security & Safety:

1. Teledyne Adimec 4K60 will follow-up current HD and 4k30 cameras. This will be in new system designs and replacement upgrades. Spin-off cameras based on the same camera architecture to industrial applications support new cost-efficient implementations in both visible and short-wave infrared (SWIR).
2. IDEAS Thermal imager faces competition from lowcost Chinese solutions in civil applications but may offer advantages for space and defense based on western supply-chain.

Consumer and Technology:

1. Compact low power sensor for eye-tracking in AR/VR glasses will take-off with penetration of such devices in consumer. Spin-off sensors find their way into applications such as medical endoscopes, allowing ams OSRAM to leverage from new 3D stacked sensor technology.
2. Cost efficient wave-length extension on CMOS sensors will drive new applications in cost-efficient environments. Potential will become clear in the ENTERTAIN project (2027-2028).

Societal & Economic Impact

Our project empowers broadcasters with advanced imaging for today’s demands—immediacy, interaction, and immersion—while IP interfacing supports remote work, cutting eventrelated travel emissions. Realised security innovations enhance detection accuracy and safety operations by system/human operators. Miniaturized sensors drive AR/VR adoption,
potentially reducing leisure travel needs, while new CMOS processes target cost-effective imaging beyond visible light, unlocking new potential for our society.
These advancements address societal needs, foster economic growth, and contribute indirectly to environmental sustainability.

Patents, Standardisation, Publications

Work in the project resulted in 15 patents: 10 by ams OSRAM, 5 by Grass Valley. Teledyne Adimec and Grass Valley are involved in standardisation work, including SMPTE, ANWA-NMOS, CoaXPress. Project released a white-paper and organised two ePictureThis workshops on imaging technologies. Work from partners is presented on several congress. See website https://project-mantis.eu for details.

Future Developments

1. Research on depth map generation continues in PENTA IMAGINATION.
2. Collaboration Teledyne Adimec & Grass Valley on broadcast 4K sensor in Machine Vision continues in next projects and will be a vehicle to research the layer exchange process development at TU Delft.
3. In Xecs ENTERTAIN Grass Valley and Teledyne Adimec will collaborate further on CMOS image sensor development and investigate the possibility to reuse broadcast CMOS image sensor IP for the development of CMOS image sensors for Teledyne Adimec products.
4. ams OSRAM, Grass Valley, Teledyne Adimec and TU Delft will work on prototype feasibility of CMOS sensors and cameras with Layer Exchange process technology in ENTERTAIN.