MANTIS vision

MANTIS vision [iMAgiNg Technologies of Industry and Security vision]

Image capture and image quality are hugely important in today’s digital world. In recent years, image quality has made major leaps forward: moving from HDTV to 4k ultra-HD (and beyond) and with the introduction of High Dynamic Range (HDR) technology. At the same time, a wider spectral range has enabled 3D ranging, infrared and thermal imaging. The MANTIS project focuses on broadcasting and security systems and aims to take current imaging applications to the next level by further improving image quality (spatial and temporal resolution, higher dynamic range), combining imaging modalities, creating new functionality with Artificial Intelligence (AI) algorithms, and optimising manufacture and application workflows to reduce costs while increasing performance. In pursuit of cost and system architecture optimisation, it will also explore if, and how much, intelligence can be integrated into image sensors.

As the digital transformation proceeds, society both wants and needs high-end technologies on a large scale and at affordable cost. Image capture with the necessary level of functionality and quality has a major role to play. In combination with image analysis and AI, image capture allows for the increasing ‘smartness’ of products. Indeed, improved, affordable image capture can bring disruptive innovations that benefit almost every aspect of life, and this in turn generates major market opportunities.

Among these many applications, MANTIS will focus on image capture systems for high-end security and live broadcasting, targeting lower costs and new functionality while maintaining ease of use.

Current challenges

Video security systems play an essential role in all kinds of situations from industrial settings to ensuring public spaces are safe and secure. However, current 2D systems still depend highly on the operator’s attention span while watching the images. In the future, 3D views and AI algorithms could raise the alert to potential risks more reliably and quickly. But for this digitalisation needs to go further.

IP-based digital technologies are still not fully rolled out, partly due to cost. Plus, technologies to enhance detection and prevention of risks which use centralised data and cloud computing need improvement and cost optimisation, as do spectrum range and video analysis algorithms. And while thermal imaging could be a valuable tool for 3D imaging, it calls for advances in small uncooled sensors for infrared imaging – particularly to reduce costs.

In live broadcasting, content creators are seeking to deliver compelling experiences for consumers in a changing market. They are looking for solutions that both reduce costs and allow for content delivery on multiple platforms, so they can compete with rights owners and ‘on demand’ providers. Improved image capture and innovations such as multiple micro cameras could place audiences right in the middle of the action and even deliver ‘bird’s eye flight’ views of stadium events.

New solutions

Although these two markets are different, the technical challenges are similar. And through its cooperative approach, MANTIS will speed up the development of solutions for both.

At the core of the project is the development of a multi-modal, multi micro camera system connected to an AI-capable edge processing box. The project will also design a new micro camera architecture that supports single- and dual-modal capture and the combination of visual imaging with either NIR or LWIR (long-wave infrared). Key innovations include:

• A micro camera demonstrator with the ability to support various sensors matching 4k60 imaging capabilities in a Size Weight Power and Cost optimised implementation, targeting high-end security applications;
• A multi camera demonstrator, in which 2 or more of the micro cameras are connected to a newly defined processing box with inter-camera processing, combining visible and LWIR imaging. This processing will consist of applications such as of fusion enhanced imaging, real-time small object detection, tracking and/or re-identification for enhanced situational awareness in surveillance and high-end security and monocular depth-map generation in live sports (broadcast);
• A super 35mm, global shutter, HDR, CMOS image sensor for visual applications  with minimal colour cross-talk for better wide colour gamut representation, and a colour 2/3-inch CMOS image sensor based on the developments in the SENSATION project, both for new single sensor cameras;
• A demonstrator IC that combines small 2um global shutter pixel with high QE at NIR wavelength in-pixel HDR, low power consumption and on-board intelligence for new consumer and industrial applications;
• A LWIR thermal image sensor with updated ROIC design, fabricated CMOS wafers with MEMS-based bolometers and cap wafers.

MANTIS will also deliver new image sensor correction algorithms, a mathematical model to describe the non-linearity and temperature dependence of the NIR sensitivity of the CMOS image sensors, and a simulation tool to optimise the electro-optical (EO) system design of the multi camera demonstrator including the new micro camera architecture.

World-leading resources

Developing and optimising these technologies calls for world-class engineering and research skills. The MANTIS consortium consists of six partners from three countries, all of whom bring industry-leading expertise. Together, they combine knowledge of technologies such as optics, image capture, video processing, and power consumption- and temperature management. And they will adopt a multi-disciplinary approach to enable rapid development of the systems and components targeted by the project.

Major market opportunities

Both of the MANTIS target markets are large and expanding. The global video surveillance equipment market was expected to reach a value of USD 19.9 billion in 2019[1].  In addition, the global Electro-Optical and Infrared Systems market was expected to grow at a CAGR of around 7% during 2016-2021[2]. Meanwhile, the uncooled thermal camera market is predicted to reach a total value of USD 8.7 billion by 2026 with a CAGR of 7.2% during 2021-2026[3].

In broadcasting, IABM estimated that the market for acquisition and production (professional cameras) reached USD 435M in 2017[4]. Going forward, market experts expect that content creators will continue to move towards higher-resolution formats for sports television and cinema. Plus, despite the growth of ‘on demand’ viewing, consumption of live events will continue to increase, resulting in a continuous investment in cameras. There is also a drive to lower production costs through cloud computing / virtualisation and AI / machine learning, along with a move to automatic workflows and remote productions[5].

The project’s outcomes have potential in other expanding markets as well, such as automotive and smartphones. Indeed, the project will identify and develop new market leadership opportunities, thereby supporting the emergence of future European champions through disruptive business and / or market approaches. Crucially, by improving quality and reducing costs, it will enable the growth of applications that make life safer and bring exciting new experiences to citizens in Europe and worldwide.

[1] https://technology.ihs.com/Services/570988/video-surveillance-intelligence-service-annual

[2] https://www.marketresearchfuture.com/reports/military-electro-optical-infrared-systems-market-1287

[3] http://www.yole.fr/Thermal_Imaging_And_Sensing_Yole_Nov2021.aspx

[4] IABM DC Global Market Valuation and Strategy Report 2017, https://www.iabmdc.com/the-report

[5] https://devoncroft.com/2018/08/16/the-most-commercially-important-trends-in-broadcast-and-media-technology-for-2018/