MIPI CSI-2’s Newest Frontier: Machine Awareness

One of the challenges we humans have is that our ability to perceive things around us and to make sense of the world relies on abstractions based upon mental constructs. It’s a limited capability. And what we have come to realize is that enabling machines to become aware of our surroundings and act on behalf of our well-being and work for us will dramatically enrich our lives. But this machine intelligence remains limited, too, in no small part because it is blind. Giving sight to machines will enable vision for real-time perception and decision making.

How does this relate to the recently released MIPI CSI-2^SM v3.0?

Initially the whole notion of camera development within MIPI Alliance was very much focused on capturing the most pristine images and video streaming for consumption by humans. And that has proven to be a tremendous, global success story. MIPI CSI-2 has completely obviated the need for us to carry around point-and-shoot cameras. Nearly every mobile phone with a camera today relies on CSI-2, the most widely used camera specification in the mobile industry.

So the frontier of development for the MIPI Camera Working Group is no longer on enabling superior image quality – that has been achieved. Over the past several years, the group’s contributors instead have focused on advancing solutions utilizing artificial intelligence (AI) and ultimately bring machine awareness through vision.

Three Critical New Features

MIPI CSI-2 v3.0 is the product of a four-year development phase exploring various use cases, such as the Internet of Things (IoT), automotive and drones.

The new version of the specification delivers three key features, which contribute to machine awareness in important ways.

Let's start with RAW-24 for representing individual image pixels with 24-bit precision. RAW-24 it is not meant for “human consumption”—it is meant to be consumed by machines. The capability is intended to enable machines to make better decisions from superior image quality. Imagine a fully autonomous car that has to navigate through very bright sunlight and then enters a very dark tunnel. With RAW-24, the vehicle can leverage very precise image quality and decipher whether darkness on an image is a harmless shadow or a pothole in the roadway to be avoided.

The second crucial new feature in CSI-2 v3.0 is Smart Region of Interest (SROI). It’s for analyzing images using inferencing algorithms and making accurate deductions. For the longest time, we have trained medical professionals to look at MRI scans to determine if there is something that requires further attention—for instance, if there is some region that perhaps maps to a tumor. Today, many of these types of manual processes are being replaced by machine inferencing algorithms. SROI is a capability for enabling, for example, medical devices to more surely recognize anomalies such as tumors in medical images—or enabling machines on a factory floor to more quickly identify potential defects on a conveyor belt. Again, the cameras with their resolutions and the type of algorithms that are running in the background are able to make these deductions far more accurately than what we are able to do as humans.

The third key new feature is Unified Serial Link (USL). Historically, a MIPI imaging conduit from an I/O perspective demands many wires. But if we want to take the camera solution we developed for mobile phones and deploy it on platforms where there are limitations like, say, routing these wires through very narrow hinges, a more efficient approach is needed. What USL does here is encapsulate imaging pixel and control transport data between an image sensor module and application processor, dramatically minimizing the number of wires needed and enabling MIPI imaging sensors to be deployed on a broad range of platforms beyond mobile form factors, such as content creation devices (notebooks, laptops, etc.), IoT systems, drones and perhaps even in-cabin monitoring for automotive use cases.

What’s Next

There’s increasing human device friction with the proliferation of connected devices. We’re working on solutions to give machines sight and enable real-time perception and decision-making through machine vision. There are select applications for which we have enabled AI to see surroundings, and it has performed remarkably well. But for the most part, the AIs that we interface with on a typical basis lack humanistic comprehension; unaware of the surroundings and unaware of emotions required for frinctionless interactions. The pathway forward is to develop an infrastructure so that these forthcoming implementations and innovations of machine intelligence can be more keenly aware of us and our surroundings in a safe manner with comprehensive provisions for privacy.

MIPI Alliance is in the business of building the infrastructure for machine awareness. Innovation of MIPI CSI-2 never stops. In fact, work is already taking place on the specification’s next version to continue to advance machine awareness through solutions including: always-on sentinel imaging, security, safety and native support for long-reach on automotive platforms using MIPI A-PHY^SM.

Join us at MIPI DevCon Taipei for several informative educational sessions that will present use cases for MIPI CSI-2.