New MIPI Camera Command Set Changes Image Sensor Design
Recent developments are making the process of integrating image sensors into devices across industrial, consumer, security and automotive markets faster and easier.
Try to find a new connected device that doesn’t have at least one embedded camera. You probably can’t. The number of image sensors in smartphones, film cameras, digital still cameras (DSCs), digital single-lens reflex cameras (DSLRs), and mirrorless cameras alone has grown from 100 million worldwide in 2004 to 1.5 billion in 2016, according to the Camera and Imaging Products Association (CIPA).
The universe of devices using image sensors has ballooned, now comprising applications from casual to biometrics to connected cars to factory automation and more.
That proliferation is because image sensors are increasingly used for more than just taking photos and videos. For example, in IoT devices, image sensors sometimes are used for biometric authentication such as facial and iris recognition. And in connected cars, image sensors enable semi-autonomous driving and safety features such as collision-avoidance systems.
Those are just a few examples of how image sensors are now key for enabling market-differentiating and revenue-generating features in a wide variety of devices. As a result, device manufacturers and their suppliers need a way to quickly and cost-effectively integrate image sensors into their products.
That’s why in November 2017, MIPI Alliance released Camera Command Set v1.0 (MIPI CCS v1.0), which significantly streamlines the process of integrating image sensors. The specification is designed for use by any company—including those that aren’t MIPI Alliance members—in any device for any imaging application, from photography to video to machine vision. Here’s an overview of the specification’s key features and benefits.
Flexibility and Repeatability Help Reduce Development Costs and Lead Time
MIPI CCS is a complete command set that covers a variety of basic and advanced features, such as resolution, frame rate, phase detection autofocus (PDAF), single-frame HDR and fast bracketing. Developers can use the specification to create a common software driver that configures the basic functionalities of any off-the-shelf image sensor that’s compliant with both MIPI CCS and MIPI Camera Serial Interface 2 v2.0 (MIPI CSI-2 v2.0).
MIPI CSI-2 is the world’s most widely used hardware interface for deploying camera and imaging components in mobile devices. MIPI CCS builds on that popularity and further increases interoperability, which helps to streamline integration and reduce costs for complex imaging and vision systems. All of that helps device manufacturers and their suppliers improve their products’ competitiveness, profitability, and time to market. MIPI CCS provides:
- Mandatory controls for all relevant basic functions
- Capability information to detect supported features and limits for system parameterization
- Embedded data for sync of sensor and host, such as 3A functionality
- Parameter retiming rules for robust operation so the host can understand better how the sensor behaves in certain conditions
- A standard register map
MIPI CCS specifies image sensor functionality at the register level. It’s independent from the device’s operating system and host system features, uses software drivers and only minimally specifies link-related items, thus enabling modular design principles. The net result is an architecture that gives developers the flexibility to use their MIPI CCS-enabled designs across multiple products. That’s also an example of how MIPI CCS enables repeatable designs, which can reduce development costs and lead time for multiple products in a company’s portfolio.
Figure 1 illustrates how MIPI CCS fits into a typical camera system. The system on chip (SoC) communicates with the image sensor over two paths. On the left is a bidirectional control link that uses MIPI CSI-2’s Camera Control Interface (CCI), with the payload defined in MIPI CCS. The right link goes from the sensor’s transmitter to the SoC’s receiver and uses MIPI CSI-2 over MIPI D-PHY/C-PHY. This link carries data in formats defined in CSI-2, as well as metadata defined in CCS.
Power Up, Identification and Data Formats
Using MIPI CCS, the software follows a sequence to power up and identify the camera module. This architecture gives developers flexibility when the image sensor shares a power supply with other modules.
Developers and manufacturers, including those that aren’t MIPI Alliance members, can request an ID from http://mid.mipi.org. The identification process includes version control information for software parameterization, as well as binary capability information (e.g., whether a particular mode or feature is supported) and the limit values for software parameterization.
MIPI CCS complements other MIPI specifications. For example, it supports all MIPI RAW and DPCM data formats defined in MIPI CSI-2 (v.2.0 and older) over D-PHY and C-PHY, such as RAW10 and DPCM10-8. Rather than defining the formats themselves, MIPI CCS defines the controls for selecting them. The CCI control interface, which can be based on I2C or MIPI I3C, provides access to standardized CCS registers and any additional manufacturer-specific registers.
Resolution and Exposure Parameters, Embedded Data
Developers have several options for changing the resolution with MIPI CCS, which covers the region of interest (analog, digital, and output crop) and readout mode (full, binning, and subsampling). To produce different resolutions, the pixel array can go through binning, subsampling, and digital cropping to produce the MIPI CSI-2 data frame, which consists of the visible pixels framed by a packet header and footer as shown in Figure 2.
MIPI CCS includes a variety of exposure parameters, starting with a mandatory set of basics that include the exposure time control and analog gain control. There’s also an optional basic global digital gain control, while advanced options include single-frame HDR with timing and synthesis modes and fast bracketing.
The specification gives developers options for customization and market-differentiating innovation by supporting manufacturer-specific registers (MSRs). One example is using MSRs for global shutter, while CCS exposure controls handle rolling shutter.
Embedded data can be used to synchronize the host and sensor. As Figure 3 shows, MIPI CSI-2 defines the embedded data at the top or bottom. Both are optional in MIPI CSI-2, but top is mandatory in MIPI CCS.
As Figure 4 shows, MIPI CCS defines the layer format for the embedded data lines, as well as the register information that must be transferred in the top line using a specific format. The specification also allows the use of the specific format for any MSRs and other camera command set registers in top-embedded data. Finally, MIPI CCS supports the use of other formats in additional embedded data lines.
Phase Detection Auto Focus, Test Modes and More
PDAF is an increasingly popular capability. MIPI CCS supports it by accommodating a variety of sensors:
- Those with only PDAF pixels
- Those able to separate PDAF pixels into different MIPI CSI-2 logical channel (virtual channel or DataType interleaving) from visible pixels
- Those with or without PDAF data processing
Finally, MIPI CCS specifies mandatory and optional test modes. These include programmable data, basic color bar, advanced color bar, and PN9.
MIPI CCS includes a variety of additional features, such as frame timings and clocking, image correction controls, and an interface for sensor non-volatile memory. For detailed information about these features and more, download the specification.
More details may also be found in a recent webinar on MIPI CCS. Image sensor designers, software developers, camera engineers, and anyone who works with image sensors will benefit from the rapid integration of basic camera functionalities in a plug-and-play fashion without requiring any device-specific drivers.
Mikko Muukki is Expert, Imaging and Video Technology at Huawei. He has 14 years’ experience in cameras and imaging, as well as additional experience in other technology fields. At MIPI Alliance, he is leading the MIPI CCS development.