Search

EP-4738874-A1 - RECONFIGURABLE HYBRID EVENT-BASED VISION SENSOR AND CMOS IMAGE SENSOR

EP4738874A1EP 4738874 A1EP4738874 A1EP 4738874A1EP-4738874-A1

Abstract

Reconfigurable hybrid event-based vision sensor and CMOS image sensor systems (an associated systems, devices, and methods) are disclosed. In one embodiment, a hybrid image sensor includes a pixel circuit array with pixel circuits reconfigurable between (i) a CMOS image sensor (CIS) configuration for obtaining intensity information and (ii) an event vision sensor (EVS) configuration for obtaining contrast information. Control circuitry can reconfigure a first subset of pixel circuits into the EVS configuration based on detected motion in an external scene, switching the hybrid image sensor between (a) a CIS-only mode in which all pixel circuits operate in the CIS configuration and (b) a hybrid mode in which the first subset operates in the EVS configuration and a second subset operates in the CIS configuration. The control circuitry can control the number of pixel circuits included in the first subset based on luminance conditions in the external scene.

Inventors

  • SUESS, ANDREAS
  • TSAU, GUANNHO

Assignees

  • OmniVision Technologies, Inc.

Dates

Publication Date
20260506
Application Date
20251016

Claims (17)

  1. A hybrid image sensor, comprising: a pixel circuit array including a plurality of pixel circuits, wherein each pixel circuit of the plurality is reconfigurable between (i) a CMOS image sensor (CIS) configuration in which the pixel circuit is usable to obtain intensity information of light from an external scene and (ii) an event vision sensor (EVS) configuration in which the pixel circuit is usable to obtain contrast information of the light; and control circuitry configured to, based at least in part on motion detected in an the external scene, reconfigure a first subset of pixel circuits of the plurality of pixel circuits into the EVS configuration and thereby switch an operating mode of the hybrid image sensor between (a) a CIS-only mode in which all of the plurality of pixel circuits are configured in the CIS configuration and (b) a hybrid mode in which the first subset of pixel circuits of the plurality of pixel circuits are configured in the EVS configuration and a second subset of pixel circuits of the plurality of pixel circuits are configured in the CIS configuration.
  2. The hybrid image sensor of claim 1, wherein the control circuitry is configured to, based at least in part on luminance conditions within the external scene, control a number of pixel circuits of the plurality of pixel circuits included within the first subset.
  3. The hybrid image sensor of claim 2, wherein the control circuitry is configured to, based at least in part on the luminance conditions being below a threshold corresponding to low luminance conditions, reconfigure 50% of the pixel circuits of the plurality of pixel circuits into the EVS configuration based at least in part on the motion detected in the external scene.
  4. The hybrid image sensor of claim 3, wherein: the plurality of pixel circuits includes a pixel arrangement having four subunits arranged in a 2x2 grid of subunits; each of the four subunits includes four pixel circuits of the plurality of pixel circuits; the four pixel circuits of each of the four subunits are arranged in a 2x2 grid of pixel circuits; two pixel circuits of each of the four subunits are included in the first subset such that only 50% of pixel circuits included in the pixel arrangement are included in the first subset; and the two pixel circuits of each of the four subunits are arranged diagonally from one another within the subunit.
  5. The hybrid image sensor of claim 4, wherein: the hybrid image sensor further includes a plurality of event driven circuits coupled to the plurality of pixel circuits; the two pixel circuits of a first subunit of the four subunits are arranged across a first diagonal of the first subunit; the two pixel circuits of a second subunit of the four subunits that is laterally or vertically aligned with the first subunit are arranged across a second diagonal of the second subunit such that at least one of the two pixel circuits of the second subunit is configured to direct image charge to a same event driven circuit of the plurality of event driven circuits as at least one of the two pixel circuits of the first subunit.
  6. The hybrid image sensor of any of claims 2-5, wherein the control circuitry is configured to, based at least in part on the luminance conditions being above a threshold corresponding to high luminance conditions, reconfigure 12.5% of the pixel circuits of the plurality of pixel circuits into the EVS configuration based at least in part on the motion detected in the external scene.
  7. The hybrid image sensor of claim 6, wherein: the plurality of pixel circuits includes a pixel arrangement having four subunits arranged in a 2x2 grid of subunits; each of the four subunits includes four pixel circuits of the plurality of pixel circuits; the four pixel circuits of each of the four subunits are arranged in a 2x2 grid of pixel circuits; one pixel circuit in each of a first subunit and a second subunit of the four subunits is included in the first subset such that only 12.5% of pixel circuits included in the pixel arrangement are included in the first subset; and the first subunit and the second subunits are arranged diagonally from one another in the 2x2 grid of subunits.
  8. The hybrid image sensor of claim 7, wherein: the hybrid image sensor further includes a plurality of event driven circuits coupled to the plurality of pixel circuits; and the one pixel circuit in the first subunit and the one pixel circuit in the second subunit are configured to direct image charge to a same event driven circuit of the plurality of event driven circuits as one another.
  9. The hybrid image sensor of any of claims 6-8, wherein: the threshold is a first threshold; and the control circuitry is configured to, based at least in part on the luminance conditions being below the first threshold and above a second threshold corresponding to low luminance conditions, reconfigure 25% of the pixel circuits of the plurality of pixel circuits into the EVS configuration based at least in part on the motion detected in the external scene.
  10. The hybrid image sensor of claim 9, wherein: the plurality of pixel circuits includes a pixel arrangement having four subunits arranged in a 2x2 grid of subunits; each of the four subunits includes four pixel circuits of the plurality of pixel circuits; the four pixel circuits of each of the four subunits are arranged in a 2x2 grid of pixel circuits; one pixel circuit in each of the four subunits are included in the first subset such that only 12.5% of pixel circuits included in the pixel arrangement are included in the first subset.
  11. The hybrid image sensor of claim 10, wherein: the hybrid image sensor further includes a plurality of event driven circuits coupled to the plurality of pixel circuits; and each pixel circuit of the pixel arrangement that is included in the first subset is configured to direct image charge to a different event driven circuit of the plurality of event driven circuits from all other pixel circuits of the pixel arrangement that are included in the first subset.
  12. The hybrid image sensor of claim 10, wherein: the hybrid image sensor further includes a plurality of event driven circuits coupled to the plurality of pixel circuits; and each pixel circuit of the pixel arrangement that is included in the first subset is configured to direct image charge to a same event driven circuit of the plurality of event driven circuits as all other pixel circuits of the pixel arrangement that are included in the first subset.
  13. The hybrid image sensor of claim 10, wherein: the hybrid image sensor further includes a plurality of event driven circuits coupled to the plurality of pixel circuits; two pixel circuits of the pixel arrangement that are included in the first subset are each configured to direct image charge to different event driven circuits of the plurality of event driven circuits from all other pixel circuits of the pixel arrangement that are included in the first subset; and two other pixel circuits of the pixel arrangement that are included in the first subset are each configured to direct image charge to a same event driven circuit of the plurality of event driven circuits as one another.
  14. The hybrid image sensor of any of claims 2-13, wherein the control circuitry is configured to, based at least in part on the luminance conditions being above a threshold corresponding to extra high luminance conditions, reconfigure 6.25% of the pixel circuits of the plurality of pixel circuits into the EVS configuration based at least in part on the motion detected in the external scene.
  15. The hybrid image sensor of claim 14, wherein: the plurality of pixel circuits includes a pixel arrangement having four subunits arranged in a 2x2 grid of subunits; each of the four subunits includes four pixel circuits of the plurality of pixel circuits; the four pixel circuits of each of the four subunits are arranged in a 2x2 grid of pixel circuits; and one pixel circuit in one of the four subunits is included in the first subset such that only 6.25% of pixel circuits included in the pixel arrangement are included in the first subset.
  16. The hybrid image sensor of any of claims 2-16, wherein the control circuitry is configured to, based at least in part on the luminance conditions being above a threshold corresponding to extremely high luminance conditions, reconfigure 3.125% of the pixel circuits of the plurality of pixel circuits in the EVS configuration based at least in part on the motion detected in the external scene.
  17. The hybrid image sensor of claim 16, wherein: the plurality of pixel circuits includes a pixel arrangement having four subunits arranged in a 2x2 grid of subunits; each of the four subunits includes four pixel circuits of the plurality of pixel circuits; the four pixel circuits of each of the four subunits are arranged in a 2x2 grid of pixel circuits; half of one pixel circuit in one of the four subunits is included in the first subset such that only 3.125% of pixel circuits included in the pixel arrangement are included in the first subset.

Description

CROSS-REFERENCE TO RELATED APPLICATION(S) This application claims priority to U.S. Provisional Application No. 63/714,458, titled RECONFIGURABLE HYBRID EVENT-BASED VISION SENSOR AND CMOS IMAGE SENSOR, filed October 31, 2024, which is hereby incorporated by reference in its entirety. TECHNICAL FIELD The present disclosure relates to image sensors, and more particularly to a reconfigurable hybrid event-based vision sensor (EVS) and CMOS image sensor (CIS) that can dynamically switch between operating modes based at least in part on motion detection and luminance conditions. For example, several embodiments of the present technology relate to hybrid image sensor systems that can reconfigure pixel functionality between CMOS image sensor operation and event-based vision sensor operation based at least in part on detected motion and ambient lighting conditions. BACKGROUND CMOS image sensors (CIS) operate in response to image light from an external scene being incident upon the image sensor. The image sensor includes an array of pixel circuits having photosensitive elements (e.g., photodiodes) that absorb a portion of the incident image light and generate image charge upon absorption of the image light. The image charge photogenerated by the pixel circuits may be measured as analog output image signals on column bitlines that vary as a function of the incident image light. In other words, the amount of image charge generated is proportional to the intensity of the image light, which is read out as analog image signals from the column bitlines and converted to digital values to provide information that is representative of the external scene. CMOS image sensors (CIS) typically capture complete image frames through synchronized pixel circuit readout operations. By contrast, event-based vision sensors (EVS) represent a specialized class of image sensors that detect and output luminance changes from individual pixel circuits, combined with coordinate and temporal information. These sensors operate asynchronously, with each pixel circuit independently monitoring for luminance changes that exceed preset threshold values. When such changes are detected, the sensor generates event data that can include pixel circuit coordinates, timing information, and/or polarity data. This approach enables high-speed data output with low latency while maintaining reduced power consumption. The asynchronous operation allows multiple pixel circuits to generate events simultaneously, with arbitration circuits managing output order based on earliest-received events. Hybrid sensor architectures combine EVS and CIS capabilities within a single device. Such hybrid image sensors offer the potential to leverage the advantages of both EVS and CIS sensing modalities. For example, hybrid image systems can utilize a first subset of pixel circuits to provide conventional image data and a second subset of pixel circuits to provide event-based information. BRIEF DESCRIPTION OF THE DRAWINGS Non-limiting and non-exhaustive embodiments of the present technology are described below with reference to the following figures, in which like or similar reference characters are used to refer to like or similar components throughout unless otherwise specified. FIG. 1A illustrates a block diagram of a stacked hybrid sensor system configured in accordance with various embodiments of the present technology.FIG. 1B illustrates a block diagram of the stacked hybrid sensor system of FIG. 1A configured in accordance with various embodiments of the present technology.FIG. 2 illustrates a circuit diagram of a pixel circuit with mode switching capability configured in accordance with various embodiments of the present technology.FIGS. 3A-3E illustrate pixel arrangements showing different ratios of CIS and EVS pixels configured in accordance with various embodiments of the present technology.FIG. 4 illustrates a circuit diagram of a multi-subpixel pixel circuit configured in accordance with various embodiments of the present technology.FIG. 5 is a flow diagram illustrating a method of operating an image sensor in accordance with various embodiments of the present technology.FIGS. 6 and 7 illustrate example methods of operating pixel arrangements for a 50% EVS/50% CIS pixel allocation in accordance with various embodiments of the present technology.FIGS. 8-10 illustrate example methods of operating pixel arrangements for a 25% EVS/75% CIS pixel allocation in accordance with various embodiments of the present technology.FIGS. 11 and 12 illustrate example methods of operating pixel arrangements for a 12.5% EVS/87.5% CIS pixel allocation in accordance with various embodiments of the present technology.FIG. 13 illustrates an example method of operating a pixel arrangement for a 6.25% EVS/93.75% CIS pixel allocation in accordance with various embodiments of the present technology.FIGS. 14A-14B illustrate an example method of operating a pixel arrangement for a 3.125% EVS/96.875% C