Search

JP-2026514353-A - Event-based image processing

JP2026514353AJP 2026514353 AJP2026514353 AJP 2026514353AJP-2026514353-A

Abstract

An event-based image processing method is disclosed. The method includes: acquiring time-series input signal data characterizing information from an input source; applying a first processing stage, which comprises applying a compression nonlinearity to the input signal data; and applying a second processing stage to the output from the first processing stage. The second processing stage includes time and/or spatial processing within a feedback loop, applying a high-pass or band-pass time filter and/or a high-pass or band-pass spatial filter to the output from the first processing stage, thereby suppressing recurring changes and improving the ratio of events to recurring changes in the output from the first processing stage.

Inventors

  • タラ、ハミルトン
  • ベンジャミン、エッサム
  • アンドリュー、ニコルソン

Assignees

  • クーボス、プロプライエタリー、リミテッド

Dates

Publication Date
20260511
Application Date
20240319
Priority Date
20230320

Claims (20)

  1. This involves acquiring time-series input signal data that characterizes the information from the input source, and Applying a first processing step, wherein the first processing step comprises applying compression nonlinearity to the input signal data, Applying a second processing stage to the output from the first processing stage, wherein the second processing stage includes spatial and/or temporal processing within a feedback loop, and applies a high-pass or band-pass spatial and/or temporal filter to the output from the first processing stage, thereby suppressing recurring changes and improving the ratio of events to recurring changes in the output from the first processing stage. An image processing method comprising:
  2. The method according to claim 1, comprising detecting an event in the output of the second processing stage.
  3. The method according to claim 2, wherein detecting the event comprises applying at least one threshold value to the pixel value of the output in the second processing stage.
  4. The method according to claim 3, wherein the at least one threshold applied includes a plurality of thresholds, and each threshold is applied to each pixel or each set of pixels.
  5. The method according to any one of claims 2 to 4, comprising setting an event rate for detected events.
  6. The method according to claim 5, comprising setting two different event rates for each of two different detected events.
  7. The method according to claim 5 or 6, wherein the event rate is set by setting a counter that is the number of clock cycles during which the detected event is expected to persist.
  8. The method according to any one of claims 1 to 7, wherein the time high-pass or band-pass filter is an n-th order filter, where n corresponds to the number of time steps incorporated into the time processing.
  9. The method according to claim 8, wherein n is equal to 1.
  10. The method according to claim 8, wherein n is greater than 1.
  11. The method according to any one of claims 1 to 10, wherein the time-pass or band-pass filter is an infinite impulse response filter.
  12. The method according to any one of claims 1 to 11, wherein the frequency cutoff of the time-pass or band-pass filter is variable based on at least one or more of the characteristics of the input source and the characteristics of the environment in which the input source acquires the input data.
  13. The method according to any one of claims 1 to 12, wherein the second processing step further comprises spatial processing configured to detect edges.
  14. The method according to claim 13, wherein the spatial processing is configured to be performed before or after the temporal processing.
  15. The method according to claim 13 or 14, wherein the spatial processing applies a high-pass filter implemented using convolution.
  16. The method according to any one of claims 1 to 15, wherein the first processing step comprises a time feedback loop, in which current and previous samples from the input signal data and at least one previous sample of the output from the first processing step are used to obtain the current sample of the output from the first processing step.
  17. The method according to any one of claims 1 to 16, wherein applying the compression nonlinearity to the input data comprises applying a gain to the input signal data, and the gain is variable based on the magnitude of the input signal data.
  18. The method according to any one of claims 1 to 17, wherein the first processing step comprises a plurality of processing modules, each configured to favorably process input data with different characteristics.
  19. The method according to claim 18, wherein the processing path comprises a first processing module for favorably processing input data of lower size or lower contrast, and a second processing module for favorably processing input data of higher size or higher contrast.
  20. The method according to claim 19, wherein each processing module in the first processing step comprises a division-type low-pass filter.

Description

This disclosure relates to a method and apparatus for event-based image processing. Event-based image sensors (EBS), also known as dynamic vision sensors (DVS), have become widespread in recent years. Their operation is based on the biology of animal eyes, and their technology is developed following a neuromorphological approach. EBS are preferred in situations where lighting conditions are harsh, data throughput is critical, and power is constrained. The earliest event-based sensors (EBS) were described by T. Delbuck in the paper "A 128x128 120 dB 15 μs Latency Asynchronous Temporal Contrast Vision Sensor," IEEE Journal of Solid-State Circuits, vol. 43, no. 2, pp. 566–576, February 2008, doi: 10.1109/JSSC. 2007.914337. That research included a simplified circuit for the EBS, reproduced here in Figure 1, and its operating principle, reproduced here in Figure 2. The circuit shown in Figure 1 stores the voltage change at each time step, and the stored voltage change corresponds to the logarithm of the change in photodiode current across capacitor C1. This value of photodiode current change is amplified by C1/C2 to generate a voltage change value Vdiff . Vdiff is used to determine whether the logarithmic change in photodiode current is large enough to generate an "event". An "ON" event is generated when the value of Vdiff increases positively by a minimum amount, and an "OFF" event is generated when there is a sufficiently large negative change. The "threshold," which is the required magnitude of change, is set globally for both positive and negative changes. When the logarithm of the photodiode current is taken, small changes in current are amplified more than large changes in current. This compressive nonlinearity is similar to that observed in biological sensory organs. However, the difference from EBS is that 1) EBS only reports whether the change is positive, negative, or no change occurred, rather than the magnitude of the change, and 2) logarithmic compressive nonlinearity is established. Figure 3 illustrates this nonlinearity. Therefore, there is no way to explore different types of nonlinearity. Furthermore, the parameters of the logarithmic response are fixed. In reality, the widespread adoption of EBS technology has been slow due to several drawbacks in its real-world performance and operational limitations. Currently, custom EBS solutions are available on the market, such as those manufactured by Prophesee (https://www.prophesee.ai/) and IniVation (https://iniVation.com/). However, currently available EBS solutions still require optimal illumination conditions to maintain the stated performance specifications. Performance reliability is compromised in low-light conditions with few photons, or in complex illumination conditions where the amount of photons may not be stable. Furthermore, processing is required to interpret the event data provided by EBS. The event data is asynchronous, meaning that when pixels generate an event, they are timestamped and sent to the event bus. In low-light or suboptimal illumination conditions where noise can easily overwhelm the "event" data, processing must be performed by very fast hardware, or the event detection threshold must be set very high, in order to enable event generation. Setting the threshold high reduces the high dynamic range benefit of EBS. On the other hand, using high-speed post-processing hardware requires high-performance computing solutions such as GPUs or FPGAs. There are several research papers on EBS embodiments using software or digital hardware such as FPGAs. Most of these papers focus on emulating the analog circuit performance of EBS, rather than the biological function of M-type retinal ganglion cells. Essentially, these models attempt to emulate something that is already an abstraction of biological function. Software simulations or digital emulators are used as substitutes for actual EBS solutions, not as improvements to existing EBS systems. Where prior art is referenced in this specification, it should be understood that such references do not constitute an acknowledgment that such prior art forms part of the common general knowledge in the art in Australia or any other country. This is a schematic diagram of a prior art circuit for implementing an event-based sensor.Figure 1 is a schematic diagram illustrating the operating principle of the event-based sensor circuit shown.This is a schematic diagram illustrating the nonlinearity of logarithmic compression.This is a schematic diagram of an event-based detection system according to one embodiment of the present invention.This is a schematic diagram illustrating the operation of the sliding kernel.This figure shows a set of sliding kernels configured to detect vertical and horizontal edges.This figure shows another set of sliding kernels configured to detect vertical and horizontal edges within a set of pixels.This diagram schematically shows the processing in the first processing sta