CN-118714436-B - Visual imaging and identifying device

CN118714436BCN 118714436 BCN118714436 BCN 118714436BCN-118714436-B

Abstract

The disclosure provides a visual imaging and recognition device, which can be applied to the technical field of image processing. The device comprises an imaging sensor for acquiring two-dimensional image information and three-dimensional depth information of a target object, a signal shaping and encoding module for performing signal shaping and encoding on the two-dimensional image information and the three-dimensional depth information to acquire target two-dimensional image information and target three-dimensional depth information, a processing module for processing and identifying the target two-dimensional image information and the target three-dimensional depth information to generate an identification result, an output module for performing signal synchronous caching and output on the identification result, and a programmable microcontroller for providing synchronous control signals for the imaging sensor, the signal shaping and encoding module, the processing module and the output module so as to control the synchronous and coordinated operation of the device. The device can naturally simulate the working mechanism of a human brain vision system, and has the advantages of high performance and low power consumption.

Inventors

LIU LIYUAN
YU SHUANGMING
YANG XU
WANG ZHE
WU NANJIAN

Assignees

中国科学院半导体研究所

Dates

Publication Date: 20260508
Application Date: 20240709

Claims (8)

1. A visual imaging and recognition device, comprising: the imaging sensor is used for acquiring two-dimensional image information and three-dimensional depth information of the target object; The signal shaping and encoding module is used for carrying out signal shaping and encoding on the two-dimensional image information and the three-dimensional depth information so as to obtain target two-dimensional image information and target three-dimensional depth information; The processing module is used for processing and identifying the target two-dimensional image information and the target three-dimensional depth information so as to generate an identification result; an output module for synchronously buffering and outputting the identification result and The programmable microcontroller is used for providing synchronous control signals for the imaging sensor, the signal shaping and encoding module, the processing module and the output module so as to control the device to synchronously and coordinately operate; wherein the processing module comprises: The nonvolatile memory and calculation integrated sub-module is used for performing pulse convolution neural network deep learning and processing on the target two-dimensional image information so as to realize intelligent processing on the target two-dimensional image information and obtain a two-dimensional image recognition processing result of an analog domain, and A volatile memory integrated sub-module for carrying out programmable digital signal processing on the target three-dimensional depth information to realize reconstruction and depth discrimination of the target three-dimensional depth information and obtain three-dimensional image reconstruction and depth discrimination results of a digital domain, The nonvolatile memory integrated sub-module and the volatile memory integrated sub-module are in heterogeneous parallel relation, and are programmed by the programmable microcontroller and used for carrying out mixed processing on two-dimensional image information and three-dimensional depth information; the recognition result comprises a two-dimensional image recognition processing result of the analog domain and a three-dimensional image reconstruction and depth discrimination result of the digital domain; the output module is used for converting the two-dimensional image recognition processing result of the analog domain into the two-dimensional image recognition processing result of the digital domain, and The output module is used for integrating the two-dimensional image recognition processing result of the digital domain with the three-dimensional image reconstruction and depth discrimination result of the digital domain and then synchronously outputting the two-dimensional image recognition processing result and the three-dimensional image reconstruction and depth discrimination result.
2. The apparatus of claim 1, wherein the imaging sensor comprises a single photon array imaging sensor.
3. The apparatus of claim 1 or 2, wherein the two-dimensional image information comprises pulsed two-dimensional image information, wherein a pulse frequency of the pulsed two-dimensional image information is positively correlated with an intensity of a light incident signal of the target object, and The method for measuring the three-dimensional depth information comprises a time-of-flight method.
4. The apparatus of claim 1, wherein the processing module further comprises: The memory system is configured to store a plurality of data, Wherein the nonvolatile memory integrated sub-module and the volatile memory integrated sub-module share the storage system.
5. The apparatus of claim 1, wherein the device comprises a plurality of sensors, The nonvolatile memory integrated sub-module comprises a resistance change type memory cell array, wherein the resistance change type memory cell has nonvolatile characteristic and in-memory computing characteristic, and/or The resistive memory cell array includes a transistor and a resistive resistor, and/or The input signal of the resistance change type memory unit comprises a digital signal, and the output signal comprises an analog signal of a current mode.
6. The apparatus of claim 1, wherein the device comprises a plurality of sensors, The volatile memory integrated sub-module comprises an array of SRAM cells, wherein the SRAM cells are capable of performing logic operations within the SRAM cells, and/or The SRAM cell includes a transistor, and/or The input signal of the sram cell comprises a digital signal and the output signal comprises a digital signal.
7. The apparatus of claim 1, wherein the programmable microcontroller receives an input synchronous clock signal, provides a synchronous control signal to the imaging sensor, the signal shaping and encoding module, the processing module, and the output module via the synchronous clock signal, and/or The programmable microcontroller supports a RISC-V architecture.
8. The apparatus of claim 1, wherein the programmable microcontroller is bi-directionally coupled to at least one of the imaging sensor, the signal shaping and encoding module, the processing module, and the output module for transmitting control signals to the connection module and receiving feedback signals from the connection module.

Description

Visual imaging and identifying device Technical Field The disclosure relates to the technical field of image processing, and in particular relates to a visual imaging and recognition device. Background Along with the continuous development of information technology integration and application, the intelligent equipment at the edge side has been widely applied in the fields of intelligent security, robot vision, intelligent auxiliary driving, consumer electronics and the like, and becomes one of the important directions of future electronic information technology development. The edge side intelligent device is a device located at the edge of the data network, and is different from a data network center type device, and has high requirements on power consumption, timeliness, safety and integration. For example, the power consumption performance of a device directly determines the duration of use of the device, and the data transmission and computation delay performance directly determines the processing timeliness of the device, so low power consumption, low delay design has been an important design goal for edge devices. The visual information acquisition and processing system is a typical edge-side intelligent device and has the function of independently completing image information acquisition and intelligent processing. The traditional visual information acquisition and processing system adopts a structure with separate sensing, storing and processing, so that the problem of limited transmission bandwidth of a large amount of sensing data between the sensor and the processor is caused, the system performance is also limited by the bottleneck of data transmission speed between the memory and the processor, and the problems of system delay and energy consumption caused by carrying a large amount of sensing data are fundamentally difficult to solve by the separate structure. The visual information acquisition and processing system integrating sensing and memory integrates the image sensor, the intelligent processor and the memory, can realize the functions of acquiring, storing and intelligent processing of image signals with high energy efficiency, and becomes one of important research subjects in the field of semiconductor information at present. Disclosure of Invention First, the technical problem to be solved In order to solve at least one of the above technical problems of the visual imaging and recognition device in the prior art, an embodiment of the present disclosure provides a visual imaging and recognition device, which can be used for imaging in a weak light environment, has a function of fusion processing of two-dimensional image information and three-dimensional depth information, can quickly recognize a target object and reconstruct depth information under extremely low power consumption, is used for judging whether the object to be detected is a three-dimensional object or a planar photo, and can be applied to application scenes such as intelligent security, face recognition, robot vision, intelligent auxiliary driving, and the like. (II) technical scheme In view of the above technical problems, embodiments of the present disclosure provide a visual imaging and recognition device. According to a first aspect of the present disclosure, a visual imaging and recognition device is provided, including an imaging sensor for acquiring two-dimensional image information and three-dimensional depth information of a target object, a signal shaping and encoding module for performing signal shaping and encoding on the two-dimensional image information and the three-dimensional depth information to obtain target two-dimensional image information and target three-dimensional depth information, a processing module for processing and recognizing the target two-dimensional image information and the target three-dimensional depth information to generate a recognition result, an output module for performing signal synchronous buffering and outputting on the recognition result, and a programmable microcontroller for providing synchronous control signals for the imaging sensor, the signal shaping and encoding module, the processing module and the output module to control the device to operate synchronously and coordinately. In some exemplary embodiments, the imaging sensor comprises a single photon array imaging sensor. In some exemplary embodiments, the two-dimensional image information includes pulsed two-dimensional image information, wherein a pulse frequency of the pulsed two-dimensional image information is positively correlated with an intensity of a light incident signal of the target object, and the method of measuring the three-dimensional depth information includes a time-of-flight method. In some exemplary embodiments, the processing module comprises a nonvolatile memory integrated sub-module for performing pulse convolution neural network depth learning and processing on target two-dimensional image