CN-114897665-B - Configurable real-time parallax point cloud computing device and method

CN114897665BCN 114897665 BCN114897665 BCN 114897665BCN-114897665-B

Abstract

The invention provides a configurable real-time parallax point cloud computing device and method, comprising an image caching unit, a caching controller, a PE array, a result shaping module, a minimum value searching module and a configuration analyzing module, wherein the image caching unit is used for outputting image window data of a designated window size and a sliding window sequence, the caching controller is used for controlling the image caching unit to output the image window data and distributing the image window data to PEs in the PE array, the PE array is used for generating a plurality of PUs of a designated structure and obtaining SAD matching cost computing results, the result shaping module is used for adding data fields to the matching cost, the minimum value searching module is used for searching the matching cost step by step to obtain a parallax value, the configuration analyzing module is used for analyzing received configuration information to generate corresponding control signals and respectively input the corresponding control signals into other modules, so that the parallax point cloud computing can be performed in real time, and the matching parameters can be configured and do not need to be reconstructed.

Inventors

Meng Zhaoteng
GAO JIE
HU WENQING
SUN YAQIANG
SHU LIN
LI NING
FAN QIUXIANG

Assignees

中国科学院自动化研究所
广东人工智能与先进计算研究院

Dates

Publication Date: 20260505
Application Date: 20220401

Claims (10)

1. A configurable real-time parallax point cloud computing device, comprising: The system comprises an image caching unit, a caching controller, a processing unit PE array, a result shaping module, a minimum value searching module and a configuration analyzing module; the image buffer unit is connected with the buffer controller and is used for outputting image window data to the buffer controller after shaping the buffered binocular image data according to the size of a designated window and the sequence of sliding windows under the control of the buffer controller; The buffer controller is respectively connected with the configuration analysis module and the PE array and is used for controlling the image buffer unit to output image window data according to the control signal transmitted by the configuration analysis module, and the image window data is distributed to PE in the PE array through the buffer controller; the PE array is respectively connected with the configuration analysis module and the result shaping module, and is used for generating a plurality of algorithm Processing Units (PU) with a designated structure according to the control signals transmitted by the configuration analysis module, processing the input image window data based on the plurality of PUs with the designated structure, obtaining SAD matching cost calculation results and outputting the SAD matching cost calculation results to the result shaping module; the result shaping module is respectively connected with the configuration analysis module and the minimum value searching module and is used for adding fields to the input SAD matching cost calculation result according to the control signal transmitted by the configuration analysis module and outputting the result to the minimum value searching module; The minimum value searching module is connected with the configuration analyzing module and is used for searching the minimum value step by step for the input SAD matching cost calculation result according to the control signal and the minimum value searching algorithm transmitted by the configuration analyzing module and outputting a parallax value corresponding to the minimum matching cost; The configuration analysis module is used for analyzing the received configuration information, generating corresponding control signals and respectively inputting the corresponding control signals to the cache controller, the PE array, the result shaping module and the minimum value searching module.
2. The configurable real-time parallax point cloud computing device of claim 1, wherein PEs in the PE array are interconnected vertically to transfer intermediate results and horizontally to transfer operands and final matching costs.
3. The configurable real-time parallax point cloud computing device of claim 2, wherein the PE array includes one or more of the following types of PEs therein: the Ultra PE is used for carrying out the difference absolute value of two operands and the accumulation operation of partial sum in the SAD matching cost calculation process; STANDARD PE, in the process of performing SAD matching cost calculation, performing difference absolute value calculation on two operands, and performing partial sum accumulation operation; Lite PE, used for executing SAD matching cost calculation process, to two operands difference absolute value operation; wherein, the computing resource corresponding to the Ultra PE is larger than STANDARD PE.
4. A configurable real-time parallax point cloud computing device according to claim 3, wherein each column in the PE array is configurable to include one or more PUs for performing SAD matching cost computation operations of a specified window size.
5. The configurable real-time parallax point cloud computing device of claim 4, wherein the first row of PEs in the PU is the Ultra PEs or the STANDARD PE.
6. A configurable real-time parallax point cloud computing method performed based on the configurable real-time parallax point cloud computing device according to any one of claims 1 to 5, the method comprising: the configuration analysis module analyzes the received configuration information to generate corresponding control signals, and the corresponding control signals are respectively input to the cache controller, the PE array, the result shaping module and the minimum value searching module; The buffer controller controls the image buffer unit to output image window data corresponding to one or more paths of binocular image data according to the control signals transmitted by the configuration analysis module, the size of a designated window and the sliding window sequence; the PE array generates a plurality of PUs of a designated structure according to the control signals transmitted by the configuration analysis module, and processes the input image window data based on the PUs of the designated structure to obtain SAD matching cost calculation results corresponding to the image window data; the result shaping module adds a field to the SAD matching cost calculation result output by the PE array according to the control signal transmitted by the configuration analysis module; and the minimum value searching module searches the minimum value step by step for the SAD matching cost calculation result after adding the field according to the control signal and the minimum value searching algorithm transmitted by the configuration analyzing module, and outputs a parallax value corresponding to the minimum matching cost.
7. The configurable real-time parallax point cloud computing method according to claim 6, wherein the configuration information includes: Image resolution, matching window size, parallax search depth, number of ways of binocular image data, and PE mode of operation.
8. The method for computing the configurable real-time parallax point cloud according to claim 7, wherein the determining manner of the configuration information includes: determining that single data stream or multiple data stream performance indexes are met; according to the number of the unit computing resources which can be distributed in the PE array, the parallax search depth corresponding to each data stream and the video frame rate corresponding to each data stream, distributing computing resources; and generating configuration information according to the allocated computing resources.
9. The method of claim 8, wherein, when the number of data streams is two, the allocating the computing resources according to the number of unit computing resources that can be allocated in the PE array, the parallax search depth corresponding to each data stream, and the video frame rate corresponding to each data stream includes: determining the remaining allocatable computing resources in the PE array after allocating a unit of computing resources for each data stream; If the remaining allocatable computing resources can provide at least one unit of computing resources for each data stream and the first condition is met, continuing to allocate one unit of computing resources for each data stream; If the remaining allocatable computing resources can provide at least one unit of computing resources for each data stream, but do not meet the first condition, allocating the unit of computing resources for each data stream individually according to the magnitude relation between the first value and the second value; The first value is determined according to the parallax search depth and the video frame rate corresponding to each data stream, the second value is determined according to the number of unit computing resources currently allocated to each data stream, and the first condition is determined according to the first value, the second value and a preset threshold.
10. The configurable real-time parallax point cloud computing method according to claim 9, further comprising: And if the residual allocatable computing resources can only provide unit computing resources for the target data stream, the residual allocatable computing resources are fully allocated to the target data stream.

Description

Configurable real-time parallax point cloud computing device and method Technical Field The invention relates to the technical field of microelectronics, in particular to a configurable real-time parallax point cloud computing device and method. Background Stereo matching is a key link in binocular stereo vision, and a stereo matching algorithm searches corresponding points of left and right images according to pixel information similarity, so that parallax is determined. By searching the corresponding points of the pixel points of the full graph, parallax point clouds of the whole graph can be generated, and further the parallax point clouds are used for tasks such as ranging or three-dimensional reconstruction. The stereo matching algorithm may be deployed on different platforms such as a central processing unit (Central Processing Unit, CPU), a graphics processor (Graphic Processing Unit, GPU), a Field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA), an Application SPECIFIC INTEGRATED Circuit (ASIC), etc. The CPU and the GPU have good programmability, can be matched with different matching parameters to the greatest extent, can meet the three-dimensional matching tasks of different scenes, but have poor instantaneity, cannot meet the application requirements of high instantaneity, and the ASIC has high energy efficiency and instantaneity, but has poor flexibility, and cannot be matched with different matching parameters. FPGAs can effectively accelerate computationally intensive tasks, but the prior art can only adapt to different matching parameters by reconstruction, which is costly in terms of time and cost of the reconstruction design. Disclosure of Invention Aiming at the problems existing in the prior art, the invention provides a configurable real-time parallax point cloud computing device and method. In a first aspect, the present invention provides a configurable real-time parallax point cloud computing device, comprising: The system comprises an image caching unit, a caching controller, a processing unit PE array, a result shaping module, a minimum value searching module and a configuration analyzing module; the image buffer unit is connected with the buffer controller and is used for outputting image window data to the buffer controller after shaping the buffered binocular image data according to the size of a designated window and the sequence of sliding windows under the control of the buffer controller; The buffer controller is respectively connected with the configuration analysis module and the PE array and is used for controlling the image buffer unit to output image window data according to the control signal transmitted by the configuration analysis module, and the image window data is distributed to PE in the PE array through the buffer controller; The PE array is respectively connected with the configuration analysis module and the result shaping module and is used for generating a plurality of PUs of a designated structure according to the control signals transmitted by the configuration analysis module, processing the input image window data based on the PUs of the designated structure to obtain SAD matching cost calculation results and outputting the SAD matching cost calculation results to the result shaping module; the result shaping module is respectively connected with the configuration analysis module and the minimum value searching module and is used for adding fields to the input SAD matching cost calculation result according to the control signal transmitted by the configuration analysis module and outputting the result to the minimum value searching module; The minimum value searching module is connected with the configuration analyzing module and is used for searching the minimum value step by step for the input SAD matching cost calculation result according to the control signal and the minimum value searching algorithm transmitted by the configuration analyzing module and outputting a parallax value corresponding to the minimum matching cost; The configuration analysis module is used for analyzing the received configuration information, generating corresponding control signals and respectively inputting the corresponding control signals to the cache controller, the PE array, the result shaping module and the minimum value searching module. Optionally, the PEs in the PE array adopt an interconnection mode of up, down, left and right, and transfer intermediate results in a vertical direction, and transfer operands and matching costs in a horizontal direction. Optionally, one or more of the following types of PEs are included in the PE array: the Ultra PE is used for carrying out the difference absolute value of two operands and the accumulation operation of partial sum in the SAD matching cost calculation process; STANDARD PE, in the process of performing SAD matching cost calculation, performing difference absolute value calculation on two operands, and pe