CN-115631086-B - Geometric correction device for real-time image sequence

CN115631086BCN 115631086 BCN115631086 BCN 115631086BCN-115631086-B

Abstract

The invention relates to a geometric correction device of a real-time image sequence, wherein a pipeline real-time calculation module is used for calculating the position coordinates of an output image pixel point relative to the position coordinates of an input image pixel point in real time according to calibrated camera parameters, an input buffer unit is used for buffering input image data, an input buffer management module is used for managing the input image data in the input buffer unit according to the integer part of the position coordinates of the input image pixel point, an interpolation calculation module is used for taking the decimal part of the position coordinates of the input image pixel point as the weight of interpolation calculation and carrying out interpolation calculation on the input image data to obtain a corrected image value, and a correction system control module is used for starting the data processing of the pipeline real-time calculation module and the control input buffer management module so as to complete time sequence control from an input distortion graph to an output correction graph. The invention can realize the correction of the high-real-time image sequence with less embedded hardware resources.

Inventors

LUO YIHUA
FU FENGJIE
LIU DEHUA
LI JIAMAO
WANG LEI
ZHANG XIAOLIN

Assignees

中国科学院上海微系统与信息技术研究所

Dates

Publication Date: 20260508
Application Date: 20221025

Claims (8)

1. A geometric correction device of a real-time image sequence is characterized by comprising a correction system control module, a pipeline real-time calculation module, an input buffer management module, an input buffer unit and an interpolation calculation module, wherein the pipeline real-time calculation module is used for calculating the position coordinates of output image pixels relative to the position coordinates of the input image pixels in real time according to calibrated camera parameters, the input end of the input buffer unit is connected with an input interface and used for buffering input image data, the input buffer management module is used for managing the input image data in the input buffer unit according to integer parts of the position coordinates of the input image pixels, the input end of the interpolation calculation module is connected with the output end of the input buffer unit, the output end of the interpolation calculation module is connected with an output interface and used for carrying out interpolation calculation on the input image data to obtain corrected image values, the correction system control module is used for starting the pipeline real-time calculation module and controlling the data processing of the input buffer management module so as to complete the control from an input distortion correction graph to an output correction graph, and the real-time sequence calculation module is used for carrying out the real-time calculation module through the pipeline calculation module Calculating the position coordinates of the pixel points of the output image relative to the position coordinates of the pixel points of the input image in real time, wherein, To output normalized position coordinates of the image pixels, , To input the position coordinates of the image pixels, As a radial distortion parameter of the camera, Is a tangential distortion parameter of the camera.
2. The apparatus according to claim 1, wherein the input interface and the output interface use a stream interface as a main mode, while supporting an AXI interface as a standby mode.
3. The apparatus according to claim 1, wherein the pipeline real-time calculation module is a 7-stage pipeline real-time calculation module, and the real-time calculation module calculates the real-time image sequence based on the real-time image sequence The first stage operation after decomposition comprises the following steps: , wherein, To output the coordinates of the pixels of the image, The second-stage operation comprises: 、 And The third stage operation includes: 、 And The fourth stage operation includes: And The fifth stage operation includes: And The sixth stage operation includes: the seventh stage operation includes: And 。
4. The geometric correction device of the real-time image sequence according to claim 1, wherein the input buffer management module dynamically updates a line range of input image data in the input buffer unit, specifically, after a line range of input image data buffer corresponding to a new output line is generated, maximum and minimum values of the line range are obtained through sequencing and compared with the maximum and minimum values corresponding to a previous line range, if the minimum value of the current line range is equal to the minimum value of the previous line range, data of the minimum line is reserved, if the minimum value of the current line range is greater than the minimum value of the previous line range, data smaller than the current minimum line is deleted, if the maximum value of the current line range is greater than the maximum value of the previous line range, new input image data does not need to be buffered, if the maximum value of the current line range is greater than the maximum value of the previous line range, new input image data needs to be continuously buffered until data in the maximum line range is buffered, and data reading of a next line is started.
5. The device for geometry correction of real-time image sequence according to claim 4, wherein said input buffer unit is updated by a loop update method for the input image data buffer, said loop update method is divided into 4 states, the first state is to buffer data from the lowest address and meet the storage requirement of the maximum line range, when the storage line range is updated, the second state is entered, the second state is to delete useless data and release the low address space, when the buffered data reaches the maximum value of the buffer address, the third state is corresponding, the storage address of the maximum line jumps to the initial address of the buffer, and the subsequent data is continuously buffered, and at the same time, the fourth state is entered.
6. The device for geometric correction of real-time image sequences according to claim 1, wherein an output image buffer is further provided between the output end of the interpolation computation module and the output interface, and the output image buffer is provided as a feedback signal to the input buffer management module through a fast full flag.
7. The apparatus according to claim 6, wherein the input buffer management module obtains output data by fetching the input image data in the input buffer unit and calculating in the interpolation calculation module and outputs the output data through the output image buffer when the fast full flag is low, and wherein the input buffer management module stops fetching the data when the fast full flag is high.
8. The device according to claim 6, wherein the fast full flag is low when the margin of the output image buffer is less than half the depth, and the fast full flag is high when the margin of the output image buffer is greater than or equal to half the depth, wherein the depth is 2 times the maximum value of two values of a first clock period n and a second clock period m, wherein the first clock period n is a clock period from when the fast full flag is just low to when the output image buffer obtains the first data, and wherein the second clock period m is a clock period from when the fast full flag is just high to when the data stops writing the output image buffer.

Description

Geometric correction device for real-time image sequence Technical Field The invention relates to the technical field of image correction, in particular to a geometric correction device for a real-time image sequence. Background The selection and design of the sensor subsystem directly affects the final performance of the whole system in the fields of autonomous navigation, autopilot, industrial detection and the like. Compared with active vision sensors such as laser radar, passive vision sensors represented by image sensors have irreplaceable significant advantages due to abundant texture information, and thus are being studied and applied more and more. However, the extra large data volume of the image information brings about an increase in operation cost and system power consumption and a decrease in real-time performance, which also becomes a disadvantageous factor affecting the application and popularization thereof. Binocular or multi-purpose image sensors allow the problem to be multiplied, further increasing the difficulty of application. Therefore, the establishment of the correction method with high real-time performance and the realization of the correction device with low power consumption and low cost have important significance for popularization of the vision sensor application. Image correction is one fundamental operation of passive vision sensors. In particular, in binocular stereo vision applications, calibration and correction of binocular image sequences is the basis and premise of binocular stereo vision sensors to develop back-end vision algorithms. The basic principle of image correction is to establish the mapping relation between the pixels of the output image and the pixels of the input image based on the optical lens model. The conventional image correction processing method is to buffer the image data of the whole frame, and then sequentially take out the image data from the storage medium for interpolation calculation according to the position of the calculation target pixel in the original image, so as to obtain a corrected output image. The conventional image sequence mapping technical scheme comprises (1) a GPU scheme of a general CPU, wherein the scheme comprises modes adopted by general operation units such as personal computers, industrial computers and servers, the scheme is high in price and power consumption, the whole frame of image data needs to be cached and serial mapping processing is executed, and the real-time performance is low, and (2) the conventional FPGA mapping scheme is required to calculate a mapping table in advance and preprocess and store the mapping table, and due to the fact that the data volume of the mapping table is large, DDR memories which are accessed in different forms are often relied on, the requirements of cache resources and system bandwidth are greatly increased, and hardware cost and power consumption are increased. Therefore, the embedded application with limited hardware resources is difficult to realize in the existing scheme, and the embedded application is characterized in that 1) a large amount of storage resources are required to be consumed if the whole frame of image data is cached by using an on-chip storage medium, 2) a large amount of DDR bandwidth is occupied if the whole frame of image data is cached to a DDR storage medium, 3) image correction is performed after the whole frame of image is cached, the delay of image processing is increased, and the instantaneity is reduced. Disclosure of Invention The invention aims to solve the technical problem of providing a geometric correction device for a real-time image sequence, which can realize the correction of the high-real-time image sequence by using less embedded hardware resources. The technical scheme includes that the geometrical correction device for the real-time image sequence comprises a correction system control module, a pipeline real-time calculation module, an input buffer management module, an input buffer unit and an interpolation calculation module, wherein the pipeline real-time calculation module is used for calculating position coordinates of output image pixels relative to position coordinates of input image pixels in real time according to calibrated camera parameters, the input end of the input buffer unit is connected with an input interface and used for buffering input image data, the input buffer management module is used for managing the input image data in the input buffer unit according to integer parts of the position coordinates of the input image pixels, the input end of the interpolation calculation module is connected with the output end of the input buffer unit, the output end of the interpolation calculation module is connected with the output interface and used for taking fractional parts of the position coordinates of the input image pixels as interpolation calculation weights to conduct interpolation calculation on the input image data,