KR-20260067778-A - A METHOD FOR SEARCHING LOCATION INFORMATION IN 3D SPACE FOR 2D DATA USING PRECISE CALIBRATION AND DIRECTIONALITY OF OBJECT RECOGNITION INFORMATION

KR20260067778AKR 20260067778 AKR20260067778 AKR 20260067778AKR-20260067778-A

Abstract

The present invention relates to a method for searching for location information in three-dimensional space of two-dimensional data, comprising the steps of: performing two-dimensional object recognition on image data by at least one processor of a computer system; performing three-dimensional object recognition and clustering on point cloud data; detecting a two-dimensional bounding box from the two-dimensional object recognition; detecting a three-dimensional bounding box from the three-dimensional object recognition and clustering; calculating the center point coordinate direction of the two-dimensional bounding box; matching a three-dimensional bounding box that is most similar in direction to the calculated center point coordinate direction of the two-dimensional bounding box; estimating a depth value from information regarding the matched three-dimensional bounding box; and estimating the position of the two-dimensional bounding box in a world coordinate system using the estimated depth value. Accordingly, it is possible to supplement undetected and unclassified information by adding an image sensor and performing calibration, and thereby, detection performance can be improved without utilizing expensive high-resolution equipment, thereby enabling cost reduction.

Inventors

조세운
정성환
김병준
김동훈
김서정
현상미

Assignees

한국전자기술연구원

Dates

Publication Date: 20260513
Application Date: 20241106

Claims (13)

In a method for searching for location information of two-dimensional data in three-dimensional space, by at least one processor of a computer system, A step of performing 2D object recognition on image data; Step of performing 3D object recognition and clustering on point cloud data; A step of detecting a 2D bounding box from the above 2D object recognition; A step of detecting a 3D bounding box from the above 3D object recognition and clustering; A step of calculating the center point coordinate direction of the above 2D bounding box; A step of matching a 3D bounding box with the direction most similar to the center point coordinate direction of a calculated 2D bounding box; A step of estimating a depth value from information regarding a matched 3D bounding box; and A step of estimating the position of the above 2D bounding box in the world coordinate system using the estimated depth value. A method including
In Article 1, The step of matching the 3D bounding box with the direction most similar to the center point coordinate direction of the calculated 2D bounding box above is a critical rotation angle A method of searching for and matching a 3D bounding box with a smaller difference in direction.
In claim 2, the critical rotation angle A method of performing filtering based on the condition of the 3D bounding box size when a 3D object with a smaller difference in direction appears from the rear.
In Article 1, A method in which the step of estimating a depth value from information regarding the matched 3D bounding box is to estimate the depth value from the world coordinate system origin of the 3D object center as a depth value for a 2D object with respect to the matched 3D bounding box.
In Article 2, Based on sensor calibration information obtained through sensor calibration of the point cloud sensor and image sensor, the above threshold rotation angle A method of searching for and matching a 3D bounding box with a smaller difference in direction.
In Article 1, A method in which the step of calculating the center point coordinate direction of the above-described two-dimensional bounding box is performed based on sensor calibration information obtained through sensor calibration of a point cloud sensor and an image sensor.
In a method for searching for location information of two-dimensional data in three-dimensional space, by at least one processor of a computer system, A step of obtaining instance pixels associated with an object by performing instance segmentation on image data; A step of calculating orientation information in the camera coordinate system of individual instance pixels for acquired instance pixels; The extraction step of searching for n points in the direction closest to the direction of the individual instance pixel to extract n points per pixel from the point cloud data, wherein n is an integer greater than or equal to 1; and A step of estimating the position of an instance pixel in the world coordinate system using the depth of the point cloud in the direction closest to the direction of the individual instance pixel as the depth value of each pixel. A method including
In Article 7, A step of performing clustering by merging the points closest to the orientation of the extracted individual instance pixels and estimating the most robust cluster from the points; and A method further comprising the step of estimating the position of the instance pixel in the world coordinate system using the estimated depth of the cluster center as the depth value of the segmentation result.
In Article 7, A method further comprising the step of estimating the position of an instance pixel in the world coordinate system by computationally estimating a depth value in the pixel direction through extrapolation and interpolation from the n points explored above.
In Article 7, When searching for n points in the direction closest to the direction of the individual instance pixel mentioned above, the rotation angle with respect to the pixel point direction is the critical rotation angle A method that targets points that do not exceed.
In Article 10, Based on sensor calibration information obtained through sensor calibration of the point cloud sensor and image sensor, the above threshold rotation angle A method of searching for n points in the direction closest to the direction of the individual instance pixel above, targeting points that do not go beyond.
In Article 7, A method in which the step of calculating the orientation information of the individual instance pixels in the camera coordinate system is performed based on sensor calibration information obtained through sensor calibration of the point cloud sensor and the image sensor.
A computer-readable recording medium storing a computer program for executing a method according to any one of claims 1 to 12 on a computer system.

Description

A Method for Searching Location Information in 3D Space for 2D Data Using Precise Calibration and Directionality of Object Recognition Information The present invention relates to a method for searching for position information in three-dimensional space of two-dimensional data using precision calibration and the directionality of object recognition information. More specifically, the invention relates to a method for generating position information in three-dimensional space of a two-dimensional object for a camera performing two-dimensional and three-dimensional object recognition and a LiDAR performing three-dimensional object recognition, using calibration information obtained through precision calibration between a camera and a LiDAR. According to conventional technology, verification of the correlation between camera and LiDAR data can be performed by projecting 3D data into a 2D space using calibration information, or by projecting 2D pixel information into a 3D space only when additional depth information can be obtained. However, the error with the actual value of depth information calculated using a stereo camera increases as the distance increases, and distorted results occur depending on environmental conditions such as lighting or background. Furthermore, when using a mono camera and regression values from a depth prediction model, additional resources are required for the prediction model, it is difficult to generate training data for the camera to be used, and the accuracy is generally lower than that obtained using a stereo camera. Due to such low-accuracy depth measurement, there are difficulties in finding the location of 2D objects in 3D space. FIG. 1 is a schematic diagram showing the transformation relationship between a two-dimensional coordinate system and a three-dimensional coordinate system used in embodiments of the present invention. FIG. 2 is a flowchart illustrating a method for searching spatial location information using respective data and object detection results for image data and point cloud data according to an embodiment of the present invention. FIG. 3 is a flowchart illustrating a method for searching spatial location information between the instance segmentation result of image data and individual point cloud coordinates according to another embodiment of the present invention. Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. However, the present invention is not limited by the following embodiments. In addition, the same reference numerals are used for identical components in the drawings, and redundant descriptions thereof are omitted. In the present invention, position information in 3D space is searched by utilizing direction information and point cloud information obtained by projecting object recognition position information, which can be obtained through 2D object detection or segmentation, into 3D space. According to this method, since depth information of all pixels is not required, the calculation of depth values using a stereo camera or the prediction process using a depth value prediction model may not be performed, thereby enabling the efficient utilization of computational resources. Furthermore, by using sensor calibration information obtained through precise calibration, depth values that are not present in the image can be supplemented during image data processing by referencing them from 3D clusters in the same direction. Regarding the search for positional information of such two-dimensional data in three-dimensional space, FIG. 1 is a schematic diagram showing the transformation relationship between a two-dimensional coordinate system and a three-dimensional coordinate system used in embodiments of the present invention. Each coordinate system illustrated in FIG. 1 is examined in detail as follows. The world coordinate system (100) is a three-dimensional coordinate system in the actual world, and a reference point is arbitrarily selected and used to represent the position of an object. At this time, coordinate system conversion is possible only if the units of the world coordinate system (100) and the camera coordinate system (101) are the same. The camera coordinate system (101) is a three-dimensional coordinate system that determines direction and position based on the camera lens. That is, the camera position is (0, 0, 0), the z-axis indicates the direction the camera faces forward, the x-axis indicates the left and right sides of the camera, and the y-axis indicates the up and down sides of the camera. In order to match a point in the camera coordinate system (101) to the world coordinate system (100), the point must be rotated and moved to match the world coordinates, which is a rotation between the camera coordinate system (101) and the world coordinate system (100). and translation Consider the camera external parameters (105) related to the conversion. The normalized ima