US-12620110-B2 - Image processing device, stereo camera device, mobile object, disparity calculating method, and image processing method

US12620110B2US 12620110 B2US12620110 B2US 12620110B2US-12620110-B2

Abstract

An image processing device 20 includes a communication unit 21 and a controller 22 (processor). The communication unit 21 acquires a standard image and a reference image captured by a stereo camera 10 . The controller 22 calculates a disparity based on the standard image and the reference image. The controller 22 calculates, at each pixel position, a cost value representing a degree of difference for each pixel position between the standard image and each of multiple images obtained by displacing the reference image by multiple different deviation amounts in a direction corresponding to a baseline length direction of the stereo camera 10 . The processor aggregates the cost values using dynamic programming and calculates a disparity for each pixel position based on the aggregated cost values for each different deviation amount. The multiple different deviation amounts are discrete values at uneven intervals.

Inventors

Eijiro SHIBUSAWA
Hiromichi Sotodate

Assignees

KYOCERA CORPORATION

Dates

Publication Date: 20260505
Application Date: 20210715
Priority Date: 20200728

Claims (12)

1 . An image processing device comprising: an acquiring unit configured to acquire a standard image and a reference image captured by a stereo camera; and a processor configured to calculate disparity based on the standard image and the reference image, wherein the processor calculates, for each pixel position, a first cost value representing a degree of difference for each pixel position between the standard image and each of multiple images obtained by displacing the reference image by multiple different reference deviation amounts, in units of pixels, in a first direction corresponding to a baseline length direction of the stereo camera, calculates, for each pixel position, a second cost value representing a degree of difference for each pixel position between the standard image and each of multiple images obtained by displacing the standard image by multiple different standard deviation amounts in the first direction, and uses the second cost values to calculate a disparity for each pixel position by estimating a correction deviation amount that adjusts a reference deviation amount among the reference deviation amounts to a deviation amount which minimizes a cost value representing a degree of difference for each pixel position between the standard image and an image obtained by displacing the reference image in a first direction down to units of less than one pixel.
2 . The image processing device according to claim 1 , wherein the processor selects, from among the first cost values, a smallest first cost value calculated in accordance with the multiple different reference deviation amounts and one or more first cost values calculated in accordance with the reference deviation amounts including a first deviation amount that gives the smallest first cost value out of the multiple different reference deviation amounts and the reference deviation amounts near the first deviation amount, and uses the selected first cost values to estimate the correction deviation amount down to units of less than one pixel.
3 . The image processing device according to claim 2 , wherein the processor calculates the second cost values representing degrees of difference between the standard image and multiple images obtained by displacing the standard image by integer numbers of pixels at and near zero towards positive and negative sides along the first direction, and uses the calculated second cost values to estimate the correction deviation amount down to units of less than one pixel.
4 . The image processing device according to claim 3 , wherein the processor estimates the correction deviation amount down to units of less than one pixel by interpolating the second cost values for the standard deviation amounts, respectively, using a prescribed function and displacing the prescribed function in units of less than one pixel.
5 . The image processing device according to claim 4 , wherein the prescribed function is a linear function or a parabolic function.
6 . The image processing device according to claim 2 , wherein the processor estimates the correction deviation amount down to units of less than one pixel using only the first cost value corresponding to the first deviation amount as the first cost value.
7 . The image processing device according to claim 2 , wherein the processor estimates the correction deviation amount by using multiple cost values corresponding to multiple reference deviation amounts as the first cost values and solving simultaneous equations consisting of multiple equations in which each of the multiple cost values is represented by an equation including a corresponding the correction deviation amount.
8 . The image processing device according to claim 7 , wherein the multiple equations include a uniform correction term and the multiple reference deviation amounts consist of two deviation amounts.
9 . The image processing device according to claim 8 , wherein the processor determines that estimation of the correction deviation amount has failed when an absolute value of the correction term is greater than or equal to a first threshold.
10 . The image processing device according to claim 7 , wherein the multiple equations include different correction terms, the number of correction terms is greater than or equal to the number of the multiple reference deviation amounts, and the processor estimates the correction deviation amount so that a sum of squares of the correction terms is minimized.
11 . The image processing device according to claim 10 , wherein the processor determines that estimation of the correction deviation amount has failed when the sum of squares of the correction terms is greater than or equal to a second threshold.
12 . The image processing device according to claim 1 , wherein the processor determines that estimation of the correction deviation amount has failed when the estimated deviation amount is larger than a size of one pixel.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority of Japanese Patent Application No. 2020-127696, Japanese Patent Application No. 2020-141243, Japanese Patent Application No. 2020-141244, and Japanese Patent Application No. 2020-175540 respectively filed in Japan on Jul. 28, 2020, Aug. 24, 2020, Aug. 24, 2020, and Oct. 19, 2020, and the entire disclosures of these applications are hereby incorporated by reference. TECHNICAL FIELD The present disclosure relates to an image processing device, a stereo camera device, a mobile object, a disparity calculating method, and an image processing method. BACKGROUND OF INVENTION Heretofore, in a known method, the distance to a subject is calculated by calculating a disparity from two images captured by a stereo camera. A known method for obtaining the disparity of positions on two captured images is a block matching method (refer to, for example, Patent Literature 1). In the block matching method, a small region (block) of a standard image is displaced in a direction corresponding to a baseline length direction relative to a reference image and a degree of difference is evaluated. The disparity is the deviation amount of pixels having the smallest degree of difference. For example, the sum of absolute difference (SAD) of the brightnesses of pixels is used to evaluate the degree of difference. However, it may not be possible to extract image features from regions where the texture of the captured image is comparatively low (regions having low differences in luminance) and it may be difficult to calculate disparity using only a method based on block matching. Therefore, disparity estimation methods based on dynamic programming methods such as the semiglobal matching method (SGM method) have been proposed, which can be used to calculate disparity even in regions having comparatively low textures (refer to, for example, Patent Literature 1). In distance measurement using a stereo camera, a reference image is compared with a standard image in units of pixels, but when the object to be measured is at a long distance, the resolution of distance measurement is reduced. Therefore, a method has been proposed in which the distance is calculated by interpolating the disparity in units of less than one pixel, while assuming that the distribution of differences is symmetrical around a minimum point (refer to, for example, Patent Literature 2). In a digital stereo camera, a position in an image is recognized in units of pixels, and therefore a real-space position corresponding to a disparity of less than one pixel cannot be calculated. Use of sub-pixel estimation for estimating disparities of less than one pixel is known, but systematic errors called pixel locking can be caused by sub-pixel estimation. Accordingly, a method has been proposed for reducing systematic errors, such as the EEC method, which uses an interpolated image obtained by displacing one out of a pair of images by a distance of 0.5 pixels (refer to, for example, Patent Literature 3). CITATION LIST Patent Literature Patent Literature 1: Japanese Unexamined Patent Application Publication No. 2000-207695 Patent Literature 2: Japanese Unexamined Patent Application Publication No. 2000-283753 Patent Literature 3: International Publication No. 2004/063991 Non Patent Literature Non Patent Literature 1: H. Hirschmuller, Stereo Processing by Semiglobal Matching and Mutual Information, IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(2), 328-341, February 2008 SUMMARY In a First Aspect of the present disclosure, an image processing device includes an acquiring unit and a processor. The acquiring unit is configured to acquire a standard image and a reference image captured by a stereo camera. The processor is configured to calculate a disparity based on the standard image and the reference image. The processor is configured to calculate, for each pixel position, a cost value representing a degree of difference for each pixel position between the standard image and each of multiple images obtained by displacing the reference image by multiple different deviation amounts in a direction corresponding to a baseline length direction of the stereo camera. The processor is configured to aggregate the cost values using dynamic programming and calculate a disparity for each pixel position based on the aggregated cost values for each different deviation amount. The multiple different deviation amounts are discrete values at uneven intervals. In a Second Aspect of the present disclosure, an image processing device includes an acquiring unit and a processor. The acquiring unit is configured to acquire a standard image and a reference image captured by a stereo camera. The processor is configured to calculate a disparity based on the standard image and the reference image. The processor is configured to calculate, for each pixel position, a first cost value representing a degree of difference for ea