US-12626385-B2 - Apparatus, method, and storage medium

Abstract

An apparatus for calculating a parallax based on a first image and a second image calculates a cross-correlation value between a first criterion image set on the first image and a first reference image set on the second image, calculates an auto-correlation value between a second criterion image and a second reference image set on either one of the first image and the second image, calculates a parallax amount between the first image and the second image by using the cross-correlation value and the auto-correlation value, and corrects the cross-correlation value by using the auto-correlation value.

Inventors

• AZUSA TSUKAHARA

Assignees

• CANON KABUSHIKI KAISHA

Dates

Publication Date

20260512

Application Date

20230320

Priority Date

20220328

Claims (20)

1. 1 . An apparatus for calculating a parallax based on a first image and a second image, the apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to: calculate a cross-correlation value between a first criterion image set on the first image and a first reference image set on the second image; calculate an auto-correlation value between a second criterion image and a second reference image set on either one of the first image and the second image; calculate a parallax amount between the first image and the second image by using the cross-correlation value and the auto-correlation value; and correct the cross-correlation value by using the auto-correlation value of a formula, wherein the cross-correlation value is corrected using the formula depending on when a sub pixel parallax is estimated to exist on a positive side or a negative side in a parallax search direction.
2. 2 . The apparatus according to claim 1 , wherein the one or more processors further execute the instructions to: change a position of the first reference image to calculate at least two cross-correlation values; and move the position of the second reference image by at least +1 pixel and −1 pixel in an optional direction to calculate at least two auto-correlation values, wherein the parallax amount is calculated based on a corrected cross-correlation value or the cross-correlation value, and the corrected cross-correlation value, the corrected cross-correlation value being obtained by correcting the cross-correlation value by using neighboring values of a minimum value of the auto-correlation value.
3. 3 . The apparatus according to claim 2 , wherein the corrected cross-correlation value is calculated by using a ratio of two auto-correlation values involving different amounts of movement of the second reference image with respect to at least one cross-correlation value.
4. 4 . The apparatus according to claim 2 , wherein the corrected cross-correlation value is calculated by using a ratio of the cross-correlation value and the auto-correlation value with respect to the at least two cross-correlation values.
5. 5 . An apparatus for calculating a parallax based on a first image and a second image, the apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to: calculate a cross-correlation value between a first criterion image set on the first image and a first reference image set on the second image; calculate an auto-correlation value between a second criterion image and a second reference image set on either one of the first image and the second image; calculate a parallax amount between the first image and the second image by using the cross-correlation value and the auto-correlation value; and correct a fitting function by using the auto-correlation value of a formula, wherein the cross-correlation value is corrected using the formula depending on when a sub pixel parallax is estimated to exist on a positive side or a negative side in a parallax search direction.
6. 6 . An apparatus for calculating a parallax based on a first image and a second image, the apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to: calculate a cross-correlation value between a first criterion image set on the first image and a first reference image set on the second image; calculate an auto-correlation value between a second criterion image and a second reference image set on either one of the first image and the second image; calculate a parallax amount between the first image and the second image by using the cross-correlation value and the auto-correlation value; and calculate the parallax amount by using the cross-correlation value and the auto-correlation value belonging to a same fitting function, by using the auto-correlation value of a formula, wherein the cross-correlation value is corrected using the formula depending on when a sub pixel parallax is estimated to exist on a positive side or a negative side in a parallax search direction.
7. 7 . The apparatus according to claim 6 , wherein the auto-correlation value is calculated by moving the second reference image by at least +1 pixel or −1 pixel in an optional direction, and wherein the parallax amount is calculated by using a ratio of the auto-correlation value and the cross-correlation value belonging to the same fitting function, by using the auto-correlation value.
8. 8 . The apparatus according to claim 6 , wherein the parallax amount is calculated based on a plurality of combinations of the auto-correlation value and the cross-correlation value belonging to the same fitting function.
9. 9 . The apparatus according to claim 8 , wherein the parallax amount is an average value of a plurality of parallax amounts calculated based on a plurality of combinations of the cross-correlation value and the auto-correlation value belonging to the same fitting function.
10. 10 . The apparatus according to claim 1 , wherein the cross-correlation value is calculated by using either one of sum of squared differences, sum of absolute differences, and normalized cross-correlation, of the first criterion image and the first reference image, and wherein the auto-correlation value is calculated by using either one of sum of squared differences, sum of absolute differences, and normalized cross-correlation, of the second criterion image and the second reference image.
11. 11 . The apparatus according to claim 1 , wherein the one or more processors further execute the instructions to determine whether to perform correction via a parallax calculation unit, based on a reliability of the cross-correlation value or the auto-correlation value.
12. 12 . The apparatus according to claim 1 , wherein the one or more processors further execute the instructions to determine whether to perform correction via a parallax calculation unit, based on a magnitude of an asymmetry of an edge texture.
13. 13 . A method for calculating a parallax based on a first image and a second image, the method comprising: calculating a cross-correlation value between a first criterion image set on the first image and a first reference image set on the second image; calculating an auto-correlation value between a second criterion image and a second reference image set on either one of the first image and the second image; and calculating a parallax amount between the first image and the second image by using the cross-correlation value and the auto-correlation value, wherein, in the calculating the parallax amount, an asymmetry of a fitting function is estimated by using the auto-correlation value to correct the cross-correlation value of a formula, wherein the cross-correlation value is corrected using the formula depending on when a sub pixel parallax is estimated to exist on a positive side or a negative side in a parallax search direction.
14. 14 . The method according to claim 13 , further comprising: changing a position of the first reference image to calculate at least two cross-correlation values; and moving the position of the second reference image by at least +1 pixel and −1 pixel in an optional direction to calculate at least two auto-correlation values, wherein the parallax amount is calculated based on a corrected cross-correlation value or the cross-correlation value, and the corrected cross-correlation value, the corrected cross-correlation value being obtained by correcting the cross-correlation value by using neighboring values of a minimum value of the auto-correlation value.
15. 15 . The method according to claim 13 , wherein the cross-correlation value is calculated by using either one of sum of squared differences, sum of absolute differences, and normalized cross-correlation, of the first criterion image and the first reference image, and wherein the auto-correlation value is calculated by using either one of sum of squared differences, sum of absolute differences, and normalized cross-correlation, of the second criterion image and the second reference image.
16. 16 . The method according to claim 13 , further comprising determining whether to perform correction via a parallax calculation unit, based on one of a reliability of the cross-correlation value or the auto-correlation value and a magnitude of an asymmetry of an edge texture.
17. 17 . A non-transitory computer-readable storage medium storing a program for causing an apparatus for calculating a parallax based on a first image and a second image to execute a method comprising: calculating a cross-correlation value between a first criterion image set on the first image and a first reference image set on the second image; calculating an auto-correlation value between a second criterion image and a second reference image set on either one of the first image and the second image; and calculating a parallax amount between the first image and the second image by using the cross-correlation value and the auto-correlation value of a formula, wherein, in the calculating the parallax amount, an asymmetry of a fitting function is estimated by using the auto-correlation value to correct the cross-correlation value, wherein the cross-correlation value is corrected using the formula depending on when a sub pixel parallax is estimated to exist on a positive side or a negative side in a parallax search direction.
18. 18 . The non-transitory computer-readable storage medium according to claim 17 , further comprising: changing a position of the first reference image to calculate at least two cross-correlation values; and moving the position of the second reference image by at least +1 pixel and −1 pixel in an optional direction to calculate at least two auto-correlation values, wherein the parallax amount is calculated based on a corrected cross-correlation value or the cross-correlation value, and the corrected cross-correlation value, the corrected cross-correlation value being obtained by correcting the cross-correlation value by using neighboring values of a minimum value of the auto-correlation value.
19. 19 . The non-transitory computer-readable storage medium according to claim 17 , wherein the cross-correlation value is calculated by using either one of sum of squared differences, sum of absolute differences, and normalized cross-correlation, of the first criterion image and the first reference image, and wherein the auto-correlation value is calculated by using either one of sum of squared differences, sum of absolute differences, and normalized cross-correlation, of the second criterion image and the second reference image.
20. 20 . The non-transitory computer-readable storage medium according to claim 17 , further comprising determining whether to perform correction via a parallax calculation unit, based on one of a reliability of the cross-correlation value or the auto-correlation value and a magnitude of an asymmetry of an edge texture.

Description

BACKGROUND Technical Field The aspect of the embodiments relates to an image processing apparatus for calculating a parallax based on a plurality of images. Description of the Related Art There are techniques for calculating three-dimensional information based on a plurality of images. Examples of techniques for calculating three-dimensional information include a sub pixel estimation method. This method calculates a sub pixel parallax by fitting a predetermined function corresponding to a cross-correlation value calculation method with respect to cross-correlation having the lowest dissimilarity and the neighbor cross-correlation values. Japanese Patent Application Laid-Open No. 2020-112881 discusses a technique for estimating a sub pixel parallax through cross-correlation and then correcting an error due to an abrupt pixel value change of a captured image by using auto-correlation. However, in particular, the technique discussed in Japanese Patent Application Laid-Open No. 2020-112881 may be unable to accurately correct an error due to the asymmetry of an edge texture. SUMMARY According to an aspect of the embodiments, an apparatus for calculating a parallax based on a first image and a second image calculates a cross-correlation value between a first criterion image set on the first image and a first reference image set on the second image, calculates an auto-correlation value between a second criterion image and a second reference image set on either one of the first image and the second image, calculates a parallax amount between the first image and the second image by using the cross-correlation value and the auto-correlation value, and corrects the cross-correlation value by using the auto-correlation value. Further features of the disclosure will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS FIGS. 1A to 1C illustrate an imaging apparatus having an image processing apparatus according to a first exemplary embodiment. FIG. 2 illustrates a light flux received by an image sensor according to the first exemplary embodiment. FIGS. 3A and 3B illustrate the image processing apparatus according to the first exemplary embodiment. FIGS. 4A to 4C illustrate the image processing apparatus according to the first exemplary embodiment. FIG. 5A illustrates a criterion image. FIGS. 5B to 5E illustrate a relation between auto-correlation and cross-correlation. FIGS. 6A and 6B illustrate a cross-correlation correction unit according to the first exemplary embodiment. FIGS. 7A and 7B illustrate an image processing apparatus according to a second exemplary embodiment. FIGS. 8A and 8B illustrate an image processing apparatus according to a third exemplary embodiment. FIG. 9 illustrates an imaging apparatus according to a fourth exemplary embodiment. DESCRIPTION OF THE EMBODIMENTS The present invention will now be described with reference to exemplary embodiments and the accompanying drawings. The present invention is not limited to the descriptions of each exemplary embodiment. Exemplary embodiments of the present invention can be suitably combined. First Embodiment (Configuration of Imaging Apparatus) FIG. 1A schematically illustrates a configuration of an imaging apparatus according to a first exemplary embodiment of the present invention. Referring to FIG. 1A, the imaging apparatus 100 includes an image processing apparatus 110, an imaging unit 120, and a distance calculation unit 130. The imaging unit 120 includes an image sensor 121 and an optical system 122. The optical system 122 is an imaging lens of the imaging apparatus 100 and has a function of forming a subject image on the image sensor 121. The optical system 122 includes a plurality of lens groups (not illustrated), a diaphragm (not illustrated), and an exit pupil 123 disposed at a predetermined distance from the image sensor 121. According to the present specification, the z axis is parallel to an optical axis 140 of the optical system 122. The x and y axes are perpendicular to each other and are perpendicular to the optical axis 140. The image sensor 121 includes a complementary metal oxide semiconductor (CMOS) sensor or a charge coupled device (CCD) sensor. A subject image via the optical system 122 is focused on the image sensor 121, and photo-electrically converted to an image signal based on the subject image. FIG. 1B is an x-y cross-sectional view illustrating the image sensor 121. The image sensor 121 includes an array formed of a plurality of 2-row by 2-column pixel groups 150. A pixel group 150 includes four different pixels: diagonally arranged green pixels 150G1 and 150G2, a red pixel 150R, and a blue pixel 150B. FIG. 1C is a cross-sectional view schematically illustrating the pixel group 150 taken along the I-I′ line. Each pixel includes a light-receiving layer 182 and a light-guiding layer 181. The light-receiving layer 182 includes two different