Search

EP-4302267-B1 - REAL-TIME ACTIVE STEREO MATCHING

EP4302267B1EP 4302267 B1EP4302267 B1EP 4302267B1EP-4302267-B1

Inventors

  • NOVER, Harris
  • ACHAR, Supreeth
  • PRABHU, Kira
  • BHATAWADEKAR, VINEET

Dates

Publication Date
20260506
Application Date
20210303

Claims (15)

  1. A real-time active stereo system (100) comprising: a capture system (102) configured to capture stereo image data (114), the stereo image data (114) including reference images (130) and secondary images (132); and a depth sensing computing system (104) configured to generate a depth map (122), the depth sensing computing system (104) configured to: compute descriptors based on the reference images (130) and the secondary images (132); compute a stability penalty (128) based on pixel change information and disparity change information; and evaluate a plurality of plane hypotheses for a group of pixels using the descriptors, including: compute matching cost between the descriptors associated with each plane hypothesis; update the matching cost with the stability penalty (128); and select a plane hypothesis from the plurality of plane hypotheses for the group of pixels based on the updated matching cost, wherein the pixel change information includes a pixel change value (142) indicating an intensity change of the pixel and the disparity change information includes a disparity change value (158), the depth sensing computing system (104) configured to: compute an intensity multiplier (146) using the pixel change value (142); compute a disparity multiplier (164) using the disparity change value (158); and compute the stability penalty (128) using the intensity multiplier (146) and the disparity multiplier (164), wherein the pixel change value (142) represents a difference between a pixel value of a pixel in a reference image (130) or a secondary image (132) for a current depth map (122a) and a pixel value of the pixel in a reference image (130) or a secondary image (132) for a previous depth map (122b).
  2. The real-time active stereo system (100) of claim 1, wherein the depth sensing computing system (104) is configured to: apply an edge-aware filter (196) to the pixel change value (142) to derive a filtered pixel change value (144), the filtered pixel change value (144) being used to compute the intensity multiplier (146).
  3. The real-time active stereo system (100) of claim 1 or 2, wherein the disparity change value (158) represents a difference between a proposed disparity of a pixel for the current depth map (122a) and a disparity of the pixel for the previous depth map (122b).
  4. The real-time active stereo system (100) of any of claims 1 o 3, wherein the depth sensing computing system (104) is configured to: compute a maximum matching cost that can be produced during plane hypothesis evaluation; and compute the stability penalty (128) based on a product of the maximum matching cost, the intensity multiplier (146), and the disparity multiplier (164).
  5. The real-time active stereo system (100) of any of claims 1 to 4, wherein the depth sensing computing system (104) is configured to compute the intensity multiplier (146) using an intensity function inputted with the pixel change value (142).
  6. The real-time active stereo system (100) of any of claims 1 to 5, wherein the depth sensing computing system (104) is configured to compute the disparity multiplier (164) using a disparity function inputted with the pixel change value (142).
  7. The real-time active stereo system (100) of any of claims 1 to 6, wherein the depth sensing computing system (104) is configured to: filter the matching cost using an edge-aware filter (196), wherein the filtered matching cost is updated with the stability penalty (128).
  8. A method for real-time active stereo comprising: receiving stereo image data (114) including reference images (130) captured from a reference camera (108) and secondary images (132) captured from a secondary camera (110); computing descriptors based on the reference images (130) and the secondary images (132); computing a stability penalty (128) based on pixel change information and disparity change information; and evaluating a plurality of plane hypotheses for a group of pixels using the descriptors, including: computing matching cost between the descriptors associated with each plane hypothesis; updating the matching cost with the stability penalty (128); and selecting a plane hypothesis from the plurality of plane hypotheses for the group of pixels having a lowest updated matching cost, wherein the pixel change information includes a pixel change value (142) indicating an intensity change of the pixel and the disparity change information includes a disparity change value (158), the method further comprising: computing an intensity multiplier (146) using the pixel change value (142); computing a disparity multiplier (164) using the disparity change value (158); computing the stability penalty (128) using the intensity multiplier (146) and the disparity multiplier (164), and wherein the pixel change value (142) represents a difference from a pixel value of a pixel in a reference image (130) for a current depth map (122a) and a pixel value of the pixel in a reference image (130) for a previous depth map (122b).
  9. The method of claim 8, further comprising: applying an edge-aware filter (196) to the pixel change value (142) to derive a filtered pixel change value (144), the filtered pixel change value (144) being used to compute the intensity multiplier (146).
  10. The method of claim 8 or 9, wherein the disparity change value (158) represents a difference between a proposed disparity of a pixel for the current depth map (122a) and a disparity of the pixel for the previous depth map (122b).
  11. The method of any of claims 8 to 10, further comprising: computing a maximum matching cost that can be produced during plane hypothesis evaluation; and computing the stability penalty (128) based on a product of the maximum matching cost, the intensity multiplier (146), and the disparity multiplier (164).
  12. The method of any of claims 8 to 11, further comprising: computing the intensity multiplier (146) using an intensity function inputted with the pixel change value (142); and computing the disparity multiplier (164) using a disparity function inputted with the pixel change value (142), the disparity function being different than the intensity function.
  13. The method of any of claims 8 to 12, further comprising: computing the matching cost based on Hamming distances between the descriptors; and filtering the matching cost using an edge-aware filter (196), wherein the filtered matching cost are updated with the stability penalty (128).
  14. A non-transitory computer-readable medium (118) storing executable instructions that when executed by at least one processor (116) are configured to cause the at least one processor (116) to: receive stereo image data (114) including reference images (130) captured from a reference camera (108) and secondary images (132) captured from a secondary camera (110); compute descriptors based on the reference images (130) and the secondary images (132); compute a stability penalty (128) based on pixel change information and disparity change information; and evaluate a plurality of plane hypotheses for a group of pixels using the descriptors, including: compute matching cost between the descriptors for each plane hypothesis; update the matching cost with the stability penalty (128); and select a plane hypothesis from the plurality of plane hypotheses for the group of pixels based on the updated matching costs, wherein the pixel change information includes a pixel change value (142) indicating an intensity change of the pixel and the disparity change information includes a disparity change value (158), the executable instructions including instructions that when executed by the at least one processor (116) cause the at least one processor (116) to: compute an intensity multiplier (146) using the filtered pixel change value (144); compute a disparity multiplier (164) using the disparity change value (158); and compute the stability penalty (128) using the intensity multiplier (146) and the disparity multiplier (164), wherein the pixel change value (142) represents a difference from a pixel value of a pixel in a reference image (130) for a current depth map (122a) and a pixel value of the pixel in a reference image (130) for a previous depth map (122b).
  15. The non-transitory computer-readable medium (118) of claim 14, wherein the executable instructions include instructions that when executed by the at least one processor (116) cause the at least one processor (116) to perform the method of any one of claims 9 to 13.

Description

TECHNICAL FIELD Embodiments relate to a real-time active stereo system to increase the resolution and/or accuracy of depth maps. BACKGROUND Stereo matching, also known as stereo vision or disparity mapping, is a process to find the depth of a scene and involves capturing two images from different viewpoints and matching those images to locate disparities (e.g., differences in positions) of elements in the scene. Active stereo matching is a process that uses structured light to simplify the stereo matching problem. However, conventional active stereo matching techniques are computationally expensive, and the resolution and accuracy of these conventional systems may not meet the needs of applications requiring real-time processing and/or applications requiring a higher level of detail and accuracy. The article ESPReSSo: Efficient Slanted Patch Match for Real-time Spacetime Stereo by H. Nover, S. Achar, and D.B. Goldman, 2018 International Conference on 3D Vision (3DV), IEEE, p. 578 - 586, relates to a real-time implementation of spacetime stereo by a local stereo reconstruction algorithm that precomputes subpixel-shifted binary descriptors, then iteratively samples them along slanted disparity plane hypotheses, applying an edge-aware filter for spatial cost aggregation. Plane hypotheses are shared within rectangular tiles, but every pixel selects a different winner from among these candidates. US 2020/0027220 A1 describes a consistent belief propagation system including a disparity map buffer that provides a disparity map of a previous time and a belief propagation unit that generates an energy function according to a first image of a present time, a second image of the present time, a first image of a previous time and the disparity map of the previous time. A disparity generating unit generates a disparity map of the present time according to the energy function. SUMMARY According to an aspect, a real-time active stereo system includes the features of claim 1. According to another aspect, a method and a computer-readable medium are provided that include the features of claim 8 and claim 14, respectively. According to yet another aspect, further embodiments of the invention are indicated in the dependent claims. . BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1A illustrates a real-time active stereo system according to an aspect.FIG. 1B illustrates an example of stereo image data received by a depth sensing computing system of the real-time active stereo system according to an aspect.FIG. 1C illustrates an example of matching costs according to an aspect.FIG. 1D illustrates a graphical representation of the computation of descriptors according to an aspect.FIG. 1E illustrates a graphical representation of evaluating plane hypotheses for a group of pixels according to an aspect.FIG. 1F illustrates sub-steps of a plane evaluation loop executable by a local stereo reconstruction algorithm of the real-time active stereo system according to an aspect.FIG. 1G illustrates a timing diagram for the stereo images in a repeating pattern-pattern-guide sequence according to an aspect.FIG. 2A illustrates a comparison graph for comparing pixels according to an aspect.FIG. 2B illustrates a descriptor generator for reducing the amount of redundant information in a descriptor according to an aspect.FIG. 3 illustrates a flowchart depicting example operations of generating a depth map using descriptors generated by the descriptor generator according to an aspect.FIG. 4 illustrates a descriptor generator for generating reliability data according to an aspect.FIG. 5 illustrates a flowchart depicting example operations of generating a depth map in which matching costs are updated with the reliability data according to an aspect.FIG. 6A illustrates a stability module for increasing the stability of depth maps according to an aspect.FIG. 6B illustrates an example of the stability module for generating a stability penalty according to an aspect.FIG. 6C illustrates an intensity function of the stability module according to an aspect.FIG. 6D illustrates a disparity function of the stability module according to an aspect.FIG. 7 illustrates a flowchart depicting example operations of generating a depth map in which matching costs are updated with a stability penalty according to an aspect.FIG. 8 illustrates a confidence weight generator for generating a confidence weight using stereo internal data according to an aspect.FIG. 9 illustrates a flowchart depicting example operations of generating a confidence weight for each pixel's stereo depth estimate according to an aspect.FIG. 10 shows an example of a computer device and a mobile computer device according to an aspect.FIG. 11 illustrates a three-dimensional telepresence system according to an aspect.FIG. 12 illustrates, in block form, the three-dimensional telepresence system for conducting three-dimensional video conferencing between two users according to an aspect. DETAILED DESCRIPTION This