US-12620107-B2 - Enhanced depth estimation using deep learning

Abstract

Retrographic sensors described herein may provide smaller sensors capable of high-resolution three-dimensional reconstruction of an object in contact with the sensor. Such sensors may be used by robots for work in narrow environments, fine manipulation tasks, and other applications. To provide a smaller sensor, a reduced number of light sources may be provided in the sensor in some embodiments. For example, three light sources, two light sources, or one light source, may be used in some sensors. When fewer light sources are provided, full color gradient information may not be provided. Instead, the missing gradients in one direction or other information related to a three-dimensional object in contact with the sensor may be determined using gradients in a different direction that were provided by the real data. This may be done using a trained statistical model, such as a neural network, in some embodiments.

Inventors

• Shaoxiong Wang
• Branden Romero
• Yu She
• Edward Adelson

Assignees

• MASSACHUSETTS INSTITUTE OF TECHNOLOGY

Dates

Publication Date

20260505

Application Date

20220106

Claims (20)

1. 1 . A method comprising: obtaining first gradients of a parameter along a first direction of an image of a three-dimensional object pressed into an elastomeric pad of a retrographic sensor; inputting, to a trained statistical model, the first gradients, wherein the trained statistical model is trained to output second gradients of the parameter along a second direction different from the first direction using the first gradients; and outputting, from the trained statistical model, the second gradients.
2. 2 . The method of claim 1 , further comprising calculating depth information related to the three-dimensional object using the first gradients and the second gradients.
3. 3 . The method of claim 1 , wherein the retrographic sensor comprises only one light source, the only one light source being a first light source configured to emit light along a first direction.
4. 4 . The method of claim 1 , wherein the retrographic sensor comprises only two light sources, the only two light sources being a first light source configured to emit light along a first direction and a second light source configured to emit light along a second direction different than the first direction.
5. 5 . The method of claim 1 , wherein: the parameter is an intensity of one or more wavelength bands; the first gradients of the parameter along the first direction comprise derivatives of the intensity of the one or more wavelength bands along the first direction; and the second gradients of the parameter along the second direction comprise estimates of derivatives of the intensity of the one or more wavelength bands along the second direction.
6. 6 . The method of claim 2 , wherein calculating depth information related to the three-dimensional object using the first gradients and the second gradients comprises applying a Poisson solver to the first gradients the second gradients.
7. 7 . A method comprising: obtaining first gradients of a parameter along a first direction of an image of a three-dimensional object pressed into an elastomeric pad of a retrographic sensor; obtaining training data, wherein the training data comprises second gradients of the parameter along a second direction different from the first direction; generating a trained statistical model using the first gradients and the training data; and storing the trained statistical model for subsequent use.
8. 8 . A retrographic sensor comprising: a transparent structure; a transparent elastomeric pad disposed on a surface of the transparent structure; a first light source configured to emit light into the transparent structure along a first direction; at least one computer processor; and at least one non-transitory computer-readable storage medium encoded with a plurality of instructions that, when executed by at least one computer processor, perform a method comprising: obtaining first gradients of a parameter along the first direction of an image of a three-dimensional object pressed into the elastomeric pad of the retrographic sensor; inputting, to a trained statistical model, the first gradients, wherein the trained statistical model is trained to output second gradients of the parameter along a second direction different from the first direction using the first gradients; and outputting, from the trained statistical model, the second gradients along the second direction.
9. 9 . The retrographic sensor of claim 8 , wherein the method further comprises calculating depth information related to the three-dimensional object using the first gradients and the second gradients.
10. 10 . The retrographic sensor of claim 8 , wherein the retrographic sensor comprises only one light source, the only one light source being the first light source.
11. 11 . The retrographic sensor of claim 8 , wherein the retrographic sensor comprises only two light sources, the only two light sources being the first light source and a second light source configured to emit light along a second direction different than the first direction.
12. 12 . The retrographic sensor of claim 8 , wherein: the parameter is an intensity of one or more wavelength bands; the first gradients of the parameter along the first direction comprise derivatives of the intensity of the one or more wavelength bands along the first direction; and the second gradients of the parameter along the second direction comprise estimates of derivatives of the intensity of the one or more wavelength bands along the second direction.
13. 13 . The retrographic sensor of claim 9 , wherein calculating depth information related to the three-dimensional object using the first gradients and the second gradients comprises applying a Poisson solver to the first gradients the second gradients.
14. 14 . The retrographic sensor of claim 8 , further comprising a photosensitive detector configured to provide the image of the three-dimensional object pressed into the elastomeric pad of the retrographic sensor.
15. 15 . A method comprising: obtaining first gradients of a parameter along a first direction of an image of a three-dimensional object pressed into an elastomeric pad of a retrographic sensor, wherein the retrographic sensor includes at most two light sources configured to illuminate the elastomeric pad in different directions; inputting, to a trained statistical model, the first gradients; and calculating depth information related to the three-dimensional object based at least in part on an output from the trained statistical model.
16. 16 . The method of claim 15 , further comprising outputting, from the trained statistical model, second gradients of the parameter along a second direction different than the first direction.
17. 17 . The method of claim 16 , wherein: the parameter is an intensity of one or more wavelength bands; the first gradients of the parameter along the first direction comprise derivatives of the intensity of the one or more wavelength bands along the first direction; and the second gradients of the parameter along the second direction comprise estimates of derivatives of the intensity of the one or more wavelength bands along the second direction.
18. 18 . The method of claim 15 , wherein calculating depth information related to the three-dimensional object using the first gradients comprises applying a Poisson solver to the first gradients.
19. 19 . The method of claim 15 , further comprising obtaining the image of the three-dimensional object pressed into an elastomeric pad of a retrographic sensor, the image including at most two wavelength bands of two respective light sources of the retrographic sensor.
20. 20 . The method of claim 15 , wherein calculating the depth information includes the trained statistical model outputting the depth information based on the first gradients.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application is a national stage filing under 35 U.S.C. § 371 of international application serial number PCT/US2022/011363, filed Jan. 6, 2022, which claims the benefit under 35 U.S.C. § 119(e) of U.S. provisional application Ser. No. 63/158,327, filed Mar. 8, 2021, the disclosure of each of which is incorporated herein by reference in its entirety. GOVERNMENT LICENSE RIGHTS This invention was made with Government support under N00014-18-1-2815 awarded by the Office of Naval Research. The Government has certain rights in the invention. FIELD Disclosed embodiments are related to retrographic sensors and related methods of use. BACKGROUND The sense of touch contributes to the dexterity of human manipulation, especially in cases where high precision is desirable. The complex ensemble of mechanoreceptors in the human hand provides extremely rich tactile sensory signals. These sensory signals encode information such as contact force and contact shape, and can be used to detect complex state transitions such as making or breaking contact or the occurrence of slippage between the finger and the grasped object. Vision based tactile sensors have become employed with robotic systems due to their high signal resolutions and the softness of their sensing surfaces. The softness of the sensing surface allows for larger contact regions as it deforms to conform with the object surface. The resulting contact areas are then characterized in great detail via the high resolution signals. Together, these properties have enabled the use of these sensors in tackling several tasks such as assessing grasp success, servoing object surfaces, detecting slip and shear force, reconstructing 3D surfaces, and distinguishing between different cloth materials. SUMMARY According to one aspect of the disclosure, there is provided a method comprising, obtaining first gradients of a parameter along a first direction of an image of a three-dimensional object pressed into an elastomeric pad of a retrographic sensor, inputting, to a trained statistical model, the first gradients, wherein the trained statistical model is trained to output second gradients of the parameter along a second direction different from the first direction using the first gradients, and outputting, from the trained statistical model, the second gradients. In some embodiments, the method further comprises calculating depth information related to the three-dimensional object using the first gradients and the second gradients. In some embodiments, the retrographic sensor comprises only one light source, the only one light source being a first light source configured to emit light along a first direction. In some embodiments, the retrographic sensor comprises only two light sources, the only two light sources being a first light source configured to emit light along a first direction and a second light source configured to emit light along a second direction different than the first direction. In some embodiments, the parameter is an intensity of one or more wavelength bands, the first gradients of the parameter along the first direction comprise derivatives of the intensity of the one or more wavelength bands along the first direction, and the second gradients of the parameter along the second direction comprise estimates of derivatives of the intensity of the one or more wavelength bands along the second direction. In some embodiments, calculating depth information related to the three-dimensional object using the first gradients and the second gradients comprises applying a Poisson solver to the first gradients the second gradients. According to one aspect of the disclosure, there is provided a method comprising obtaining first gradients of a parameter along a first direction of an image of a three-dimensional object pressed into an elastomeric pad of a retrographic sensor, obtaining training data, wherein the training data comprises second gradients of the parameter along a second direction different from the first direction, generating a trained statistical model using the first gradients and the training data, and storing the trained statistical model for subsequent use. In some embodiments, the trained statistical model is a neural network model. In some embodiments, the parameter is an intensity of one or more wavelength bands. In some embodiments, the first gradients of the parameter along the first direction comprise derivatives of the intensity of the one or more wavelength bands along the first direction and the second gradients of the parameter along the second direction comprise estimates of derivatives of the intensity of the one or more wavelength bands along the second direction. According to one aspect of the disclosure, there is provided a retrographic sensor comprising a transparent structure, a transparent elastomeric pad disposed on a surface of the transparent structure, a first light source configured to emit l