CN-122016138-A - Visual touch sensor and sensing method thereof

CN122016138ACN 122016138 ACN122016138 ACN 122016138ACN-122016138-A

Abstract

The application relates to a visual touch sensor and a sensing method thereof. The visual touch sensing method is applied to a visual touch sensor provided with a soft elastomer and comprises the steps of configuring a light source, enabling the light source to irradiate a reflecting medium which is arranged on one side of the soft elastomer, contacting with an object and has non-directional reflecting characteristics or partial non-directional reflecting characteristics, of the soft elastomer through the soft elastomer, wherein the reflecting medium comprises a first reflecting medium which acts on internal environment reflection of the soft elastomer and a second reflecting medium which is used for marking, albedo of the first reflecting medium is different from albedo of the second reflecting medium, acquiring visual change images of local positions and normal lines of the reflecting medium due to the fact that the reflecting medium contacts with the object to the soft elastomer, and performing photometric stereo resolving on the visual change images to sense mechanical characteristic data of the soft elastomer in a current state. According to the application, the normal resolving blind area caused by the damage of the mark to the photometric stereo reflection model is eliminated by arranging the first reflection medium and the second reflection medium with different reflectivities.

Inventors

HU FUTAI
PAN XUN
HU LIHUI
WANG YUE
LI ZHIQIANG

Assignees

南京一目智能科技有限公司
深圳一目科技有限公司
一目传感(深圳)科技有限公司

Dates

Publication Date: 20260512
Application Date: 20251218

Claims (13)

1. A visual tactile sensation method applied to a visual tactile sensor provided with a soft elastomer, characterized by comprising the following steps: The method comprises the steps of configuring a light source, enabling the light source to irradiate a reflecting medium which is arranged on one side of the soft elastic body, contacting an object and has non-directional reflecting characteristics or partial non-directional reflecting characteristics through the soft elastic body, wherein the reflecting medium comprises a first reflecting medium which acts on the internal environment reflection of the soft elastic body and a second reflecting medium which is used for marking; The visual change image of local position and normal change of the reflecting medium caused by the contact of the soft elastomer with an object is acquired, wherein the visual change image comprises light intensity distribution information of the first reflecting medium and the second reflecting medium, which change along with the normal under the irradiation of the light source; And carrying out luminosity three-dimensional calculation on the visual change image, and sensing mechanical characteristic data of the soft elastomer in the current state based on the light intensity distribution information of the first reflecting medium and the second reflecting medium.
2. The method according to claim 1, wherein the mechanical feature data includes normal displacement and corresponding normal force, tangential displacement and corresponding tangential force of the soft elastomer, and the step of performing photometric stereo resolution on the visual change image and sensing the mechanical feature data in the current state of the soft elastomer based on the light intensity distribution information of the first reflective medium and the second reflective medium specifically includes the steps of: Based on the photometric stereo principle, calculating the three-dimensional contour feature of one side of the soft elastomer contacting with the object by utilizing the light intensity distribution information of the first reflecting medium and the second reflecting medium in the visual change image; Extracting displacement features and/or deformation features of the second reflective medium from the visually changing image; And (3) integrating the three-dimensional contour features with the displacement features and/or the deformation features, and calculating normal displacement, normal force, tangential displacement and tangential force of the soft elastomer in the current state.
3. The method according to claim 2, wherein said step of calculating the three-dimensional contour feature of the side of the soft elastic body contacting the object using the light intensity distribution information of the first reflective medium and the second reflective medium in the visually changing image based on the photometric stereo principle, further comprises the steps of: extracting reflection characteristic difference characteristics of a first reflection medium and a second reflection medium in the visual change image; And eliminating interference of the difference of the albedo of the first reflecting medium and the second reflecting medium on the photometric stereo solution according to the reflection characteristic difference characteristic.
4. The method according to claim 1, wherein the step of performing photometric stereo resolution on the visual change image, sensing the mechanical characteristic data of the soft elastomer in the current state based on the light intensity distribution information of the first reflective medium and the second reflective medium, specifically comprises the steps of: acquiring a non-contact reference image obtained by reflecting the first reflecting medium and the second reflecting medium when the soft elastic body is not contacted with an object; Extracting plane displacement information of the first reflecting medium and the second reflecting medium based on the non-contact reference image and the visual change image to establish a global displacement field model of the soft elastomer; And performing photometric stereo calculation based on the displacement information model to sense the mechanical characteristic data of the soft elastomer in the current state, wherein the mechanical characteristic data of the soft elastomer comprises normal displacement, normal force, tangential displacement, tangential force and torque of the global sum of the soft elastomer, the tangential displacement and the tangential force are determined based on the plane displacement information of the second reflecting medium, and the torque is determined based on the non-scattered component in the global displacement field model.
5. The method according to claim 1, wherein the step of performing photometric stereo resolution on the visual change image, sensing the mechanical characteristic data of the soft elastomer in the current state based on the light intensity distribution information of the first reflective medium and the second reflective medium, specifically comprises the steps of: when the soft elastic body is not contacted with an object, a light source is reflected by the first reflecting medium and the second reflecting medium to obtain a non-contact reference image; extracting local normal change characteristics of one side of the soft elastomer contacting with the object in the non-contact reference image and the visual change image according to photometric stereo calculation; extracting the positions of a first reflecting medium and a second reflecting medium and reflection characteristic difference data in the non-contact reference image and the visual change image; Integrating the local normal change characteristics with the position and reflection characteristic difference data to generate comparison information of a non-contact reference image and a visual change image; Inputting the comparison information into a pre-trained solving neural network model to output mechanical characteristic data of the soft elastomer, wherein the mechanical characteristic data of the soft elastomer comprises distributed normal displacement, normal force, tangential displacement and tangential force of the soft elastomer.
6. The method according to claim 1, characterized in that said step of acquiring a visually changing image of the local position and normal changes of the reflective medium due to the soft elastomer contacting the object, comprises in particular the steps of: continuously collecting images reflected by the reflecting medium to obtain a multi-frame image sequence under at least two groups of illumination angles, wherein the independent light source irradiates the reflecting medium in a mode of at least one of time-sharing illumination, different-wavelength illumination or different-polarization illumination; And carrying out time sequence synchronization processing on the multi-frame image sequence to obtain a visual change image, wherein the visual change image is obtained by separating illumination data corresponding to different illumination directions based on acquisition time difference, light wavelength difference or light polarization difference.
7. A visual and tactile sensor, which comprises a sensor body, characterized by comprising the following steps: A soft elastic body which is deformed in response to external pressure, wherein one side of the soft elastic body, which is contacted with an object, is provided with a reflecting medium with non-directional reflecting characteristics or partial non-directional reflecting characteristics, and the reflecting medium comprises a first reflecting medium which acts on the internal environment reflection of the soft elastic body and a second reflecting medium which is used for marking; a light source for illuminating the soft elastomer and providing illumination required for photometric stereo resolution; The light-sensitive structure can receive light rays of the reflection medium, which are illuminated by the light source and change due to local normal, and generate a visual change image, wherein the visual change image comprises light intensity distribution information of the first reflection medium and the second reflection medium, and the visual change image is used for luminosity three-dimensional calculation so as to obtain mechanical characteristic data of the soft elastomer.
8. A tactile-viewing sensor according to claim 7, wherein said light source comprises at least two sets of independent light sources, each set of independent light sources having a different direction of illumination, said reflective medium being individually illuminable from different directions.
9. The visual touch sensor of claim 7, wherein the light source comprises at least two sets of independent light sources, wherein the light emission wavelengths of the at least two sets of independent light sources are different.
10. The tactile sensor according to claim 7, wherein the first reflective medium is a reflective layer covered on a stress side surface of the soft elastic body, and the second reflective medium is a block-shaped logo of the soft elastic body disposed on the reflective layer and/or on a side close to the reflective layer.
11. The visual touch sensor according to claim 7, wherein the surface of the stressed side of the soft elastomer is covered with a light shielding layer, the light shielding layer is used for shielding external ambient light from entering the photosensitive unit, and the second reflecting medium is a block sign of the soft elastomer distributed on the light shielding layer and/or on the side close to the light shielding layer.
12. A tactile sensor according to claim 7 or 11, wherein the first reflective medium is a layer of scattering particles arranged in a soft elastomer, the layer of scattering particles being constituted by one or more agglomerated scattering particles.
13. A tactile sensor according to claim 7, wherein said second reflective medium is a block-shaped logo, said block-shaped logo being provided with different albedo according to its location area of distribution of the soft elastomer.

Description

Visual touch sensor and sensing method thereof Technical Field The invention relates to the technical field of sensors, in particular to a visual touch sensor and a sensing method thereof. Background The Visual touch sensor (Visual-Tactile Sensor) is used as a high-precision sensor capable of simulating human touch perception, and captures deformation information of the elastic body when the elastic body is stressed by optical means so as to infer mechanical characteristics of a contact interface. In the fields of smart operation of robots, medical palpation, man-machine interaction and the like, a visual touch sensor is paid attention to because of the advantages of high spatial resolution, electromagnetic interference resistance and the like. In the existing visual and tactile perception technology, photometric stereo (Photometric Stereo) is one of core technologies for realizing high-resolution three-dimensional morphology reconstruction. The basic principle is that the elastic body surface is irradiated by a multi-angle light source, and a continuous surface Normal field is reconstructed according to the brightness change (Shading) of the reflected light intensity of the surface, so that the Normal Force (Normal Force) and the surface texture are solved with high precision. On the other hand, in order to sense tangential Force (Shear Force) or detect slippage, the prior art generally requires the placement of Marker points (markers) on or within the elastomer surface, and the tangential deformation is calculated by tracking the displacement of the Marker points within the image plane. However, in an attempt to achieve both high-precision normal perception and tangential perception, existing visual tactile sensors face the technical difficulty of "point-of-mark destruction photometric stereo-solution model". Although some documents have attempted to solve the relevant problems, it is difficult to completely eliminate the "resolving blind area" introduced by the marker points under the photometric stereo resolving framework. For example, european patent application publication No. EP3693139A1 proposes an optical tactile sensor that resolves the normal and lateral components of force by embedding an array of multilayer marks in an elastic layer. The scheme uses a combination of partially transparent and opaque markers (opaque markers). However, this approach does not take into account the impeding effect of the opaque markers on the photometric stereo-resolution. Upon surface normal reconstruction, these opaque markers may block the transmission of the optical path or exhibit reflection characteristics that are distinct from the surrounding medium, resulting in an algorithm that cannot obtain valid normal gradient information in the region covered by the marker. As another example, international patent application publication No. WO2022/264472A1 discloses an optical tactile sensor with a wear-resistant surface layer configured with a secondary pattern (e.g., a black matrix printed grid or lattice) on the back of the first layer to detect deformation. While this approach improves the durability of the sensor by layering, the secondary pattern that it uses is typically designed as a light absorbing material (e.g., black) with high contrast to the background. Although the design is beneficial to the two-dimensional tracking of the characteristic points, for the photometric stereo algorithm which depends on light reflection, the light absorption patterns cannot generate diffuse reflection light intensity which changes along with the normal direction, so that the loss of local normal information is caused. For another example, chinese patent application publication No. CN1842701a describes an optical tactile sensor for large area force vector distribution measurement that uses multiple colored markers (e.g., red, blue spherical markers) in a transparent elastomer to capture the motion of the markers by multiple cameras to calculate the force vector. However, this approach is essentially a measurement method based on discrete feature points. The sensor can only acquire displacement data at the marker points which are sparsely distributed, and the sensor lacks direct measurement information in a wide area between the marker points. More importantly, the scheme does not involve the use of photometric changes of the marker points themselves for continuous surface reconstruction. However, the problem of "dead zone" caused by the introduction of marks is not considered in the prior art, as shown in fig. 17, and fig. 17 is an acquired image of a typical visual touch sensor under different stress states in the prior art. In fig. 17, the background area exhibits a rich texture or color gradient as a function of surface deformation. However, the mark points (black dots shown in the figure) distributed therein always appear as uniform color patches of low albedo. This is because prior art designs typically design the mark