CN-122008301-A - Visual touch integrated manipulator based on sparse visual angle three-dimensional reconstruction and hardness nondestructive testing method

CN122008301ACN 122008301 ACN122008301 ACN 122008301ACN-122008301-A

Abstract

The invention discloses a vision-touch integrated manipulator based on sparse visual angle three-dimensional reconstruction and a hardness nondestructive testing method. The hardware of the manipulator mainly comprises a mechanical clamping jaw, a transparent PDMS film, a square reflecting lens barrel, a polaroid, an annular polarized light source, a camera and a manipulator interface. The acquired images are transmitted to external computing equipment and input to a depth learning network based on a transducer architecture, and a high-precision three-dimensional point cloud model of the deformation region is directly reconstructed. The external computing equipment processes the point cloud to accurately calculate the maximum deformation depth, and converts the maximum deformation depth into a quantized hardness index by combining a pre-calibrated force-displacement relation model, so as to classify and judge the maturity level of the object. The invention breaks through the limitation of the traditional detection depending on apparent characteristics, realizes the nondestructive detection with high precision and high reliability by directly quantifying the fundamental physical characteristics, and has wide application prospect in the fields of automatic perception and quality control.

Inventors

ZOU RONG
LI SIYANG
GUO JING
LIU YUQIN

Assignees

江苏大学

Dates

Publication Date: 20260512
Application Date: 20260311

Claims (7)

1. The vision-touch integrated manipulator based on sparse visual angle three-dimensional reconstruction is characterized by comprising a mechanical clamping jaw (1), a transparent PDMS film (2), a square reflecting lens barrel (3), a polaroid (4), an annular polarized light source (5), a camera (6) and a mechanical arm interface (7); The mechanical clamping jaw (1) is configured to perform a controlled pressing action to enable an external tested object to press the transparent PDMS film (2); the transparent PDMS film (2) is fixed on the surface of the square reflecting lens cone (3) and is used as a sensing interface for contacting and being pressed with an external tested object; the square reflecting lens barrel (3) is configured to fold a light path, so that the camera (6) can synchronously capture a direct central view angle and a plurality of virtual indirect view angle images of the deformation area of the transparent PDMS film (2) in single shooting; The annular polarized light source (5) and the camera (6) are coaxially arranged, and the polaroid sheet (4) is arranged in front of a lens of the camera (6) and is used for inhibiting specular reflection on the surface of the transparent PDMS film (2); the camera (6) is arranged at one end of the square reflecting lens barrel (3) far away from the transparent PDMS film (2), and the light path faces the transparent PDMS film (2); The camera (6) is in communication connection with an external computing device and is used for transmitting the acquired multi-view images to the external computing device for three-dimensional reconstruction and quantitative evaluation by the external computing device; the mechanical arm interface (7) is used for being connected with an external supporting structure.
2. The vision-touch integrated manipulator according to claim 1, characterized in that the external computing device is configured to receive the multi-view image acquired by the camera (6) in cooperation with the square reflecting lens barrel (3), reconstruct the three-dimensional deformation of the transparent PDMS film (2), calculate the maximum deformation depth based on the three-dimensional deformation, and then calculate the hardness index of the external measured object in combination with a pre-calibrated force-displacement model, so as to quantitatively evaluate the hardness level of the external measured object.
3. The visual touch integrated manipulator hardness nondestructive testing method based on sparse visual angle three-dimensional reconstruction according to any one of claims 1-2 is characterized in that a logic calculation module operated by external computing equipment comprises a system calibration module, a mode judgment module, a three-dimensional reconstruction module, a point cloud processing module, a hardness evaluation module and a visual classification module, wherein the specific implementation comprises the following steps: the system calibration module applies a series of known displacements to the transparent PDMS film (2) through external precision mechanical testing equipment, synchronously records the force born by the transparent PDMS film, and establishes a force-displacement function relation model of the transparent PDMS film (2) through data fitting; the detection mode judgment comprises the steps of acquiring an initial image of an external detected object acquired by the camera (6) by the mode judgment module, judging whether the external detected object is applicable to a pure visual detection mode or not, executing a visual detection branch if the external detected object is applicable to the pure visual detection mode, and executing a touch detection branch if the external detected object is not applicable to the pure visual detection mode; The external computing equipment is configured to run a visual classification module based on YOLOv framework when adopting a pure visual detection method, the module adopts a self-adaptive sparse neural architecture and can dynamically adjust the network calculation amount according to the calculation force of deployment hardware; A tactile detection branch comprising the sub-steps of: The method comprises the steps of collecting images, namely enabling an external mechanical clamping jaw (1) to enable a tested object to be in contact with a transparent PDMS film (2), applying a preset pressure under a controlled state to enable the transparent PDMS film (2) to generate elastic deformation, and then utilizing a camera (6) to be matched with a square reflecting lens cone (3) to shoot a multi-view image comprising a direct central view angle and a plurality of virtual indirect views of a deformation area of the transparent PDMS film (2); Inputting the acquired multi-view images into a pre-trained end-to-end deep learning network model based on a transducer architecture by the three-dimensional reconstruction module, wherein the network model can directly regress a dense three-dimensional coordinate diagram from the input multi-view images so as to generate a three-dimensional point cloud representing the deformed surface of the transparent PDMS film (2); The point cloud processing module is configured to register the output three-dimensional point cloud through a random sampling consensus algorithm RANSAC, and divide the three-dimensional point cloud by combining color space analysis and a density clustering algorithm DBSCAN so as to extract an effective deformation area of the transparent PDMS film (2); The hardness evaluation module calculates the maximum deformation depth value from the processed effective deformation area point cloud, converts the depth value into a corresponding force value according to the force-displacement relation model of the mechanical property of the transparent PDMS film (2) calibrated in the step one, and the force value is defined as a hardness quantization index of the fruit under the current deformation quantity; finally, comparing the hardness quantification index with a pre-established standard hardness database, thereby realizing classification and discrimination of the hardness level of the external tested object.
4. A method according to claim 3, wherein the visual classification module is configured to introduce a context-aware cross-scale fusion mechanism at the feature fusion stage, and to aggregate high-level semantic and low-level detail features by means of an attention mechanism to enhance discrimination of external objects under test with similar illumination variation, partial occlusion and hardness.
5. A method according to claim 3, characterized in that the three-dimensional reconstruction module is configured to receive a composite image comprising a plurality of virtual perspective information acquired by the camera (6) in cooperation with the square reflecting tube (3) and input it to a deep learning network, directly regressively outputting a three-dimensional point cloud model characterizing the deformed surface of the transparent PDMS film (2).
6. The method of claim 5, wherein the depth learning network employs a Transformer-based encoder-decoder architecture, comprising ViT encoders for extracting features from the multi-view image, and a dual-branch cross-attention decoder for fusing different view features through a cross-attention mechanism to regress to output a three-dimensional point cloud, the decoder configured to fuse feature information between different views through a cross-attention mechanism to capture geometric correspondences.
7. The method according to claim 3, wherein the visual classification module is configured to calculate a maximum deformation depth value from the processed point cloud, convert the depth value into a quantized hardness index according to a pre-calibrated force-displacement relation model characterizing the mechanical properties of the transparent PDMS film (2), and compare the hardness index with a pre-established standard hardness database, so as to perform final classification discrimination on the hardness level of the external measured object.

Description

Visual touch integrated manipulator based on sparse visual angle three-dimensional reconstruction and hardness nondestructive testing method Technical Field The invention belongs to the field of agricultural intelligence, and particularly relates to a vision-touch integrated manipulator based on sparse visual angle three-dimensional reconstruction and an analysis method. Background The maturity of fruits and vegetables is a key quality index for determining the harvest, storage and transportation and market value of fruits and vegetables. Existing detection methods are divided into two main categories, destructive detection and nondestructive maturity analysis. Although the destructive detection can accurately measure core indexes such as internal components, the sample loss is large, the cost is high, the time consumption is long, all fruits cannot be detected, and the full detection requirement of a supply chain is difficult to meet. In the nondestructive test, the subjective evaluation is strong and the efficiency is low, but the instrument based on sound, light and electricity is limited, for example, a pure vision method is easy to be interfered by illumination, fruit surface conditions and variety differences, only the correlation of maturity is evaluated instead of the fundamental index, and the acoustic and electric analysis equipment is high in price and limited in application range. Therefore, a new technology capable of directly, accurately and nondestructively quantifying the core physical characteristics such as fruit hardness and the like and realizing large-scale detection is urgently needed in the market. Disclosure of Invention Aiming at the problems, the invention aims to overcome the defects of the prior art and provide a vision-touch integrated manipulator based on sparse visual angle three-dimensional reconstruction and a fruit hardness nondestructive testing method. The invention breaks through the limitation that the traditional detection method depends on indirect related indexes such as color, smell and the like, and quantifies the elastic modulus of the fruit by directly measuring the three-dimensional deformation of the fruit under controlled pressure, so that the hardness and the maturity of the fruit are judged more fundamentally and accurately. According to the method, a camera is matched with a folding light path of a square reflecting lens cone, multi-view images of a contact area can be synchronously captured through single shooting, the external computing device further calls a deep learning network model to reconstruct three-dimensional morphology of the contact area, and accurately calculates the maximum deformation depth, and then according to the maximum deformation depth value and a mechanical characteristic model of the transparent PDMS film, quantitative hardness indexes of the external measured object are converted, and classified and judged according to the hardness grades of the object, so that accurate and reliable nondestructive detection of the fundamental physical characteristics of the external measured object is realized, and the limitation of the traditional detection method depending on apparent characteristics is thoroughly broken through. In order to achieve the aim, the invention adopts the following technical scheme that the vision-touch integrated manipulator based on sparse visual angle three-dimensional reconstruction structurally comprises a mechanical clamping jaw (1), a transparent PDMS film (2), a square reflecting lens barrel (3), a polaroid (4), an annular polarized light source (5), a camera (6) and a mechanical arm interface (7); The mechanical clamping jaw (1) is configured to perform a controlled pressing action to enable an external tested object to press the transparent PDMS film (2); the transparent PDMS film (2) is fixed on the surface of the square reflecting lens cone (3) and is used as a sensing interface for contacting and being pressed with an external tested object; the square reflecting lens barrel (3) is configured to fold a light path, so that the camera (6) can synchronously capture a direct central view angle and a plurality of virtual indirect view angle images of the deformation area of the transparent PDMS film (2) in single shooting; The annular polarized light source (5) and the camera (6) are coaxially arranged, and the polaroid sheet (4) is arranged in front of a lens of the camera (6) and is used for inhibiting specular reflection on the surface of the transparent PDMS film (2); the camera (6) is arranged at one end of the square reflecting lens barrel (3) far away from the transparent PDMS film (2), and the light path faces the transparent PDMS film (2); The camera (6) is in communication connection with an external computing device and is used for transmitting the acquired multi-view images to the external computing device for three-dimensional reconstruction and quantitative evaluation by the external computing de