CN-121680646-B - Intelligent obstacle avoidance and gesture control method and system integrating machine vision and ultrasonic waves

CN121680646BCN 121680646 BCN121680646 BCN 121680646BCN-121680646-B

Abstract

The invention relates to the field of obstacle avoidance and gesture control, and particularly provides an intelligent obstacle avoidance and gesture control method and system integrating machine vision and ultrasonic waves. The invention realizes cooperative obstacle avoidance of fusion of the machine vision sensor and the ultrasonic sensor, expands the detection range and improves the environmental adaptability, and improves the gesture recognition accuracy and the response speed through fusion of the multi-mode sensor. The invention finally forms a new generation of sensing and interaction system with high reliability, high accuracy, low delay and intelligent self-adaption through the fusion effect of the machine vision and the ultrasonic sensor.

Inventors

LIU CHUN
TAN XUEMIN
LIU BINCHENG
PU HUILAN
QIAN SIWEI
LIANG TINGTING

Assignees

成都信息工程大学

Dates

Publication Date: 20260512
Application Date: 20260209

Claims (8)

1. The intelligent obstacle avoidance and gesture control method integrating machine vision and ultrasonic waves is characterized by comprising the following steps of: step 1, data acquisition, namely respectively acquiring machine vision original data, ultrasonic original echo data and environmental temperature data through a machine vision sensor, an ultrasonic sensor and a temperature sensor; step 2, preprocessing and space-time synchronization: The method comprises the steps of preprocessing data, namely filtering machine vision original data, illumination compensation and distortion correction to obtain preprocessed vision data, digitally filtering ultrasonic original echo data, and calibrating a ranging value by combining with ambient temperature data to obtain preprocessed ultrasonic data; Time synchronization, namely aligning the time stamp of the preprocessed visual data and the time stamp of the preprocessed ultrasonic data to obtain time-synchronized bimodal data; Space registration, namely taking time synchronized bimodal data as an object, and optimizing space coordinate conversion parameters through a calibration algorithm to obtain space-time synchronized bimodal data; step 3, multimode fusion: Obstacle avoidance fusion, namely calculating visual confidence coefficient and ultrasonic confidence coefficient based on the bimodal data after space-time synchronization, and outputting three-dimensional perception results of the obstacle through dynamic weighting fusion weights according to the relative difference of the two types of confidence coefficients, wherein the visual confidence coefficient is calculated based on image definition scoring and the quantity and distribution uniformity of feature points; The gesture fusion comprises the steps of carrying out data registration before fusion on bimodal data after time-space synchronization, extracting visual semantic features and ultrasonic motion features, and fusing the visual semantic features and the ultrasonic motion features to output gesture recognition results, wherein the visual semantic features are extracted through a pretrained convolutional neural network, the ultrasonic motion features comprise instantaneous distances, distance change rates and accelerations, and the fusion adopts an attention model or feature splicing mode; And step 4, outputting a decision, namely generating obstacle avoidance action instructions based on the three-dimensional perception result of the obstacle, and generating equipment control instructions based on the gesture recognition result.
2. The intelligent obstacle avoidance and gesture control method integrating machine vision and ultrasonic waves according to claim 1 is characterized by further comprising the step 5 of executing an obstacle avoidance action instruction and a device control instruction, collecting environment feedback data after the execution of the instruction, and returning the environment feedback data to the step 1 as the supplement of new machine vision original data and ultrasonic wave original echo data.
3. The intelligent obstacle avoidance and gesture control method integrating machine vision and ultrasonic waves according to claim 2, further comprising step 6, system stability assurance, namely periodically calibrating a clock to maintain time precision of the bimodal data after space-time synchronization, monitoring sensor states to ensure effective acquisition of machine vision raw data and ultrasonic wave raw echo data, and re-executing spatial registration to ensure spatial precision of the bimodal data after space-time synchronization when environmental temperature data changes by over-threshold.
4. The intelligent obstacle avoidance and gesture control method integrating machine vision and ultrasonic waves according to claim 1 is characterized in that in step 1, the machine vision original data comprise RGB images and depth images or common RGB images, and the machine vision sensor is a depth camera or a photosensitive module matched with a monocular depth estimation algorithm.
5. The intelligent obstacle avoidance and gesture control method integrating machine vision and ultrasonic waves according to claim 1, wherein in step 1, the original echo data of the ultrasonic waves comprise distance data, echo intensity data and echo signal to noise ratio data, and the ultrasonic wave sensor supports external triggering, echo signal output and temperature compensation functions.
6. The intelligent obstacle avoidance and gesture control method that fuses machine vision and ultrasound according to claim 1, wherein in step 2, the distortion correction of the data preprocessing is performed based on an internal reference matrix and a distortion coefficient of the machine vision sensor, which are obtained through a camera calibration experiment.
7. The intelligent obstacle avoidance and gesture control method integrating machine vision and ultrasonic waves according to claim 1, wherein in the step 2, the time synchronization adopts hardware-level synchronization or software time synchronization, wherein the hardware-level synchronization generates a synchronization pulse through a microcontroller to trigger a sensor and record a reference time stamp, and the software time synchronization generates time stamp matching data through a precision clock protocol.
8. An intelligent obstacle avoidance and gesture control system integrating machine vision and ultrasonic waves, which is characterized in that the intelligent obstacle avoidance and gesture control system integrating machine vision and ultrasonic waves comprises: The data acquisition module is used for acquiring machine vision original data, ultrasonic original echo data and environmental temperature data; a preprocessing and space-time synchronization module for: The method comprises the steps of preprocessing data, namely filtering machine vision original data, illumination compensation and distortion correction to obtain preprocessed vision data, digitally filtering ultrasonic original echo data, and calibrating a ranging value by combining with ambient temperature data to obtain preprocessed ultrasonic data; Time synchronization, namely aligning the time stamp of the preprocessed visual data and the time stamp of the preprocessed ultrasonic data to obtain time-synchronized bimodal data; Space registration, namely taking time synchronized bimodal data as an object, and optimizing space coordinate conversion parameters through a calibration algorithm to obtain space-time synchronized bimodal data; a multi-modal fusion module for: The obstacle avoidance fusion comprises the steps of calculating visual and ultrasonic confidence coefficient based on the space-time synchronized bimodal data, and outputting an obstacle three-dimensional sensing result through dynamic weighting fusion weight, wherein the visual confidence coefficient is calculated based on image definition scoring and feature point quantity and distribution uniformity, the ultrasonic confidence coefficient is calculated based on echo signal-to-noise ratio and echo signal peak voltage, and the dynamic weighting fusion weight is calculated according to the relative difference of two types of confidence coefficient through a Softmax function; The gesture fusion comprises the steps of carrying out data registration before fusion on bimodal data after time-space synchronization, extracting visual semantic features and ultrasonic motion features, and fusing the visual semantic features and the ultrasonic motion features to output gesture recognition results, wherein the visual semantic features are extracted through a pretrained convolutional neural network, the ultrasonic motion features comprise instantaneous distances, distance change rates and accelerations, and the fusion adopts an attention model or feature splicing mode; The decision output module is used for generating obstacle avoidance action instructions based on the three-dimensional perception result of the obstacle; The instruction execution module is used for executing the obstacle avoidance action instruction and the equipment control instruction; The environment feedback module is used for collecting environment feedback data after the instruction execution and returning the environment feedback data as the supplement of the new round of machine vision original data and the ultrasonic original echo data.

Description

Intelligent obstacle avoidance and gesture control method and system integrating machine vision and ultrasonic waves Technical Field The invention relates to the technical field of obstacle avoidance and gesture control, in particular to an intelligent obstacle avoidance and gesture control method and system integrating machine vision and ultrasonic waves. Background Gesture control technology is an important way of man-machine interaction, and is widely applied in the fields of consumer electronics, smart home and robots in recent years. The current mainstream gesture recognition schemes include: ① And based on vision recognition, acquiring a gesture image through a camera, and then performing recognition by utilizing image processing and a machine learning algorithm. ② Gesture recognition technology based on depth sensors, common structured light technology and time of flight (ToF) technology. ③ Gesture recognition technology based on inertial sensors generally uses inertial sensors such as accelerometers and gyroscopes to detect the motion state and direction of a hand, thereby recognizing gestures. ④ The millimeter wave radar technology developed by the ProjectSoli team of Google can recognize fine gestures with high precision, such as pinching, sliding and the like, and is applied to Google Pixel series mobile phones. ⑤ The multi-mode fusion gesture recognition technology combines the advantages of various sensors, such as combining a visual sensor with an inertial sensor, a millimeter wave radar and the like, integrates data through an algorithm, improves the accuracy and the robustness of gesture recognition, and is widely applied to scenes such as intelligent cabins. The intelligent obstacle avoidance technology mainly comprises the following steps of vision obstacle avoidance, laser radar obstacle avoidance, ultrasonic obstacle avoidance and infrared obstacle avoidance. Disadvantages of the prior art: Gesture recognition, namely, vision-based recognition is greatly influenced by environment and shielding, privacy and calculation cost are problems, technology based on a depth sensor is high in cost, limited in recognition distance and large in size, is easy to interfere with the environment, technology based on an inertial sensor is provided with accumulated errors, depends on wearing positions, is narrow in recognition range and high in power consumption, technology based on millimeter wave radar is high in cost, recognition accuracy is influenced by distance angles, is easy to interfere with electromagnetic waves and is limited in complex gesture recognition capability, and multi-mode fusion technology is complex in system, high in cost, high in data fusion difficulty and also has power consumption and size problems. Obstacle avoidance function-the obstacle avoidance scheme of a single sensor has insufficient reliability in a complex environment. The visual obstacle avoidance is greatly influenced by illumination, transparent reflecting objects and complex backgrounds, and depends on efficient algorithms and calculation forces, the laser radar obstacle avoidance has high cost, performance is reduced in severe weather, reflection on nonmetallic objects is weak, the ultrasonic obstacle avoidance detection distance is short, noise interference is easy to occur, an angle blind area exists, the resolution is low, the infrared obstacle avoidance detection distance is short, the range is narrow, the interference of ambient light is easy to occur, and the recognition capability on dark colors and transparent objects is weak. The control mode is single, and the control mode is dependent on a remote controller or a mobile phone APP and lacks natural interaction (such as gesture control) The real-time performance is poor, and the sensor data fusion and decision algorithm is not optimized, so that the control delay is obvious (for example, more than 1 second is needed for gesture → execution). The prior art is blank that most of the current gesture control and intelligent obstacle avoidance are independent systems (such as gesture control depends on vision/millimeter wave, obstacle avoidance depends on laser radar/infrared), and the global obstacle avoidance and high robust gesture interaction are not realized through a low-cost fusion scheme of machine vision and ultrasonic at the same time, so that the integrated requirements of autonomous obstacle avoidance and convenient control under multiple scenes are difficult to meet due to low system integration, high cost or weak environmental adaptability. Disclosure of Invention The invention provides an intelligent obstacle avoidance and gesture control method and system integrating machine vision and ultrasonic waves, which aims to solve the problems: (1) How to realize the cooperative obstacle avoidance of the fusion of the machine vision sensor and the ultrasonic sensor, expand the detection range and improve the environmental adaptability; (2) How to improve