Search

CN-121987467-A - Human body action blind guiding system based on exoskeleton

CN121987467ACN 121987467 ACN121987467 ACN 121987467ACN-121987467-A

Abstract

The application provides a human body action blind guiding system based on an exoskeleton, which comprises a user state sensing module, an executor module, a sensing fusion module, an interaction and decision module, an algorithm module and a target path determining module, wherein the user state sensing module is used for acquiring limb movement data of a user, the executor module is used for driving the exoskeleton, the sensing fusion module is configured to acquire environment information and establish a three-dimensional point cloud map, identify various objects in the three-dimensional point cloud map, mark object semantic tags to corresponding object positions in the three-dimensional point cloud map to obtain a semantic map, the interaction and decision module is configured to determine three-dimensional information of a target object, the algorithm module is configured to determine an initial transformation matrix between a camera coordinate system and the exoskeleton coordinate system, determine position data of the user under the semantic map based on the initial transformation matrix, generate a plurality of planning paths, and drive the exoskeleton by utilizing the executor module to guide the user to reach the target object positions and execute corresponding actions of voice instructions so as to solve the problems of information dimension loss and interaction inefficiency of the exoskeleton at present.

Inventors

  • XIA DAN
  • XU JIE

Assignees

  • 天津瞰见科技有限公司

Dates

Publication Date
20260508
Application Date
20260119

Claims (10)

  1. 1. An exoskeleton-based human motion blind guiding system, comprising: The system comprises a perception fusion module (1), a user state perception module (2), an interaction and decision module (3), an actuator module (4) and an algorithm module (5), wherein the user state perception module (2) is integrated on an exoskeleton, the user state perception module (2) is used for acquiring limb movement data of a user, and the limb movement data comprise exoskeleton joint angles and exoskeleton tail end position coordinates; the perceptual fusion module (1) is configured to: Collecting environment information and establishing a three-dimensional point cloud map, wherein the environment information comprises texture, semantics and space information; Identifying various objects in the three-dimensional point cloud map by utilizing a semantic segmentation neural network, and marking semantic tags of the objects to corresponding object positions in the three-dimensional point cloud map to obtain a semantic map; the interaction and decision module (3) is configured to: receiving a voice instruction of a user and determining three-dimensional information of a target object under the semantic map, wherein the three-dimensional information comprises shape, azimuth and volume information; The algorithm module (5) is configured to: Determining an initial transformation matrix between a camera coordinate system and an exoskeleton coordinate system based on the limb movement data, wherein the camera coordinate system is determined by the perception fusion module (1), and the exoskeleton coordinate system is determined by the exoskeleton; Determining position data of a user under the semantic map based on the initial transformation matrix; generating a plurality of planned paths based on the three-dimensional information and the position data; Determining a target path based on the planned path; And driving the exoskeleton by using the actuator module (4) according to the target path, guiding a user to reach the target object position and executing the action corresponding to the voice instruction.
  2. 2. The exoskeleton-based human action blind guiding system of claim 1, wherein the interaction and decision module (3) is further configured to: When the exoskeleton is started, a first set voice is sent out, wherein the first set voice is used for guiding a user to execute a first predefined action; The perceptual fusion module (1) is further configured to: collecting limb moving images of a user; the algorithm module (5) is further configured to: And judging whether the user completes the first predefined action based on the limb moving image, if so, determining the initial position of the exoskeleton and executing the step of determining an initial transformation matrix between a camera coordinate system and the exoskeleton coordinate system.
  3. 3. The exoskeleton-based human motion blind guide system of claim 1, wherein the algorithm module (5) is further configured to: The interaction and decision module (3) is used for sending out second set voice and the perception fusion module (1) is used for collecting first Marker point coordinates under a camera coordinate system, wherein the second set voice is used for guiding a user to execute second predefined actions; determining a second Marker point coordinate under an exoskeleton coordinate system based on the limb movement data; and determining an initial transformation matrix based on the first Marker point coordinates and the second Marker point coordinates.
  4. 4. A human motion blind guiding system based on exoskeleton of claim 3, wherein said algorithm module (5) is further configured to: And performing extended Kalman filtering fusion on the first Marker point coordinate and the limb movement data, and correcting the initial transformation matrix.
  5. 5. The exoskeleton-based human action blind guiding system of claim 1, wherein the interaction and decision module (3) is further configured to: converting the voice instruction into a text and analyzing a target object; And judging whether the semantic map has the target object, if so, determining three-dimensional information of the target object under the semantic map and sending a confirmation voice instruction, wherein the confirmation voice instruction is used for informing a user that the position of the target object is determined and whether the voice instruction is executed or not.
  6. 6. A human motion blind guiding system based on exoskeleton of claim 3, wherein said algorithm module (5) is further configured to: Determining position coordinates and orientations of the trunk of the user under the semantic map based on the initial transformation matrix and the limb movement data; And determining the position coordinates and the gestures of the tail end of the hand of the user under an actuator coordinate system and the semantic map by utilizing a filtering algorithm based on the initial transformation matrix, the first Marker point coordinates and the limb motion data, wherein the actuator coordinate system is determined by the actuator module (4).
  7. 7. The exoskeleton-based human motion blind guide system of claim 1, wherein the algorithm module (5) is further configured to: judging whether an obstacle exists in the planned path; if yes, the corresponding path is removed from the planning path, and a target path is determined by using inverse kinematics, wherein the path length of the target path is minimum.
  8. 8. The exoskeleton-based human motion blind guide system of claim 1, wherein the algorithm module (5) is further configured to: acquiring an actual motion trail of a limb of a user by using the perception fusion module (1); Determining a desired motion trail of a user limb based on the target path; calculating the position and posture errors of the limbs of the user based on the actual motion trail and the expected motion trail; Based on the position and posture errors, a guiding torque is calculated and the exoskeleton is driven with the guiding torque using an actuator module (4).
  9. 9. The exoskeleton-based human motion blind guide system of claim 1, wherein the actuator module (4) is further configured to: when a user approaches the fragile product, driving the tail end of the exoskeleton to shake at a preset frequency; when a user approaches a dangerous article, driving the exoskeleton by a reverse moment in the opposite direction of the dangerous article; wherein, when the distance between the user and the dangerous goods is shorter, the reverse moment is larger.
  10. 10. The exoskeleton-based human motion blind guide system of claim 1, wherein the actuator module (4) is further configured to: And stopping driving the exoskeleton when the reverse moment is greater than a safety threshold in the process of executing the voice command by the exoskeleton.

Description

Human body action blind guiding system based on exoskeleton Technical Field The application relates to the technical field of computer vision and artificial intelligence, in particular to a human body action blind guiding system based on exoskeleton. Background In a daily life scenario, the blind population faces a number of difficulties in terms of movement and interaction. The auxiliary technology aiming at the blind is focused on navigation and obstacle avoidance on a macroscopic level, such as an electronic blind guiding stick, a blind guiding robot, voice navigation application based on a GPS and a smart phone and the like, and mainly solves the problem of navigation and obstacle avoidance from point A to point B. However, when the blind arrives at a specific location, fine interaction is performed with the surrounding environment, such as accurately taking a specified commodity on a supermarket shelf, pressing a correct elevator button in a public place, starting a cup at the upper end of a dining table, sweeping a code for payment, swiping a card, inserting coins, and the like, the tasks require space positioning and motion control capability in the millimeter to centimeter level, and the blind guiding technology cannot meet the fine interaction requirement. There are some attempts to address the above-mentioned needs. For example, an object recognition and voice broadcasting system based on a smart phone camera (or AI glasses, etc.) and a large AI model, the system acquires environmental information through the camera, and informs a user of the position of a target object in a voice form, such as 'a water cup is 30 cm in front of you' after recognizing the object by using the large AI model. Or an object grabbing auxiliary method based on voice prompt firstly captures an environment image through a camera worn on a user, then utilizes a deep learning model (such as YOLO and SSD) to identify a target object appointed by voice of the user, then estimates rough azimuth and distance of the target object relative to the user through a stereoscopic vision or depth camera, and finally guides the user to move arms through voice (such as please move hands to the right for 10 cm and then stretch forwards for 15 cm) or simple audio prompt (such as left and right ear sound size). However, the problems of missing information dimension and low interaction efficiency exist in the technical scheme that voice or audio prompt is one-dimensional or two-dimensional information, grabbing action is three-dimensional space continuous motion, a user needs to convert discrete voice instructions into continuous space motion in the brain, the process is difficult and counterintuitive, interaction is slow and hard, and a large number of repeated heuristics are needed. And after the system sends out the instruction, whether the actual motion trail of the arm of the user accords with the expectation cannot be perceived in real time, once the user understands deviation or execution error, the system cannot correct in time, task failure is easy to cause, and even danger is caused by false touch. Disclosure of Invention The application provides a human motion blind guiding system based on an exoskeleton, which aims to solve the technical problems of information dimension loss and low interaction efficiency of the exoskeleton at present. The application provides a human body action blind guiding system based on exoskeleton, which comprises: The system comprises a sensing fusion module, a user state sensing module, an interaction and decision module, an actuator module and an algorithm module which are in communication connection, wherein the user state sensing module is integrated on an exoskeleton, and the user state sensing module is used for acquiring limb movement data of a user, wherein the limb movement data comprise exoskeleton joint angles and exoskeleton tail end position coordinates; The perceptual fusion module is configured to: Collecting environment information and establishing a three-dimensional point cloud map, wherein the environment information comprises texture, semantics and space information; Identifying various objects in the three-dimensional point cloud map by utilizing a semantic segmentation neural network, and marking semantic tags of the objects to corresponding object positions in the three-dimensional point cloud map to obtain a semantic map; the interaction and decision module is configured to: receiving a voice instruction of a user and determining three-dimensional information of a target object under the semantic map, wherein the three-dimensional information comprises shape, azimuth and volume information; The algorithm module is configured to: determining an initial transformation matrix between a camera coordinate system and an exoskeleton coordinate system based on the limb movement data, wherein the camera coordinate system is determined by the perception fusion module; Determining position dat