Search

CN-122015845-A - Wearable intelligent navigation equipment and system for blind person traveling

CN122015845ACN 122015845 ACN122015845 ACN 122015845ACN-122015845-A

Abstract

A wearable intelligent navigation device and a wearable intelligent navigation system for the travel of the blind belong to the technical field of auxiliary navigation and positioning. The invention establishes an auditory search behavior discrimination model by analyzing the head angular velocity characteristics and the visual optical flow characteristics, recognizes and eliminates the non-travelling directional head rotation generated by the blind person due to the lateral ear hearing, realizes head-body heading decoupling, captures an environment image by using a camera, extracts the visual semantic characteristics of blind road navigation, constructs a lateral position strong constraint mechanism based on environment semantics, corrects the multipath deviation of satellite positioning and suppresses the accumulated error of inertial estimation, designs a full scene self-adaptive filter, evaluates the confidence coefficient according to the noise characteristics of the camera, an inertial measurement unit and satellite observation data, and outputs the optimal position and gesture estimation result. The invention solves the problem of course angle calculating errors caused by frequent auditory path finding actions of the head when the blind person walks by utilizing the portable advantages of the glasses form, and realizes high-precision and robust full scene navigation.

Inventors

  • LI TUAN
  • HAN BING
  • Lv Yuezu
  • GAO ZHEN

Assignees

  • 北京理工大学

Dates

Publication Date
20260512
Application Date
20260116

Claims (9)

  1. 1. A wearable intelligent navigation system for the travel of the blind is characterized by comprising the following steps: s1, data acquisition and preprocessing, namely acquiring angular velocity by utilizing intelligent glasses Acceleration of Environmental image GNSS positioning data Using an internal reference matrix Will be Feature points extracted from the medium Back projection into camera coordinate system Then the external parameter matrix is utilized Conversion to IMU coordinate system while using lever arm vector Satellite positioning data Conversion to IMU center, ensure And Consistency in physical space; S2, the blind person hearing search behavior discrimination and course decoupling, namely analyzing the variance of the yaw angular speed of the head and the average module length of the visual light flow, and identifying whether the user is in a hearing search state currently; S3, extracting visual semantic features and constructing constraints, namely performing semantic segmentation on the environment image, identifying blind road features, and constructing a lateral position strong constraint equation according to the geometric relationship between the user and the features; And S4, full scene self-adaptive fusion positioning, namely establishing an extended Kalman filter model, taking the real travelling course, the strong constraint of the lateral position and the characteristic points and positioning data in the step S1 as observation vectors, evaluating the confidence coefficient according to the noise characteristics of each observation quantity, and outputting the optimal position and posture estimation result.
  2. 2. The method of claim 1, wherein the step S1 is implemented by: For the feature points extracted from the environment image, the pixel coordinates are recorded as By means of an internal reference matrix Back projecting it to the camera coordinate system to obtain three-dimensional coordinate points in the camera coordinate system : In the formula, Is depth information of the feature points; Subsequently, the extrinsic matrix is utilized Will be Conversion to IMU coordinate system: In the formula, For the camera to IMU rotation matrix, In order to translate the vector of the vector, Three-dimensional point coordinates of the feature points in an IMU coordinate system; due to physical deviation between the glasses frame and the IMU center, the lever arm vector is utilized Converting satellite positioning data to an IMU center: In the formula, Is a rotation matrix calculated from the IMU pose, Is the lever arm vector, Is the satellite positioning coordinates of the satellite positioning data in the IMU coordinate system.
  3. 3. The method according to claim 1, wherein the variance of the yaw rate at step S2 The method comprises the following steps: In the formula, For the instantaneous yaw rate within the window, And N is the number of IMU data in the sliding window.
  4. 4. The method of claim 1, wherein the average module length of the visual light stream is determined in step S2 The method comprises the following steps: In the formula, Represents the optical flow component of the jth feature point in the horizontal direction under the image coordinate system, The optical flow component of the jth feature point in the vertical direction under the image coordinate system is represented, and K represents the total number of feature points extracted from the image.
  5. 5. The method according to claim 1, wherein step S2 is performed by identifying whether the user is currently in an auditory search state when a condition is satisfied When the user is judged to be in the auditory search state.
  6. 6. The method of claim 1, wherein the method of obtaining the true heading of the body in step S2 is that the time is when the method is not in the auditory search state The predicted attitude quaternion of the extended Kalman filter of-1 is In the normal state, the pose update relies on IMU integration: In the formula, For sampling interval time The equivalent rotation vector in the inner space is, , And finally obtaining the real advancing course of the body for the gesture quaternion of the current k moment: ; when the user is in the auditory search state, the head is obviously swung, but the environment background is not obviously translated, namely, the user is in the in-situ or low-speed exploratory path, and the observation of the yaw angle is forcedly isolated and updated at the moment, so that the real advancing course of the body is realized Locking is an estimated value at the last moment, namely: 。
  7. 7. the method of claim 1, wherein the step S3 is implemented by: environmental image using lightweight convolutional neural network Performing pixel-level semantic segmentation to identify central skeleton line of blind road region, and using camera internal reference matrix And the current pitch angle, projecting a central skeleton line on the image plane to a local navigation coordinate system, and fitting to obtain a linear equation Based on this, a lateral position observation equation is constructed: Wherein, the For the position coordinates predicted by the system at the current moment, For visual observation of noise, obeying gaussian distribution ABC is a coefficient of a fitting linear equation; the lateral position is strongly constrained when the blind spot feature is detected for consecutive multiframes The estimated position is forced to converge toward the center of the blind road by being input as strong observation information to a filter.
  8. 8. The method of claim 1, wherein the step S4 is implemented by: defining a state vector for a system : Wherein, the 、 、 Respectively representing three-dimensional position, speed and gesture quaternions, 、 Zero offset of the accelerometer and the gyroscope respectively, and k is time; The state prediction stage of the extended kalman filter uses IMU kinematics equations for time updating: In the formula, Is a rotation matrix corresponding to the quaternion, The force vector of the gravity is used to determine, For an equivalent rotation vector within the sampling interval, Is the sampling interval time; in the measurement updating stage of the extended Kalman filter, the combined observation vector of the k moment system is defined The method comprises the following steps: In the formula, Converting to three-dimensional point coordinates of the visual features under the IMU coordinate system in the step S1, For the conversion in step S1 to satellite positioning coordinates in IMU coordinate system, For the true body travel heading obtained in step S2, The lateral position distance obtained in the step S3 is obtained; Three-dimensional point coordinates for visual features Satellite positioning coordinates in IMU coordinate system True body travel course And lateral position distance , And Son observation equation of (2) The corresponding noise components are spread as follows: Characteristic point reprojection observation equation: In the formula, Is a gesture quaternion A rotation matrix is determined from the navigational coordinate system to the IMU coordinate system, Is the coordinates of the feature points in the navigational coordinate system, Extracting noise for the visual features; Satellite positioning data absolute position observation equation under IMU coordinate system: In the formula, Is a three-dimensional position component in the state vector, Is a matrix of units which is a matrix of units, Measuring noise for satellite positioning; true travel heading observation equation: In the formula, Observing noise for the heading; Lateral position strong constraint equation: Wherein A, B and C are blind road center line equation coefficients fitted in the step S3, Observing noise for lateral constraints; Using jacobian matrices Calculating Kalman gain And updates the state vector: In the measurement of the noise covariance matrix The elements are dynamically adjusted according to the signal quality, so that self-adaptive fusion positioning is realized, and the optimal position and posture estimation result is output.
  9. 9. The intelligent glasses for realizing the method according to claim 1,2,3, 4, 5, 6, 7 or 8 are characterized by comprising a glasses main body, a sensor module, a calculation processing unit and a feedback model, wherein the glasses main body comprises a glasses frame and left and right glasses legs, the sensor module is integrated in the glasses main body and comprises a camera positioned at the front part of the glasses frame and used for collecting environment image data, an inertial measurement unit positioned inside the glasses legs and used for collecting head angular velocity and acceleration data, a micro antenna and a satellite positioning module positioned above the glasses legs and used for receiving satellite signals, the calculation processing unit is embedded inside the glasses legs and is provided with a processor and a memory and used for executing the positioning method according to any one of claims 1 to 5, and the feedback model is used for feeding the positioning result output by the step S4 back to a user through Bluetooth audio signals.

Description

Wearable intelligent navigation equipment and system for blind person traveling Technical Field The invention belongs to the technical field of auxiliary navigation and positioning, and relates to a blind person full-scene self-adaptive positioning method and equipment based on intelligent glasses multi-sensor fusion, which are designed aiming at walking characteristics of visually impaired people. Background With the development of wearable technology, the use of Inertial Measurement Units (IMUs) and Global Navigation Satellite Systems (GNSS) for personal positioning has become a mainstream solution for assisting the blind in traveling. Conventional auxiliary devices mostly adopt a hand-held type or a head-mounted type (such as a helmet), but in practical application, a plurality of technical bottlenecks are still faced. During walking, the blind person frequently swings his head left and right or listens to his side ear in order to sense the surrounding environment through hearing. Particularly when wearing portable intelligent glasses, the head can rotate more flexibly. The existing navigation algorithm usually defaults to the head direction, namely the body traveling direction, which can cause the error of the positioning track, and can not truly reflect the straight traveling intention of the blind. Secondly, GNSS positioning is easy to generate multipath effect in urban high-rise areas, so that the position is randomly drifted by a plurality of meters, and the position error of the IMU is rapidly accumulated with time although the accuracy of the IMU is high in short time. The blind person has extremely high requirements on position accuracy, and once GNSS drift causes the positioning point to deviate to the motor vehicle lane, serious safety accidents can be caused. In addition, most of the existing pedestrian dead reckoning algorithms are blind pushing, do not have the sensing capability on the surrounding environment, and cannot calibrate the errors of the sensor by utilizing the environment characteristics. Therefore, a multi-sensor fusion method that can adapt to the wearing form of smart glasses, effectively decouple head-body movements, and accurately position using the visual semantics of the environment is needed. Disclosure of Invention The invention aims to provide wearable intelligent navigation equipment and system for the travel of the blind person, which can decouple head-body movement, accurately position by utilizing environment visual semantics and ensure continuous accuracy of navigation heading in a complex sound field environment. In order to achieve the above purpose, the present invention provides the following technical solutions: The invention discloses a wearable intelligent navigation system for the travel of the blind, which comprises the following steps: s1, data acquisition and preprocessing, namely acquiring angular velocity by utilizing intelligent glasses Acceleration ofEnvironmental imageGNSS positioning data. Using an internal reference matrixWill beFeature points extracted from the mediumBack projection into camera coordinate systemThen the external parameter matrix is utilizedConversion to IMU coordinate system while using lever arm vectorSatellite positioning dataConversion to IMU center, ensureAndConsistency in physical space. And S2, judging and decoupling the auditory search behavior of the blind person from the course, namely analyzing the variance of the yaw rate of the head and the average modular length of the visual optical flow, and identifying whether the user is in an auditory search state currently. If not, returning to search behavior discrimination, and if so, decoupling the locking relation between the head orientation and the body advancing direction, and acquiring the real advancing heading of the body. S3, extracting visual semantic features and constructing constraints, namely performing semantic segmentation on the environment image, identifying blind road features, and constructing a lateral position strong constraint equation according to the geometric relationship between the user and the features; And S4, full scene self-adaptive fusion positioning, namely establishing an extended Kalman filter model, taking the real travelling course, the strong constraint of the lateral position and the characteristic points and positioning data in the step S1 as observation vectors, evaluating the confidence coefficient according to the noise characteristics of each observation quantity, and outputting the optimal position and posture estimation result. Further, the implementation method of the step S1 is as follows: For the feature points extracted from the environment image, the pixel coordinates are recorded as By means of an internal reference matrixBack projecting it to the camera coordinate system to obtain three-dimensional coordinate points in the camera coordinate system: In the formula,Is depth information of the feature points; Subsequently, t