Search

US-12619254-B2 - Unmanned platform with bionic visual multi-source information and intelligent perception

US12619254B2US 12619254 B2US12619254 B2US 12619254B2US-12619254-B2

Abstract

Disclosed is an unmanned platform with bionic visual multi-source information and intelligent perception. The unmanned platform is equipped with a bionic polarization vision/inertia/laser radar combined navigation module, a deep learning object detection module and an autonomous obstacle avoidance module; the bionic polarization vision/inertia/laser radar combined navigation module is configured to position and orient the unmanned platform in real time; the deep learning object detection module is configured to sense an environment around the unmanned platform according to RGB images of a surrounding environment collected by the bionic polarization vision/inertia/laser radar combined navigation module; and the autonomous obstacle avoidance module determines whether there are any obstacles around the unmanned platform during running according to the objects identified by the target, and performs autonomous obstacle avoidance in combination with the carrier navigation and positioning information. Concealment, autonomous navigation, object detection and autonomous obstacle avoidance capabilities of the unmanned platform are thus improved.

Inventors

  • Xianghong CHENG
  • Fengyu Liu
  • Zhiwei Zhong

Assignees

  • SOUTHEAST UNIVERSITY

Dates

Publication Date
20260505
Application Date
20231219
Priority Date
20221124

Claims (8)

  1. 1 . An unmanned platform with bionic visual multi-source information and intelligent perception, comprising a bionic polarization vision/inertia/laser radar combined navigation module, a deep learning object detection module and an autonomous obstacle avoidance module, wherein the bionic polarization vision/inertia/laser radar combined navigation module contains two application scenarios: an outdoor environment and an unknown enclosed environment, wherein the bionic polarization vision/inertia/laser radar combined navigation module comprises a bionic polarization camera, a laser radar sensor, an inertial sensor and a monocular vision camera sensor, wherein the bionic polarization vision/inertia/laser radar combined navigation module is configured to position and orient the unmanned platform in real time; the deep learning object detection module is configured to sense an environment around the unmanned platform according to RGB images of a surrounding environment collected by the bionic polarization vision/inertia/laser radar combined navigation module; the autonomous obstacle avoidance module determines whether there are any obstacles around the unmanned platform during running according to objects identified by a target, and performs autonomous obstacle avoidance in combination with carrier navigation and positioning information obtained by the bionic polarization vision/inertia/laser radar combined navigation module; the bionic polarization camera is configured to sense an atmospheric polarization state, and solve course angle information; the laser radar sensor is configured to obtain point cloud information of a surrounding environment and construct a dense map; the inertial sensor is configured to update positions and orientations of the unmanned platform; the monocular vision camera sensor is configured to perform object detection of the surrounding environment; and the bionic polarization camera and the inertial sensor are used in the outdoor environment for the navigation and orientation precision of the unmanned platform; when the bionic polarization camera fails to maintain polarization-based orientation tracking in the unknown enclosed environment, the laser radar sensor and the inertial sensor are used together for the navigation, positioning and mapping capabilities of the unmanned platform.
  2. 2 . The unmanned platform with bionic visual multi-source information and intelligent perception according to claim 1 , wherein the bionic polarization vision/inertia/laser radar combined navigation module is used in the outdoor environment for the navigation and orientation precision of the unmanned platform, comprising the following working steps: ( 101 ) using the bionic polarization camera to acquire a sky polarization pattern, and solving to obtain course angle information of the unmanned platform in a horizontal state; and using the inertial sensor to measure a roll angle, a pitch angle and a course angle of the unmanned platform, performing inclination angle compensation for the course angle in the horizontal state to obtain course angle information of the unmanned platform in a non-horizontal state; and ( 102 ) fusing the course angle information of the unmanned platform in the horizontal state and the course angle information of the unmanned platform in the non-horizontal state obtained in the step ( 101 ) by using a cubature Kalman filter algorithm assisted by a radial basis function neural network, and outputting an optimal course angle information.
  3. 3 . The unmanned platform with bionic visual multi-source information and intelligent perception according to claim 2 , wherein the step ( 101 ) has the following specific method: since the course angle of the carrier in a geographic coordinate system in the horizontal state is φ=α s −α b , wherein α b and α s are azimuth angles of the solar meridian in a carrier coordinate system and the geographic coordinate system; under the condition that the local longitude and latitude information of the carrier and accurate time information at the time of observation are known, a solar elevation angle h s and a solar azimuth angle α s in the geographic coordinate system are respectively as follows: sin ⁡ ( h s ) = sin ⁢ δ ⁢ sin ⁢ β + cos ⁢ δ ⁢ cos ⁢ β ⁢ cos ⁢ t s cos ⁡ ( α s ) = sin ⁡ ( h s ) ⁢ sin ⁢ β - sin ⁢ δ cos ⁡ ( h s ) ⁢ cos ⁢ β in the formula, δ, β and t s represent a declination angle of the sun, the latitude of an observation point and an hour angle of the sun respectively; then, three-channel polarization images are collected through the bionic polarization camera carried by the unmanned platform, feature point sets are searched and identified based on all-sky-domain polarization data by using an E-vector in a direction of the solar meridian is perpendicular to a scattering plane, the feature points are subjected to a linear fitting once to solve α b , and finally the course angle φ of the carrier in the geographic coordinate system in the horizontal state can be solved; when the unmanned platform is in the non-horizontal state, an E-vector direction will not be perpendicular to the solar meridian, but perpendicular to a solar vector, shown in the formula as below: e p n · a OS n = 0 in the formula, n represents a navigation coordinate system, b represents the carrier coordinate system, a OS n represents a sun vector in the navigation coordinate system, and e p n represents the E vector in the navigation coordinate system; the E-vector direction in the carrier coordinate system is defined as e p b = [ cos ⁢ α ⁢ sin ⁢ α ⁢ 0 ] T , , wherein α represents a clockwise angle between a reference direction of the bionic polarization camera and an E-vector direction of incident light, and C b n represents a direction cosine matrix from the carrier coordinate system to the navigation coordinate system, with e p n = C b n ⁢ e p b ; then in the navigation coordinate system, the sun vector a OS n can be calculated by the solar azimuth angle α s and the solar elevation angle h s , as shown in the following formula: a OS n = [ a OSx n a OSy n a OSz n ] = [ cos ⁢ ( h s ) ⁢ cos ⁢ ( α s ) cos ⁢ ( h s ) ⁢ sin ⁢ ( α s ) sin ⁢ ( h s ) ] in the formula, a OS x n , a OS y n , a OS z n represent projection of the solar vector in the navigation system; combined the above formula, the course angle Ψ after inclination angle compensation in the non-horizontal state can be calculated: Ψ=arcsin c+arctan d, wherein c and d are intermediate parameters of the solution, and c = cos ⁢ α ⁢ sin ⁢ θ - sin ⁢ α ⁢ cos ⁢ θ ⁢ sin ⁢ ϕ c ⁢ tan ⁢ h s ⁢ ( sin ⁢ αsinϕsinθ + cos ⁢ αcosθ ) 2 + cos 2 ⁢ ϕsin 2 ⁢ α d = cos ⁢ α ⁢ cos ⁢ θ ⁢ cos ⁢ α s + sin ⁢ αsinϕsinθcosα s + cos ⁢ ϕsinαsinα s cos ⁢ α ⁢ cos ⁢ θ ⁢ sin ⁢ α s + sin ⁢ αsinϕsinθsinα s - cos ⁢ ϕsinαcosα s in the formula, φ and θ represent the roll angle and the pitch angle of the carrier in a non-horizontal state, which are obtained by the inertial sensor through solution.
  4. 4 . The unmanned platform with bionic visual multi-source information and intelligent perception according to claim 2 , wherein the step ( 102 ) has the following specific method: first, the inertial sensor realizes initial alignment through the course angle information provided by the bionic polarization camera, a bionic polarization vision/inertial combination model is then established, the course angle information output by the inertial system is taken as the quantity of state, the course angle information obtained by the bionic polarization camera through the inclination angle compensation is used as a quantity measurement, and the cubature Kalman filter algorithm is adopted to conduct the data fusion; a state model in a discrete form is expressed as: x k = f ⁡ ( x k - 1 , u k - 1 ) + w k - 1 z k = h ⁡ ( x k ) + v k in the formula, x k and X k−1 represent the quantity of state at moments k and k−1 respectively; u k−1 is a filter input; z k is the quantity measurement; f( ) and h( ) are a state transition equation and a measurement equation respectively; and w k−1 and v k are uncorrelated zero-mean Gaussian process noise and measurement noise respectively; a radial basis function neural network is introduced into a cubature Kalman filter framework to perform error compensation on a filter: first, a window size h is set, which reflects that the cubature Kalman filter algorithm iteratively estimates the optimal course angle information for h moments within one window size, the quantity of state within a window, and innovations of the cubature Kalman filter and the optimal estimated course angle information are input to the radial basis function neural network for training; when data of the next moment come, the quantity of state, as well as gains and innovations of the cubature Kalman filter at that moment are input to the radial basis function neural network trained in the previous window, and the input and the output are as follows: x r = [ K r I r p r v r q r z r ] T y r = b b ⁢ i ⁢ a ⁢ s + ∑ We -  x r - μ  2 2 ⁢ σ 2 ⁢ r = 1 , 2 , … , h in the formula, r represents a certain moment when the r th data within the window come, K r and I r represent gains and innovations of the cubature Kalman filter within the window respectively, p r , v r and q r represent the quantities of state of positions, velocities and attitudes of the inertial sensor within the window, z r represents the quantity measurement collected by the bionic polarization camera within the window, y r represents compensation output of the neural network, b bias represents bias matrix of network nodes, and W represents a weight matrix of the network nodes; and e -  x r - μ  2 2 ⁢ σ 2 represents a Gaussian kernel, wherein μ and σ represent a center and a width of the Gaussian kernel function respectively, and wherein ∥xr−μ∥ represents an Euclidean distance between a state quantity xr and a center μ of a Gaussian kernel function; and estimated values of the cubature Kalman filter algorithm at the current moment are then compensated and corrected through error values output by the radial basis function neural network, data of the next window size h, starting from the current correction moment, are used to continue training the radial basis function neural network, when data of the next moment after the end of a second window come, the estimated values of the cubature Kalman filter algorithm are compensated and corrected again through error values output by the retrained radial basis function neural network, and iteration is performed in such a way, so that the cubature Kalman filter can be optimized by the radial basis function neural network.
  5. 5 . The unmanned platform with bionic visual multi-source information and intelligent perception according to claim 1 , wherein the bionic polarization vision/inertia/laser radar combined navigation module is used in the unknown enclosed environment to perform navigation, positioning, and mapping of the unmanned platform, comprising the following working steps: ( 201 ) pre-integrating the inertial sensor, and performing discretization to obtain a relationship among positions, velocities and attitudes at two quantity measurement values of the inertial sensor by using a median method; ( 202 ) designing a residual term through the pre-integration of the inertia sensor in the step ( 201 ) and calculating a Jacobian of the residual term relative to variables to be optimized to obtain pre-integration relative constraints, fusing the pre-integration relative constraints with relative pose constraints of the laser radar, constructing a nonlinear optimization model, constructing a laser/inertia odometry, and solving the optimal pose; and ( 203 ) mapping the unknown enclosed environment according to landmark points obtained by the laser/inertia odometry in the step ( 202 ).
  6. 6 . The unmanned platform with bionic visual multi-source information and intelligent perception according to claim 5 , wherein the step ( 201 ) has the following specific method: in the step ( 201 ), performing navigation according to the measurement information of the inertial sensor and the laser radar when the unmanned platform enters the unknown enclosed environment and the bionic polarization camera suffers a failure; first, pre-integration is carried out on the inertial sensor, and a continuous expression of a pre-integration method is as follows: p b k + 1 w = p b k w + v b k w ⁢ Δ ⁢ t + ∫ ∫ t ∈ [ k , k + 1 ] ( q b t w ⁢ a b t - b a t - n a - g w ) ⁢ d ⁢ t 2 v b k + 1 w = v b k w + ∫ t ∈ [ k , k + 1 ] ( q b t w ⁢ a b t - b a t - n a - g w ) ⁢ dt q b k + 1 w = q b k w ⊗ ∫ t ∈ [ k , k + 1 ] 1 2 ⁢ q b t b k ⊗ ( ω b t - b ω t - n ω ) ⁢ d ⁢ t in the formula, subscript b represents the carrier coordinate system, wis a world coordinate system, a time interval between two frames is Δt, p b k + 1 w , p b k w represent positions at moments k+1 and k in the world coordinate system respectively, v b k + 1 w , v b k w represent velocities at the moments k+1 and k in the world coordinate system respectively, q b k + 1 w , q b k w represent attitudes at the moments k+1 and k in the world coordinate system respectively, a b t and ω b t represent outputs of an accelerometer and a gyroscope at a moment t, b a t , b ω t represent zero bias of the accelerometer and the gyroscope at the moment t, n a and n ω represent random noise of the accelerometer and the gyroscope, and g w represents a gravity acceleration in the world coordinate system, the median method is adopted to perform discretization, and the relationship of outputs between the i+1 th and the i th inertial sensors is as follows: p b i + 1 w = p b i w + v b i w ⁢ δ ⁢ t + 1 2 ⁢ a ¯ ⁢ δ ⁢ t 2 v b i + 1 w = v b i w + a ¯ ⁢ δ ⁢ t q b i + 1 w = q b i w ⊗ [ 1 1 2 ⁢ ω ¯ ⁢ δ ⁢ t ] in the formula, p b i + 1 w , p b i w represent positions at an output moment of the i+1 th and the i th inertial sensors after discretization in the world coordinate system respectively, v b i + 1 w , v b i w represent velocities at the output moment of the i+1 th and the i th inertial sensors after discretization in the world coordinate system respectively, q b i + 1 w , q b i w represent attitudes at the output moment of the i+1 th and the i th inertial sensors after discretization in the world coordinate system respectively, and δt represents an time interval of outputs between the i+1 th and the i th inertial sensors; and acceleration ā and angular velocity ω are the average value of two quantity measurements, that is: a ¯ = 1 2 [ q b i w ( a b i - b a i - n a ) - g w + q b i + 1 w ( a b i + 1 - b a i + 1 - n a ) - g w ] ω ¯ = 1 2 [ ( ω b i - b ω i - n ω ) + ( ω b i + 1 - b ω i + 1 - n ω ) ] formula, a b i , a b i+l represent outputs of the accelerometer at the output moment of the i+1 th and the i th inertial sensors respectively, ω b i , ω a i+1 represent outputs of the gyroscope at the output moment of the i+1 th and the i th inertial sensors respectively, b a i , b a i+1 represent zero bias of the accelerometer at the output moment of the i+1 th and the i th inertial sensors respectively, and b ω i , b ω i+1 represent zero bias of the gyroscope at the output moment of the i+1 th and the i th inertial sensors respectively.
  7. 7 . The unmanned platform with bionic visual multi-source information and intelligent perception according to claim 5 , wherein the step ( 202 ) has the following specific method: in the step ( 202 ), defining the quantity of state to be optimized and estimated of the system within a sliding window as: X = [ x 1 imu , … , x k imu , … , x s imu , x 1 lidar , … , x k lidar , … , x s lidar ] T x k imu = [ p b k w , v b k w , q b k w , b a k , b ω k ] T x k lidar = [ p l k w , q l k w ] T , k ∈ [ 0 , s ] in the formula, the sliding window is provided with the quantities of state pre-integrated by the inertial sensor at s moments and the quantities of state obtained by the laser radar at s moments, the laser radar has a frequency of 10 Hz, the inertial sensor has a frequency of 200 Hz, and pre-integration is carried out between every two frames of the laser radar through the inertial sensor; x 1 i ⁢ m ⁢ u , x k i ⁢ m ⁢ u , x s i ⁢ m ⁢ u represent pre-integrated states of the inertial sensor at an initial moment (a first moment), a moment k and a moment s respectively; and x 1 lidar , x k lidar , x s lidar represent the quantities of state of the laser radar at the initial moment, the moment k and the moment s respectively; and b a k , b ω k are zero bias of the accelerometer and the gyroscope at the moment k, and p l k w , q l k w are a position and an attitude estimated by the laser radar at the moment k; after state variables are defined, the residual term of the inertia sensor and the laser radar is designed, the Jacobian of the residual term relative to variables to be optimized is calculated, and a nonlinear optimization function of the laser radar/inertial combined navigation is then constructed: min X { ∑ k = 0 s ⌈ r B k ( z k - 1 k , x k i ⁢ m ⁢ u ) ⌉ 2 + ∑ k = 0 s ⌈ r L k ( m k , x k lidar ) ⌉ 2 } in the formula, ┌ ┐ represents a Mahalanobis Distance of the vector, r B k ( ) represents a measurement residual of the inertial sensor, r L k ( ) represents a point cloud measurement residual of the laser radar, z k k + 1 represents relative measurement variation of the inertial sensor between the moments k and k−1, and m k represents measurement information of the laser radar at the moment k; finally, a Gauss-Newton method is adopted to solve the nonlinear optimization function, so as to obtain the optimal pose estimation.
  8. 8 . The unmanned platform with bionic visual multi-source information and intelligent perception according to claim 1 , wherein the deep learning object detection module has the following specific method: first, the monocular vision camera sensor is used to collect RGB images of a surrounding environment, with the collection frequency of 30 Hz; a bilinear interpolation algorithm is then adopted to adjust grayscale images to be a size of 640×640, and a median filtering algorithm is adopted to denoise the images and complete pre-processing of the images; and a multi-directional blur superposition algorithm is adopted to simulate a blurred image problem caused by a vision camera in a real environment, as shown in the following formula: P s ⁢ t ⁢ a ⁢ c ⁢ k R = ∑ j m T j ( P j R · γ j ) P s ⁢ t ⁢ a ⁢ c ⁢ k G = ∑ j m T j ( P j G · γ j ) ⁢ j = 1 , 2 , 3 , … , m P s ⁢ t ⁢ a ⁢ c ⁢ k B = ∑ j m T j ( P j B · γ j ) in the formula, P stack R , P stack G , P stack B represent pixel values of corresponding coordinates on the images after superposition on three RGB channels respectively, P j R , P j G , P j B represent pixel values of the original coordinates on the three RGB channel respectively, γ j represents a transparency factor of the superposed images, and T j represents a blurring superposition displacement matrix, which is determined by a superposition direction, a step length and the number of superposed images; and an image data set simulated by the multi-directional blur superposition algorithm is input to a YOLOv7 deep learning network for training, finally enhanced images are input to the trained YOLOv7 deep learning network to obtain object detection frames and vertex coordinates of various objects in the images.

Description

CROSS-REFERENCE TO RELATED APPLICATION This application is a continuation of international application of PCT application serial no. PCT/CN2023/101257 filed on Jun. 20, 2023, which claims the priority benefit of China application no. 202211481989.3 filed on Nov. 24, 2022. The entirety of each of the above mentioned patent applications is hereby incorporated by reference herein and made a part of this specification. TECHNICAL FIELD The present disclosure belongs to the technical field of computer vision and deep learning, and particularly relates to an unmanned platform with bionic visual multi-source information and intelligent perception. BACKGROUND Modern combat attaches great importance to intelligence and reconnaissance. In recent years, with the development of equipment technology, an unmanned platform having such functions as object detection, environmental perception, and autonomous navigation and positioning has been increasingly applied to military reconnaissance. Therefore, the positioning precision, operational autonomy and concealment of the unmanned platform have become key technologies in urgent need of development. Being subject to such defects as error accumulation, and susceptibility to interference, inertial navigation, geomagnetic navigation and other traditional navigation methods are difficult to obtain the information about precise and stable course heading and position in complex environments. Therefore, other sensors are needed to be developed to assist in improving the positioning precision. By making reference to insects, polarization vision navigation is developed as a new bionic navigation method, which features error-free accumulation, being difficult to be interfered, neither emitting signals externally nor exposing the existence of a carrier itself. Moreover, by combing the polarization vision navigation with inertial in a field environment, the unmanned platform can achieve high-precision positioning and concealed operation. However, when the unmanned platform is in a closed environment, a polarized vision sensor loses lock, in which case, other sensors are needed to provide navigation and positioning information, and the navigation method of traditional single sensors is unable to meet the navigation and positioning requirements in the complex environment changing scenarios. In addition, in order to realize efficient reconnaissance, it is necessary to improve the environment perception capability of the unmanned platform, to which object recognition technology is crucial. With continuous development of the deep learning technology, the tedious process of artificially designed features is avoided. Compared with the traditional object detection technology, the object detection technology based on deep learning has been proven to have higher precision and stronger robustness. With the development of the deep neural network, research in the field of object detection is basically divided into two directions, one is a two-stage object detection algorithm based on a region proposal, such as Faster-RCNN; and the other is a single-stage object detection algorithm based on regression calculation, such as YOLOv7. Although the two-stage object detection algorithm based on a region proposal has a high precision, its detection efficiency is not high. While the single-stage object detection algorithm, such as YOLOv7, proposed in 2022, surpasses all known object detectors in both velocity and accuracy in the range from 5 FPS to 160 FPS. At present, no relevant research on unmanned platforms based on autonomous navigation, object detection and environment perception is available across China. SUMMARY Aiming at the problem of a lack of research on unmanned platforms based on autonomous navigation, object detection and environmental perception, the present disclosure provides an unmanned platform with bionic visual multi-source information and intelligent perception, the combined navigation technology based on the bionic polarization vision/inertia/laser radar, and the object detection and environment perception technology based on the deep learning are applied to the unmanned platform, such that, on the one hand, the problem that satellite navigation is susceptible to counter-reconnaissance and interference during wartime can be avoided, the concealment and autonomous navigation capabilities of the unmanned platform in complex environments (field environment and unknown enclosed environment) can be improved, and the stability and robustness of a combined navigation system are enhanced; on the other hand, a vision camera is used for intelligent perception aiming in the complex environments to improve the object detection and autonomous obstacle avoidance capabilities of the unmanned platform. The above objective is implemented through the following technical solution: an unmanned platform with bionic visual multi-source information and intelligent perception, including a bionic polarization vi