CN-121432459-B - Unmanned aerial vehicle visual radar tight coupling positioning method, device, equipment, medium and product

CN121432459BCN 121432459 BCN121432459 BCN 121432459BCN-121432459-B

Abstract

The application discloses a method, a device, equipment, a medium and a product for positioning visual radars of unmanned aerial vehicles in a tight coupling way, and relates to the technical field of autonomous navigation of unmanned aerial vehicles. The method comprises the steps of processing image data by adopting a semantic segmentation network, screening feature points according to semantic segmentation results, carrying out feature matching constraint on static feature points obtained by screening based on a semantic consistency principle, projecting point cloud data under an image coordinate system, endowing semantic information and carrying out semantic filtering processing, carrying out plane feature extraction on structural semantic point clouds based on semantic-geometric consistency serving as constraint conditions, carrying out merging optimization on a plurality of semantic planes to obtain a structural semantic map of an environment, determining an observation model based on point-plane constraint residual errors and semantic weight coefficients, and carrying out tight coupling filtering and optimizing processing by adopting IESKF to obtain updated state estimation information. The application can improve the positioning accuracy and robustness.

Inventors

PANG ZHONGHUA
CHEN MINGHAO
WANG LING
YUAN FEI
Du yalong
GUO HAIBIN
Meng xiangze

Assignees

北方工业大学
北京中科神探科技有限公司

Dates

Publication Date: 20260508
Application Date: 20251202

Claims (10)

1. The utility model provides an unmanned aerial vehicle vision radar tight coupling positioning method which is characterized in that the method includes: the method comprises the steps of acquiring information data, wherein the information data comprises image data based on an environment acquired by an unmanned aerial vehicle onboard camera and point cloud data acquired based on an onboard laser radar; Processing the image data by adopting a semantic segmentation network YOLOv-seg to obtain a semantic segmentation result containing a dynamic object and a structural static object; Screening feature points in the image data extracted based on the visual odometer according to the semantic segmentation result, and carrying out feature matching constraint on static feature points obtained by screening based on a semantic consistency principle to obtain visual feature points; Projecting the point cloud data to an image coordinate system corresponding to the image data, endowing semantic information to each laser point contained in the projected point cloud data according to the semantic segmentation result, filtering dynamic and low-confidence point clouds according to the semantic information, and reserving high-confidence point clouds to obtain structural semantic point clouds, wherein the low-confidence point clouds are point clouds with confidence degrees lower than preset confidence degrees; based on semantic-geometric consistency as constraint conditions, extracting plane characteristics of the structural semantic point cloud to obtain a plurality of semantic planes, and combining and optimizing the plurality of semantic planes to obtain a structural semantic map of the environment; After the visual feature points are transformed to a radar coordinate system, searching a nearest neighbor semantic plane in the structural semantic map, and determining by calculating the vertical distance between the structural semantic point cloud and the visual feature points to the nearest neighbor semantic plane; And determining an observation model based on the point-plane constraint residual error and the semantic weight coefficient, and performing tight coupling filtering and optimizing processing by adopting IESKF to obtain updated state estimation information, wherein the updated state estimation information is used for realizing tight coupling positioning of the unmanned aerial vehicle.
2. The unmanned aerial vehicle visual radar tight coupling positioning method according to claim 1, wherein feature points in image data extracted based on a visual odometer are screened according to the semantic segmentation result, and feature matching constraint is carried out on static feature points obtained by screening based on a semantic consistency principle, so that visual feature points are obtained, and the method specifically comprises the following steps: According to the semantic segmentation result, feature points in the image data extracted based on the visual odometer are screened through a semantic mask, wherein the expression corresponding to the semantic mask is as follows: ; performing feature matching constraint on the static feature points obtained by screening based on a semantic consistency principle to obtain visual feature points, wherein the semantic consistency principle is determined by adopting a semantic consistency function and a semantic weighted feature matching cost function, and the expression of the semantic consistency function is as follows: ; The expression of the feature matching cost function is: ; Wherein, the Is a semantic mask; Pixel coordinates of feature points in the image data; category labels obtained based on semantic segmentation results; Obtained for the result based on semantic segmentation A corresponding category label; Is a set of dynamic object categories; is a semantic consistency function; Is the first Frame No Semantic tags of the feature points; Is the first Frame No Semantic tags of the feature points; matching a cost function for the feature; Is the first Frame No Feature points; First, the Frame No Feature points; Is that Corresponding feature descriptors; Is that Corresponding feature descriptors; Is that And (3) with Hamming distance between them; Is a super parameter.
3. The unmanned aerial vehicle visual radar tight coupling positioning method according to claim 1, wherein the point cloud data is projected under an image coordinate system corresponding to the image data, semantic information is given to each laser point contained in the projected point cloud data according to the semantic segmentation result, dynamic and low-confidence point clouds are filtered according to the semantic information, and high-confidence point clouds are reserved to obtain structural semantic point clouds, and the method specifically comprises the following steps: Transforming each point in the point cloud data into a camera coordinate system through an external parameter matrix between an airborne laser radar obtained through calibration and an airborne camera of the unmanned aerial vehicle to obtain point cloud data in the camera coordinate system; projecting the point cloud data under a camera coordinate system to an image coordinate system corresponding to the image data through a camera internal reference matrix of an unmanned aerial vehicle-mounted camera to obtain projected point cloud data; according to the semantic segmentation result, semantic information is endowed to each laser point contained in the projected point cloud data, point cloud filtering processing is carried out on the basis of semantic probability distribution to obtain structural semantic point cloud, the semantic probability distribution is determined by adopting a bilinear interpolation method, and a mathematical expression corresponding to the semantic probability distribution is as follows: ; Wherein, the Is a semantic probability distribution; coordinates of neighboring pixels; is interpolation weight; And All are cyclic index variables used for bilinear interpolation calculation; to pair(s) Is rounded downwards; to pair(s) Is rounded downwards on the ordinate of (2); Pixel coordinates of feature points in the image data; Is a category probability distribution vector.
4. The unmanned aerial vehicle vision radar tight coupling positioning method according to claim 1, wherein the constraint conditions specifically comprise a geometric proximity condition and a semantic consistency condition based on semantic-geometric consistency as constraint conditions; The mathematical expression corresponding to the geometric proximity condition is as follows: ; the mathematical expression corresponding to the semantic consistency condition is as follows: ; Wherein, the Is a plane normal vector; the points contained in the point set corresponding to the structural semantic point cloud; constant terms that are plane equations; Is that Is a norm of (2); is a preset distance threshold; Is that Semantic tags of (2); Semantic tags for generating seed points for the hypothetical planes; Is a semantic consistency function.
5. The unmanned aerial vehicle visual radar tight coupling positioning method of claim 1, wherein the mathematical expression of the observation model comprises: ; ; Wherein, the The method is a vision-radar point-plane observation model; Is a semantic label Corresponding weight coefficients; estimating x for pose by unmanned aerial vehicle Distance when projected onto a radar map plane; Nearest neighbor semantic plane matched with structural semantic map Is a transpose of the parameters of (a); a transformation matrix from world system to laser radar system; The pose estimation x is transformed into a corresponding structural semantic point cloud after a laser radar coordinate system; 、 are nearest neighbor semantic planes matched with the structural map Parameters of (2); the method is a laser radar point-plane observation model; estimating x for pose by unmanned aerial vehicle Distance when projected onto the semantic plane; is a radar semantic point cloud.
6. The unmanned aerial vehicle vision radar tight coupling positioning method according to claim 1, wherein an observation model is determined based on the point-plane constraint residual error and the semantic weight coefficient, and the tight coupling filtering and optimization processing is performed by IESKF to obtain updated state estimation information, and the method specifically comprises the following steps: Determining an observation model based on the point-plane constraint residual error and the semantic weight coefficient; determining a composite observation model based on the observation model, and determining a jacobian matrix of the composite observation model on a preset error state vector, wherein a mathematical expression corresponding to the jacobian matrix is as follows: ; Wherein, the Is a jacobian matrix; Is a composite observation model; Is a preset error state vector; and based on the jacobian matrix, performing tight coupling filtering and optimization processing by IESKF to obtain updated state estimation information.
7. Unmanned aerial vehicle vision radar tight coupling positioner, its characterized in that includes: The information data acquisition module is used for acquiring information data, wherein the information data comprises image data based on an environment acquired by an unmanned aerial vehicle-mounted camera and point cloud data acquired based on an airborne laser radar; The image processing module is used for processing the image data by adopting a semantic segmentation network YOLOv-seg to obtain a semantic segmentation result containing a dynamic object and a structural static object; the screening matching module is used for screening the feature points in the image data extracted based on the visual odometer according to the semantic segmentation result, and carrying out feature matching constraint on the static feature points obtained by screening based on a semantic consistency principle to obtain visual feature points; The projection processing module is used for projecting the point cloud data under an image coordinate system corresponding to the image data, endowing semantic information to each laser point contained in the projected point cloud data according to the semantic segmentation result, filtering dynamic and low-confidence point clouds according to the semantic information, and reserving high-confidence point clouds to obtain structural semantic point clouds, wherein the low-confidence point clouds are point clouds with confidence degrees lower than preset confidence degrees; The extraction optimization module is used for extracting plane characteristics of the structural semantic point cloud based on semantic-geometric consistency serving as constraint conditions to obtain a plurality of semantic planes, and combining and optimizing the plurality of semantic planes to obtain a structural semantic map of the environment; The point-plane constraint residual determination module is used for determining a point-plane constraint residual, wherein the point-plane constraint residual is determined by searching a nearest neighbor semantic plane in the structural semantic map after the visual feature point is transformed to a radar coordinate system and calculating the vertical distance between the structural semantic point cloud and the visual feature point and the nearest neighbor semantic plane; the optimization estimation module is used for determining an observation model based on the point-plane constraint residual error and the semantic weight coefficient, performing tight coupling filtering and optimization processing by adopting IESKF to obtain updated state estimation information, and the updated state estimation information is used for realizing tight coupling positioning of the unmanned aerial vehicle.
8. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor executes the computer program to implement the unmanned aerial vehicle visual radar tight coupling positioning method of any of claims 1-6.
9. A computer readable storage medium having stored thereon a computer program, characterized in that the computer program, when executed by a processor, implements the unmanned aerial vehicle visual radar tight coupling positioning method of any of claims 1-6.
10. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the unmanned aerial vehicle visual radar tight-coupling positioning method of any of claims 1-6.

Description

Unmanned aerial vehicle visual radar tight coupling positioning method, device, equipment, medium and product Technical Field The application relates to the technical field of unmanned aerial vehicle autonomous navigation, in particular to an unmanned aerial vehicle visual radar tight coupling positioning method, device, equipment, medium and product. Background With the development of low-altitude economy, the real-time accurate synchronous positioning and map construction (Simultaneous Localization AND MAPPING, SLAM) technology is a core foundation for realizing complete autonomy in application scenes such as unmanned aerial vehicle autonomous navigation, unmanned aerial vehicle inspection and intelligent storage. The vision sensor can provide abundant environment texture information and has low cost, and the laser radar can directly acquire high-precision three-dimensional geometric information. However, the method has the limitations of using only a visual sensor or a laser radar, is easily influenced by illumination and texture change and has scale drift, and the laser radar provides accurate geometric information, but semantic information is not easy to obtain, is easily interfered by noise and has reduced performance in severe weather. To solve the single sensor defect, the vision-radar fusion scheme is the mainstream, but the following problems still exist: 1. the dynamic interference suppression is insufficient, namely an effective characteristic filtering mechanism is not designed aiming at the dynamic environment, in the scene of dynamic objects such as pedestrians, vehicles and the like, dynamic characteristic points are easy to be mistakenly identified as static reference characteristics, obvious drift of a visual odometer is caused, positioning accuracy is greatly reduced, and robustness is insufficient. 2. The feature matching reliability is low, the traditional feature matching only depends on the similarity of descriptors, is not constrained by combining semantic information, is easy to have a large number of mismatching in a texture repeated area (such as a long corridor and a dense building group), and further aggravates pose estimation errors and influences the accuracy of map construction. 3. The multisource data fusion is loose, most fusion schemes adopt a loose coupling mode, only carry out post-processing on independent positioning results, and complementation of two sensors is not deeply excavated. The visual point feature precision is high but lacks global structure information, lei Dadian clouds can provide depth information but are easy to be interfered by noise, and loose fusion cannot fully exert the advantages of the depth information and the noise, so that the positioning precision and the stability are difficult to be considered. 4. Map construction practicability is poor, namely most of constructed maps are unordered point clouds or sparse feature maps, semantic and structural information are not integrated, effective structural prior cannot be provided for subsequent tasks, and practical application value of the SLAM system is limited. Therefore, an innovative visual radar tight coupling positioning method is needed, the problems of dynamic interference, mismatching, loose fusion, poor map practicality and the like can be effectively solved, and high-precision and high-robustness positioning and map construction in complex dynamic and structured scenes are realized. Disclosure of Invention The application aims to provide a method, a device, equipment, a medium and a product for tightly coupling and positioning an unmanned aerial vehicle vision radar, which can improve the positioning accuracy and robustness. In order to achieve the above object, the present application provides the following solutions: in a first aspect, the application provides a method for positioning visual radars of unmanned aerial vehicles in tight coupling, which comprises the following steps: the method comprises the steps of acquiring information data, wherein the information data comprises image data based on an environment acquired by an unmanned aerial vehicle onboard camera and point cloud data acquired based on an onboard laser radar; Processing the image data by adopting a semantic segmentation network YOLOv-seg to obtain a semantic segmentation result containing a dynamic object and a structural static object; Screening feature points in the image data extracted based on the visual odometer according to the semantic segmentation result, and carrying out feature matching constraint on static feature points obtained by screening based on a semantic consistency principle to obtain visual feature points; Projecting the point cloud data to an image coordinate system corresponding to the image data, endowing semantic information to each laser point contained in the projected point cloud data according to the semantic segmentation result, filtering dynamic and low-confidence point clouds according