CN-121982125-A - Panoramic image generation method and device, ship and medium

CN121982125ACN 121982125 ACN121982125 ACN 121982125ACN-121982125-A

Abstract

The invention discloses a panoramic image generation method, a panoramic image generation device, a ship and a medium. The method is applied to a ship, a laser radar, a double-light pod and millimeter wave radars are arranged on the top end of the ship, the bow and the stern of the ship are respectively provided with the millimeter wave radars, the method comprises the steps of obtaining point cloud data of a ship operation area through the laser radars, obtaining double-light image data of multiple subarea views of the ship operation area through the double-light pod, obtaining obstacle detection data in the ship operation area through the millimeter wave radars, performing depth extraction processing on the point cloud data to obtain pixel-level depth information, performing coordinate conversion processing on the double-light image data based on the pixel-level depth information to obtain target visual data, performing clustering processing on the point cloud data to obtain obstacle point cloud clusters, performing feature extraction on the target visual data to obtain visual target detection results, and generating a target panoramic image of the ship operation area according to the obstacle detection data, the obstacle point cloud clusters and the visual target detection results.

Inventors

PANG JINGDUN
HONG GUOJUN
WU YI
YU KANGKANG
DING QI
ZHANG XIAOLIN

Assignees

中交疏浚技术装备国家工程研究中心有限公司

Dates

Publication Date: 20260505
Application Date: 20260407

Claims (10)

1. The method is characterized by being applied to a ship, wherein a laser radar and a double-light pod are installed on the top end of the ship, and millimeter wave radars are respectively installed on the bow and the stern of the ship, and the method comprises the following steps: acquiring point cloud data of a ship operation area through the laser radar, acquiring double-light image data of multiple subarea visual angles of the ship operation area through the double-light pod, and acquiring obstacle detection data in the ship operation area through the millimeter wave radar; Performing depth extraction processing on the point cloud data to obtain pixel-level depth information, and performing coordinate conversion processing on the dual-light image data based on the pixel-level depth information to obtain target visual data; clustering the point cloud data to obtain an obstacle point cloud cluster, and extracting features of the target visual data to obtain a visual target detection result; And generating a current panoramic image of the ship operation area according to the obstacle detection data, the obstacle point cloud cluster and the visual target detection result.
2. The method of claim 1, wherein the visual target detection results include a visible light detection result and an infrared detection result, wherein generating the current panoramic image of the marine operation area based on the obstacle detection data, the obstacle point cloud cluster, and the visual target detection result comprises: performing distortion correction and splicing processing on the double-light image data to obtain a double-light panoramic map; Matching the visible light detection result and the infrared detection result to obtain a cross-modal homologous target; Determining obstacle detection data and/or obstacle point cloud clusters corresponding to the cross-modal homologous targets based on a preset space threshold to obtain a target feature set; Performing feature stitching on the cross-modal homologous targets and the corresponding target feature sets to obtain target labeling information; And overlapping the target marking information to the pixel position corresponding to the double-light panoramic map to obtain the current panoramic image of the ship operation area.
3. The method according to claim 2, further comprising, after determining the obstacle detection data and/or the obstacle point cloud cluster corresponding to the cross-modal homology target based on a preset spatial threshold, obtaining a target feature set: Determining whether missing data exists, wherein the missing data is obstacle detection data and/or obstacle point cloud clusters which are not included in the target feature set; if missing data exists, traversing and matching the missing data with different data types to obtain a matching result, and determining the target labeling information based on the matching result and the missing data.
4. The method according to claim 2, further comprising, before superimposing the target labeling information to a pixel position corresponding to the dual-light panoramic map to obtain a current panoramic image of the ship operation area: Determining current predicted position information of each historical obstacle based on the position information and motion vector information of each historical obstacle in the last panoramic image; Matching the target position information in the target labeling information with the predicted position information of each historical obstacle to obtain a position matching result; And updating the target identification information in the target labeling information with the successful matching result as the identification information of the corresponding historical obstacle in the position matching result.
5. The method according to claim 2, wherein the dual-light image data includes visible light image data and infrared image data, the dual-light panoramic map includes a visible light panoramic map and an infrared panoramic map, and the performing distortion correction and stitching processing on the dual-light image data to obtain the dual-light panoramic map includes: Performing distortion correction processing on the visible light image data of each view angle to obtain visible light image data of each target, and performing distortion correction processing on the infrared image data of each view angle to obtain infrared image data of each target; Extracting homonymous feature points of the visible light image data of each target, and carrying out fusion splicing on the visible light image data of each target based on the homonymous feature points to obtain the visible light panoramic map; and carrying out projection splicing processing on the infrared image data of each target to obtain the infrared panoramic map.
6. The method of claim 1, wherein performing depth extraction processing on the point cloud data to obtain pixel-level depth information comprises: Performing space projection processing on the point cloud data based on the internal reference matrix and the external reference matrix of the double-light pod to obtain pixel position information and vertical position information of the point cloud data under a double-light pod coordinate system; and determining pixel-level depth information of the pixel point corresponding to the pixel position information according to the pixel position information and the vertical position information corresponding to the pixel position information.
7. The method of claim 1, further comprising, prior to generating the current panoramic image of the marine work area based on the obstacle detection data, the obstacle point cloud cluster, and the visual target detection result: filtering the obstacle detection data based on a preset signal-to-noise ratio threshold value and a preset reflection sectional area threshold value to obtain initial obstacle detection data; Performing dynamic compensation processing on the initial obstacle detection data according to the self-ship speed and the relative speed of the ship to obtain intermediate obstacle detection data; And performing coordinate conversion processing on the intermediate obstacle detection data to obtain updated obstacle detection data.
8. A panoramic image generation device, characterized in that is applied to boats and ships, install laser radar and two light pods on the top of boats and ships, install millimeter wave radar respectively on bow and stern of boats and ships, the device includes: the acquisition module is used for acquiring point cloud data of a ship operation area through the laser radar, acquiring double-light image data of a multi-partition view angle of the ship operation area through the double-light pod and acquiring obstacle detection data in the ship operation area through the millimeter wave radar; The conversion module is used for carrying out depth extraction processing on the point cloud data to obtain pixel-level depth information, and carrying out coordinate conversion processing on the double-light image data based on the pixel-level depth information to obtain target visual data; the extraction module is used for carrying out clustering treatment on the point cloud data to obtain an obstacle point cloud cluster, and carrying out feature extraction on the target visual data to obtain a visual target detection result; And the generation module is used for generating a current panoramic image of the ship operation area according to the obstacle detection data, the obstacle point cloud cluster and the visual target detection result.
9. A vessel, the vessel comprising: And a memory communicatively coupled to the at least one processor; Wherein the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the method of generating a panoramic image as claimed in any one of claims 1 to 7.
10. A storage medium containing computer executable instructions, which when executed by a computer processor are for performing the method of generating a panoramic image as claimed in any one of claims 1 to 7.

Description

Panoramic image generation method and device, ship and medium Technical Field The embodiment of the invention relates to the technical field of images, in particular to a panoramic image generation method, a panoramic image generation device, a ship and a medium. Background Along with continuous promotion of automation and intelligent level of ship operation, comprehensive and accurate perception of operation environment has become key for guaranteeing navigation and operation safety. Particularly in the scenes of complex water areas, port operations, offshore projects and the like, the ship needs to master surrounding environment information in real time and in all directions. Therefore, the generation of panoramic images of a ship work area (i.e., the surrounding environment) has become an important means for realizing the omni-directional perception of the environment, and the necessity and importance thereof are increasingly highlighted. Currently, the panoramic image generation method mainly adopts an image stitching technology based on a single vision sensor (such as a monocular camera and a multi-eye camera array). However, the method is difficult to accurately judge the distance between the ship and the obstacle and the three-dimensional form of the obstacle, so that the generated panoramic image is not accurate and reliable enough. Therefore, a new method is needed to solve the above problems. Disclosure of Invention The invention provides a panoramic image generation method, a panoramic image generation device, a ship and a medium, which can effectively improve the accuracy and reliability of panoramic image generation. In a first aspect, an embodiment of the present invention provides a method for generating a panoramic image, which is applied to a ship, wherein a laser radar and a dual-light pod are installed on a top end of the ship, and millimeter wave radars are installed on a bow and a stern of the ship, respectively, and the method includes: acquiring point cloud data of a ship operation area through the laser radar, acquiring double-light image data of multiple subarea visual angles of the ship operation area through the double-light pod, and acquiring obstacle detection data in the ship operation area through the millimeter wave radar; Performing depth extraction processing on the point cloud data to obtain pixel-level depth information, and performing coordinate conversion processing on the dual-light image data based on the pixel-level depth information to obtain target visual data; clustering the point cloud data to obtain an obstacle point cloud cluster, and extracting features of the target visual data to obtain a visual target detection result; And generating a current panoramic image of the ship operation area according to the obstacle detection data, the obstacle point cloud cluster and the visual target detection result. According to the technical scheme, point cloud data of a ship operation area are acquired through the laser radar, double-light image data of a multi-partition view angle of the ship operation area are acquired through the double-light pod, obstacle detection data in the ship operation area are acquired through the millimeter wave radar, environment information of the ship operation area can be comprehensively acquired from multiple dimensions such as a three-dimensional space structure, a multispectral visual image and obstacle position information, multi-source, multi-mode and all-directional perception of the operation area is achieved, the defects of a single sensor in imaging, ranging or obstacle detection are effectively overcome, the integrity and reliability of environment perception are improved, and a rich and accurate data base is provided for subsequent generation of a high-precision and high-robustness panoramic image. And then, carrying out depth extraction processing on the point cloud data to obtain pixel-level depth information, carrying out coordinate conversion processing on the double-light image data based on the pixel-level depth information to obtain target visual data, realizing the spatial precise alignment of the image data and the point cloud data, and providing a strictly registered data base for the subsequent generation of a high-precision panoramic image. And then, clustering the point cloud data to obtain an obstacle point cloud cluster, extracting the characteristics of the target visual data to obtain a visual target detection result, improving the accuracy and effectiveness of obstacle positioning, defining the specific form and visual attribute of the visual target, realizing the accurate classification and image layer positioning of the obstacle, compensating the limitation of single-mode visual detection, and providing reliable data support for the generation of subsequent panoramic images. And finally, generating a current panoramic image of the ship operation area according to the obstacle detection data, the obstacle point