Search

CN-121999062-A - Method and system for enhancing color in a set of images

CN121999062ACN 121999062 ACN121999062 ACN 121999062ACN-121999062-A

Abstract

Methods and systems for enhancing colors in a set of images. The method includes obtaining input data comprising 3D data and image data, the image data being a set of individual images, each image having a portion overlapping with the other images and being comprised of a plurality of pixels, each pixel providing radiation measurement information, performing joint integration on the input data to create integrated data, joint integration comprising 3D registration of the 3D data and the image data, spatial map decomposition to generate a structured representation, and homography decomposition to estimate homography in the image data, performing multi-scale feature extraction and block-wise feature embedding on the integrated data to integrate features of the 3D data and the image data into a single feature vector and embed both geometric information and radiation measurement information, generating a block-wise weight map for each homography in the overlapping portion based on the block-wise feature embedding, performing pixel-wise correction in the set of images based on the weight maps to generate a set of corrected images having homogenized radiation measurement information.

Inventors

  • L. Lopez Fernandez
  • A. Prado de Torreblanca
  • WINISTOERFER MARTIN

Assignees

  • 海克斯康创新中心有限公司

Dates

Publication Date
20260508
Application Date
20251107
Priority Date
20241108

Claims (15)

  1. 1. A computer-implemented method (100) for homogenizing radiometric information in a set of images (12 a-f) of an environment, the computer-implemented method comprising: -acquiring (110) input data (1), the input data comprising 3D data (10), the 3D data comprising geometrical information of the environment and image data (12) of the environment, wherein the image data (12) is acquired as a set of images (12 a-f), each image having an overlapping portion (15) with other images of the set of images and being composed of a plurality of pixels, each pixel providing radiometric information; -performing (120) a joint integration (2) on the input data (1) to create integrated data, the joint integration comprising a 3D registration (21) of the 3D data (10) and the image data (12), a spatial map decomposition (22) for generating a structured representation, and a homography decomposition (23) for estimating homographies in the image data (12); performing (130) multi-scale feature extraction (31) and block-wise feature embedding on the integration data, thereby integrating features of the 3D data and the image data in a single feature vector, and embedding geometry and radiometric information; Generating (140) a block-wise weight map (32) for each homography in the overlap portion (15) based on the block-wise feature embedding, and Based on the weight map (32), pixel-by-pixel correction (33) in the set of images (12 a-f) is performed (150) to generate a set of corrected images (18) with homogenized radiation measurement information.
  2. 2. The method (100) of claim 1, the method comprising propagating (160) the correction image (18) to the 3D data (10) to generate a colorized 3D model (6) of the environment, in particular wherein the colorized 3D model (6) is one of a colorized 3D point cloud or a colorized 3D grid.
  3. 3. The method (100) of claim 1, the method comprising stitching (170) the corrected images (18) to generate a panoramic image (5) of the environment.
  4. 4. The method (100) according to any one of the preceding claims, wherein the 3D data (10) and the image data (12) are acquired (110) by the same reality capturing device (50, 51).
  5. 5. The method (100) of claim 4, wherein Acquiring (110) the input data (1) at a plurality of moments and/or from a plurality of locations of the reality capturing device (50, 51), and The 3D registration (21) comprises registration of 3D data acquired at each of the time instants and/or positions.
  6. 6. The method (100) according to claim 4 or claim 5, wherein the method (100) is performed by a computing unit of the reality capturing device (50, 51), in particular wherein the computing unit is configured to control the acquisition of the input data (1) by the reality capturing device (50, 51).
  7. 7. The method (100) according to any one of the preceding claims, wherein At least a subset of the images (12 a-f) having non-overlapping portions (16), each non-overlapping portion imaging a portion of the environment that is not imaged in any other image (12 a-f) of the set of images, and -Performing (150) the pixel-by-pixel correction (33) in the overlapping portion (15) and the non-overlapping portion (16) of the images (12 a-f).
  8. 8. The method (100) according to any one of the preceding claims, wherein the 3D data (10) comprises: a point cloud acquired by a LiDAR unit (53) or a plurality of ToF cameras (54), and/or Depth map.
  9. 9. The method (100) according to any one of the preceding claims, wherein the 3D registration (21) comprises image-LiDAR internal and external parameters.
  10. 10. The method (100) according to any one of the preceding claims, wherein the radiometric information comprises at least a color, in particular wherein the radiometric information further comprises a brightness and/or a contrast.
  11. 11. The method (100) according to any one of the preceding claims, wherein the geometric information comprises 3D coordinates of a plurality of points.
  12. 12. The method (100) according to any of the preceding claims, wherein the multi-scale feature extraction (31) and the block-wise feature embedding are performed (130) by a neural network (3).
  13. 13. The method (100) according to any one of the preceding claims, wherein the plurality of single images (12 a-f) are acquired by a plurality of cameras having overlapping fields of view.
  14. 14. A reality capturing device (50, 51) comprising a plurality of sensors configured to acquire input data (1) comprising image data (12) and 3D data (10) of an environment, and a computing unit configured to control the acquisition of the input data (1), in particular wherein the plurality of sensors comprises: a plurality of image sensors for capturing the image data (12), and A LiDAR unit (53) or a plurality of ToF-cameras (54), the LiDAR unit (53) or the plurality of ToF-cameras (54) being for capturing 3D data (10), It is characterized in that The computing unit stores program code for performing the method (100) according to any of the preceding claims.
  15. 15. Computer program product comprising a program code stored on a machine readable medium or embodied by an electromagnetic wave comprising program code segments and having computer executable instructions for performing the method (100) according to any one of claims 1 to 13, in particular when the computer executable instructions are executed in a computing unit of a reality capturing device (51, 52) according to claim 14.

Description

Method and system for enhancing color in a set of images Technical Field The present invention relates to a method and system for enhancing color and other radiometric information in a set of images of an environment, for example, to generate panoramic images or colorized three-dimensional point clouds or grids. In particular, enhancing the color includes improved normalization of color information of image data from multiple images of the environment using 3D data of the same environment. For example, this allows for more uniform colorization of 2D or 3D data of an environment even though images are captured under different lighting conditions. Background Three-dimensional point clouds are generated for investigation of many different scenarios (setting), such as building sites, building facades, industrial facilities, house interiors or any other suitable scenario. The survey thus achieved may be used to obtain an accurate three-dimensional (3D) model of the scene, where the model includes a point cloud. The points of such a cloud are stored as coordinates in a coordinate system that can be defined by the surveying instrument that recorded the point cloud. Usually, the surveying instrument constitutes the origin of the coordinate system by the instrument center, in particular by a so-called node of the surveying instrument. The points are usually surveyed by correlating the distance measured with the laser beam (by means of a time-of-flight method) with the alignment under which the distance is measured. Typically, the coordinate system is a spherical coordinate system such that the point is characterized by a distance value, an elevation angle, and an azimuth angle of the origin of the reference coordinate system. A common surveying instrument comprises a unit for emitting a scanning beam and for receiving a reflected beam for measuring the distance of the point at which the beam is directed. Typically, these surveying instruments also comprise means for rotatably changing the beam direction, typically a vertical rotation axis and a horizontal rotation axis, both axes being sensed with an angle sensor. Typically, rotation of the vertical axis is measured by azimuth and rotation of the horizontal axis is measured by elevation. If the surveying instrument is embodied as a laser scanner, one of the axes may be the slow axis and the other may be the fast axis. The distance can be calculated by observing the time between sending and receiving the signal using a time of flight measurement (time of flight) method. The alignment angle is achieved with the angle sensors arranged at the vertical axis and the horizontal axis. In the survey field, providing a colorized 3D point cloud is a desirable feature, for example for LiDAR-based survey tools. The color features facilitate understanding and navigating through the scene, as well as identifying elements of interest, thereby providing a more "friendly" product to the human visual system than the uncolored point cloud. In addition, color features are widely used as input features for many of the most advanced point cloud post-processing algorithms (e.g., segmentation, classification, and/or modeling algorithms). Furthermore, calibration and projection of 3D data to an image is a desirable feature, as this creates a "metrology" image in which certain measurements can be performed directly in the image. To provide better visualization, the point cloud may be digitally colorized. In various applications, to provide color information for colorizing the point cloud, the surface survey is thus supported by imaging data of at least one calibrated imaging sensor (e.g., camera) that is combined with the survey instrument by including the camera in the instrument or mounting it on the same platform as the instrument. These imaging sensors are integrated or attached to a LiDAR measurement system with accurate intrinsic and extrinsic camera calibration so that two features acquired by the imaging sensors can be projected/mapped to a 3D LiDAR point cloud and vice versa. Devices configured to generate a digital three-dimensional representation of an environment by capturing 3D data simultaneously with a panoramic image of the environment are also referred to as "reality capturing devices". WO 2020/126123 A2 discloses such a reality capturing device with a laser scanner and a plurality of RGB cameras. EP 4 095 561 A1 discloses a real-world capturing device combining a plurality of time-of-flight cameras for capturing 3D data with a plurality of RGB cameras. Some LiDAR measurement systems integrate simultaneous localization and mapping (simultaneous location AND MAPPING, SLAM) techniques and/or additional localization techniques that enable dynamic use of these devices. As such, to measure and reconstruct a 3D scene along a dynamic trajectory, liDAR may be carried by an operator or mounted to a transport platform, such as an unmanned ground vehicle (unmanned ground vehicle, UGV).