Search

EP-4742145-A1 - METHOD AND SYSTEM FOR ENHANCING COLOUR IN A SET OF IMAGES

EP4742145A1EP 4742145 A1EP4742145 A1EP 4742145A1EP-4742145-A1

Abstract

Computer-implemented method for homogenizing radiometry information in a set of images of an environment, the method comprising: acquiring input data (1) comprising 3D data (10) comprising geometry information and image data (12) that is acquired as a set of single images, each image having overlapping portions (15) with other images of the set and being composed of a multitude of pixels, each pixel providing radiometry information; performing a joint integration (2) on the input data to create integrated data, the joint integration comprising 3D registration (21) of the 3D data and the image data, a spatial graph decomposition (22) to generate a structured representation, and a multi-homography decomposition (23) to estimate homographies in the image data; performing multi-scale feature extraction (31) and per-patch feature embedding on the integrated data, thereby integrating features of the 3D data and the image data in a single feature vector and embedding both geometry and radiometry information; generating, based on the per-patch feature embedding, a per-patch weight map (32) for each homography in the overlapping portions; and performing, based on the weight maps, a per-pixel correction (33) in the set of images to generate a set of corrected images (18) having the homogenized radiometry information.

Inventors

  • LOPEZ FERNANDEZ, LUIS
  • PRADOS-TORREBLANCA, Andres
  • Winistörfer, Martin

Assignees

  • Hexagon Innovation Hub GmbH

Dates

Publication Date
20260513
Application Date
20241108

Claims (15)

  1. Computer-implemented method (100) for homogenizing radiometry information in a set of images (12a-f) of an environment, the method comprising - acquiring (110) input data (1), the input data comprising 3D data (10) comprising geometry information of the environment and image data (12) of the environment, wherein the image data (12) is acquired as a set of images (12a-f), each image having overlapping portions (15) with other images of the set and being composed of a multitude of pixels, each pixel providing radiometry information; - performing (120) a joint integration (2) on the input data (1) to create integrated data, the joint integration comprising 3D registration (21) of the 3D data (10) and the image data (12), a spatial graph decomposition (22) to generate a structured representation, and a multi-homography decomposition (23) to estimate homographies in the image data (12); - performing (130) multi-scale feature extraction (31) and per-patch feature embedding on the integrated data, thereby integrating features of the 3D data and the image data in a single feature vector and embedding both geometry and radiometry information; - generating (140), based on the per-patch feature embedding, a per-patch weight map (32) for each homography in the overlapping portions (15); and - performing (150), based on the weight maps (32), a per-pixel correction (33) in the set of images (12a-f) to generate a set of corrected images (18) having the homogenized radiometry information.
  2. Method (100) according to claim 1, comprising propagating (160) the corrected images (18) onto the 3D data (10) to generate a colourized 3D model (6) of the environment, particularly wherein the colourized 3D model (6) is one of a colourized 3D point cloud or a colourized 3D mesh.
  3. Method (100) according to claim 1, comprising stitching (170) the corrected images (18) to generate a panoramic image (5) of the environment.
  4. Method (100) according to any one of the preceding claims, wherein the 3D data (10) and the image data (12) are acquired (110) by the same reality-capture device (50, 51).
  5. Method (100) according to claim 4, wherein - the input data (1) is acquired (110) at a plurality of instants of time and/or from a plurality of locations of the reality-capture device (50, 51); and - the 3D registration (21) comprises registration of 3D data acquired at each of the instants of time and/or plurality of locations.
  6. Method (100) according to claim 4 or claim 5, wherein the method (100) is performed by a computing unit of the reality-capture device (50, 51), particularly wherein the computing unit is configured to control the acquisition of the input data (1) by the reality-capture device (50, 51).
  7. Method (100) according to any one of the preceding claims, wherein - at least a subset of the images (12a-f) has non-overlapping portions (16), each overlapping portion imaging a part of the environment that is not imaged in any other image (12a-f) of the set; and - the per-pixel correction (33) is performed (150) in the overlapping portions (15) and the non-overlapping portions (16) of the images (12a-f).
  8. Method (100) according to any one of the preceding claims, wherein the 3D data (10) comprises - a point cloud acquired by a LiDAR unit (53) or a plurality of ToF cameras (54); and/or - a depth map.
  9. Method (100) according to any one of the preceding claims, wherein the 3D registration (21) includes image-LiDAR intrinsics and extrinsic.
  10. Method (100) according to any one of the preceding claims, wherein the radiometry information comprises at least a colour, particularly wherein the radiometry information also comprises brightness and/or contrast.
  11. Method (100) according to any one of the preceding claims, wherein the geometry information comprises 3D coordinates of a multitude of points.
  12. Method (100) according to any one of the preceding claims, wherein the multi-scale feature extraction (31) and the per-patch feature embedding are performed (130) by a neural network (3).
  13. Method (100) according to any one of the preceding claims, wherein the multitude of single images (12a-f) are acquired by a multitude of cameras having overlapping fields of view.
  14. Reality-capture device (50, 51) comprising a plurality of sensors configured to acquire input data (1) comprising image data (12) and 3D data (10) of an environment, and a computing unit configured to control the acquisition of the input data (1), particularly wherein the plurality of sensors comprise - a plurality of image sensors for capturing the image data (12); and - a LiDAR unit (53) or a plurality of ToF-cameras (54) for capturing the 3D data (10), characterized in that the computing unit has program code stored for performing the method (100) according to any one of the preceding claims.
  15. Computer program product comprising program code, which is stored on a machine-readable medium, or being embodied by an electromagnetic wave comprising a program code segment, and having computer-executable instructions for performing, particularly when executed in a computing unit of a reality-capture device (51, 52) according to claim 14, the method (100) according to any one of claims 1 to 13.

Description

The present invention relates to a method and a system for enhancing colour and other radiometry information in a set of images of an environment, e.g. to generate panoramic images or colourised three-dimensional point clouds or meshes. In particular, enhancing the colour comprises an improved normalization of colour information from image data of a multitude of images of the environment using 3D data of the same environment. For instance, this allows for an evener colouring of 2D or 3D data of the environment even if the images are captured under different lighting conditions. Generating three-dimensional point clouds is used to survey many different settings such as construction sites, building façades, industrial facilities, interior of houses, or any other applicable setting. The surveys achieved therewith may be used to obtain accurate three-dimensional (3D) models of a setting, wherein the models comprise point clouds. The points of such a cloud are stored as coordinates in a coordinate system, which may be defined by a surveying instrument which recorded the point cloud. Usually, the surveying instrument constitutes the origin of the coordinate system by an instrument centre, in particular by the so-called nodal point of the surveying instrument. The points are usually surveyed by associating a distance measured with a laser beam (with help of a time-of-flight method) with the alignment under which the distance was measured. Usually, the coordinate system is a spherical coordinate system, such that a point is characterised by a distance value, an elevation angle and an azimuth angle with reference to the origin of the coordinate system. Common surveying instruments comprise a unit for sending out a scanning beam and for receiving the reflected beam in order to measure the distance of a point the beam was directed at. Usually, these surveying instruments furthermore comprise means to rotatably alter the direction of the beams, commonly a vertical rotation axis and a horizontal rotation axis, wherein both axes are sensed with angle sensors. Usually, the rotation of the vertical axis is measured by an azimuth angle and the rotation of the horizontal axis is measured by an elevation angle. If the surveying instrument is embodied as a laser scanner, one of said axes may be a slow axis and the other one a fast axis. The distances may be calculated with the travel time measurement (time-of-flight) method by observing the time between sending out and receiving a signal. The alignment angles are achieved with said angle sensors arranged at the vertical axis and at the horizontal axis. In the field of surveying, providing colourised 3D point clouds is a desired feature, e.g. for LiDAR based surveying tools. Colour features facilitate the understanding and navigation through the scene, as well as the identification of elements of interest, thereby providing a more "friendly" product to the human vision system, than uncoloured point clouds. In addition, colour features are widely used as input features for many state-of-the-art point cloud post-processing algorithms like segmentation, classification and/or modelling algorithms. Also, the calibration and projection of 3D data to an image is a desired feature, as this create a "metric" image where certain measurements can be executed directly in the image. In order to provide a better visualization, the point cloud may be digitally colourised. In various applications, to provide the colour information used for colourizing the point cloud, terrestrial surveying is hence supported by imaging data of at least one calibrated imaging sensor, e.g. a camera, which is combined with a surveying instrument by including the camera in the instrument or mounting it on the same platform as the instrument. Those imaging sensors are integrated or attached to the LiDAR measuring system with accurate intrinsic and extrinsic camera calibration, so that both the features acquired by the imaging sensor can be projected/mapped to the 3D LiDAR point cloud, and vice versa. Devices that are configured to generate a digital three-dimensional representation of an environment by capturing 3D data simultaneously with panoramic images of the environment are also known as "reality capture devices". WO 2020/126123 A2 discloses such a reality capture device having a laser scanner and a plurality of RGB cameras. EP 4 095 561 A1 discloses a reality capture device combining a plurality of time-of-flight cameras for capturing the 3D data with a plurality of RGB cameras. Some LiDAR measurement systems integrate simultaneous location and mapping (SLAM) and/or additional positioning technologies that enable dynamic use of those devices. This way, for measuring and reconstructing the 3D scene along a dynamic trajectory the LiDAR can be carried by an operator or be mounted to a transport platform, e.g. an unmanned ground vehicle (UGV). While LiDAR measurement techniques are robust to changes both in locat