CN-121999049-A - Building decoration assembly system and method based on image point cloud processing

CN121999049ACN 121999049 ACN121999049 ACN 121999049ACN-121999049-A

Abstract

The invention relates to the technical field of building automation and machine vision, and particularly discloses a building decoration assembly system and method based on image point cloud processing. The system performs semantic labeling and feature enhancement on a three-dimensional point cloud through an image semantic segmentation map to generate a semantic enhancement point cloud, performs iterative registration based on semantic consistency on a decoration component digital model and the semantic enhancement point cloud, outputs an accurate installation pose, plans a collision-free assembly track according to the pose and the semantic segmentation map, drives a robot to perform assembly along the track, and performs pose deviation detection and dynamic compensation based on a real-time image and the point cloud. The invention realizes the depth penetration and unified driving of the image semantic information in the whole process of environment perception, pose calculation, path planning and closed-loop execution, and effectively solves the technical problems of insufficient precision, poor adaptability and dependence on manual intervention caused by the uncoupling of the perception, planning and execution links of the traditional assembly system.

Inventors

HE LUE
Sheng Yufu
HE QIJUN
LIN YONGTONG
CHEN YE

Assignees

广东新六维建筑装饰有限公司

Dates

Publication Date: 20260508
Application Date: 20260211

Claims (10)

1. Architectural decoration assembly system based on image point cloud processing, characterized by comprising: The scene understanding module is used for carrying out pixel-level image semantic segmentation on the color images acquired at the construction site and outputting an image semantic segmentation map containing structural base planes, preset connection points and site obstacle category information; The fusion registration module is used for receiving the image semantic segmentation map and the construction site three-dimensional point cloud, carrying out semantic annotation and feature enhancement on the construction site three-dimensional point cloud according to the image semantic segmentation map, generating a semantic enhancement point cloud, carrying out iterative registration based on semantic consistency on the decoration component digital model and the semantic enhancement point cloud, and outputting the accurate installation pose of the decoration component; The intelligent planning module is used for planning a collision-free assembly track from the initial position to the accurate installation pose of the decorative component according to the accurate installation pose and the image semantic segmentation graph; The closed loop execution module is used for driving the robot to execute assembly operation along the collision-free assembly track, and carrying out pose deviation detection and dynamic motion compensation based on the image and the point cloud acquired in real time in the assembly process.
2. The architectural ornament assembly system based on image point cloud processing of claim 1, wherein the scene understanding module comprises: The multi-scale feature coding unit is used for extracting multi-level apparent features from the color image by adopting a convolutional neural network; and the context perception decoding unit is used for fusing and upsampling the multi-level apparent features through the cavity convolution and the feature pyramid structure and outputting a pixel-level image semantic segmentation graph.
3. The architectural ornament assembly system based on image point cloud processing of claim 1, wherein the fusion registration module comprises: The point cloud semantic projection unit is used for projecting each point in the three-dimensional point cloud of the construction site to the corresponding pixel position of the image semantic segmentation map, endowing each point with a semantic category label of the corresponding pixel position, and generating an initial point cloud with the semantic label; The semantic region point cloud extraction unit is used for extracting point cloud data of semantic category labels belonging to a structural base plane or preset connection points from initial point clouds with semantic labels to form semantic pure point clouds; the convolution feature extraction unit is used for extracting multi-scale convolution features of pixel positions corresponding to each point in the semantically pure point cloud from the color image based on the projection mapping relation generated by the point cloud semantic projection unit; the cross-modal feature fusion unit is used for carrying out cross-modal feature fusion on the multi-scale convolution features and the depth geometric features extracted through the point cloud convolution neural network to generate enhanced depth feature vectors; the point cloud depth feature enhancement unit is used for generating semantic enhancement point clouds based on the enhanced depth feature vectors; The semantic constraint registration unit is used for constructing an optimization objective function in the registration process of the decoration member digital model point cloud and the semantic enhancement point cloud, punishing the distance between the point pairs with the unmatched semantic categories based on the optimization objective function, enabling the point pairs with the consistent semantic meanings to be aligned preferentially through iterative solution, and outputting accurate installation pose.
4. A building decoration assembly system based on image point cloud processing as claimed in claim 3, wherein the semantic constraint registration unit comprises: the point cloud feature calculation subunit is used for acquiring pre-calculation semantic confidence coefficient of each point in the semantic enhancement point cloud in the iterative registration process and dynamically calculating the corresponding local geometric feature significance; The weight distribution subunit is used for distributing a dynamic matching weight for each potential point pair between the semantic enhancement point cloud and the decoration component digital model point cloud based on the semantic confidence and the geometric feature significance, wherein the point pair with high semantic confidence and the geometric feature significance obtains a higher matching weight; The semantic constraint registration unit is used for constructing an optimized objective function, and when calculating the point pair distance, dynamic matching weights are used for weighting; And the pose output subunit is used for outputting the accurate installation pose through the optimization objective function after iterative optimization weighting.
5. The architectural ornament assembly system based on image point cloud processing of claim 1, wherein the intelligent planning module comprises: The semantic cost map construction unit is used for rasterizing the image semantic segmentation map, distributing low pass cost for the structural basal plane grid, distributing negative cost for the preset connection point grid, distributing infinite pass cost for the field obstacle grid and generating a semantic cost map; The global path searching unit is used for running a path searching algorithm integrating a passing cost heuristic function on a semantic cost map by taking the current position of the robot as a starting point and the position corresponding to the accurate installation pose as an end point to generate a global path point sequence; The track optimizing and generating unit is used for carrying out smoothing processing on the global path point sequence and interpolating to generate a collision-free assembly track conforming to the robot kinematics constraint.
6. The architectural ornament assembly system based on image point cloud processing of claim 1, wherein the closed loop execution module comprises: the assembly process sensing unit is used for acquiring partial images and partial point clouds of the component and the mounting surface in real time through a visual sensor on the end effector of the robot in the process of executing assembly by the robot; the pose deviation detection unit is used for comparing the local image and the local point cloud with an expected image and an expected point cloud under the accurate installation pose respectively, and calculating the position deviation and the angle deviation of the decorative member in the three-dimensional space; and the dynamic motion compensation unit is used for correcting the motion instruction of the robot end effector in real time according to the deviation vector when the position deviation or the angle deviation exceeds the allowable threshold value, so as to realize the dynamic compensation in the assembly process.
7. The architectural ornament assembly system based on image point cloud processing of claim 6, wherein the pose deviation detection unit comprises: The image deviation resolving operator unit is used for parallelly executing pose resolving based on feature point matching, pose resolving based on dense optical flow field and pose resolving based on deep learning pose regression network on the local image acquired in real time and the expected image under the accurate installation pose to respectively obtain a first image deviation, a second image deviation and a third image deviation; The point cloud deviation resolving operator unit is used for parallelly executing pose resolving based on feature descriptor matching and pose resolving based on a global point cloud registration algorithm on the local point cloud acquired in real time and the expected point cloud under the accurate installation pose to respectively obtain a first point cloud deviation and a second point cloud deviation; A deviation set setting subunit, configured to set an image deviation set and a point cloud deviation set, where the image deviation set includes a first image deviation, a second image deviation, and a third image deviation, and the point cloud deviation set includes a first point cloud deviation and a second point cloud deviation; a consistency calculation subunit, configured to calculate internal consistency scores between the image deviation set and each set of deviation data in the point cloud deviation set respectively; The deviation selecting subunit is used for selecting the image deviation with the highest internal consistency score in the image deviation set as the preferred image deviation and selecting the point cloud deviation with the highest internal consistency score in the point cloud deviation set as the preferred point cloud deviation; And the deviation fusion subunit is used for carrying out weighted fusion on the preferred image deviation and the preferred point cloud deviation and outputting the comprehensive position deviation and the comprehensive angle deviation which are finally used for dynamic motion compensation.
8. The architectural ornament assembly system based on image point cloud processing of claim 1, further comprising an assembly quality assessment module comprising: The finished image acquisition unit is used for acquiring high-definition images of the splice joint areas of the decorative components and the peripheral structures after the decorative components are assembled; the defect image detection unit is used for analyzing the high-definition image of the splicing seam area by adopting a deep learning target detection model, identifying and positioning cracks, dislocation and pollution defects in the image, and obtaining the types, the quantity and the positions of the defects; And the quality evaluation report generation unit is used for automatically generating an assembly quality quantitative evaluation report based on the identified defect types, the number and the positions.
9. The architectural ornament assembly system based on image point cloud processing of claim 1, further comprising a multi-component co-scheduling module comprising: the assembly dependency graph construction unit is used for constructing an assembly dependency graph describing space and sequence constraint among the components based on the accurate installation pose and the installation process logic of all the decoration components to be assembled; The assembly sequence optimizing unit is used for optimizing the assembly sequence of all the decorative components by adopting a heuristic search algorithm based on the assembly dependency graph to generate a global assembly sequence with the shortest total time consumption; the resource conflict coordination unit is used for pre-judging and coordinating the space-time conflict of the robot, the materials and the station resources based on the global assembly sequence and the collision-free assembly track of each decoration component, and generating a resource scheduling instruction.
10. The building decoration assembly method based on the image point cloud processing is characterized by comprising the following steps of: performing pixel-level image semantic segmentation on a color image acquired at a construction site, and outputting an image semantic segmentation map containing structural base planes, preset connection points and site obstacle category information; Receiving an image semantic segmentation map and a construction site three-dimensional point cloud, carrying out semantic annotation and feature enhancement on the construction site three-dimensional point cloud according to the image semantic segmentation map, generating a semantic enhancement point cloud, carrying out iterative registration based on semantic consistency on a decoration component digital model and the semantic enhancement point cloud, and outputting an accurate installation pose of a decoration component; Planning a collision-free assembly track from an initial position to an accurate mounting pose of the decorative component according to the accurate mounting pose and the image semantic segmentation map; And driving the robot to execute assembly operation along the collision-free assembly track, and performing pose deviation detection and dynamic motion compensation based on the real-time acquired images and the point cloud in the assembly process.

Description

Building decoration assembly system and method based on image point cloud processing Technical Field The invention relates to the technical field of building automation and machine vision, in particular to a building decoration assembly system and method based on image point cloud processing. Background In the field of building decoration engineering, automatic and intelligent assembly is an important direction for improving construction quality and efficiency. Currently, automated assembly systems combining robotics with three-dimensional vision (e.g., lidar, depth cameras) have been used. Such systems typically work based on point cloud data by first determining component mounting locations through point cloud registration, then performing path planning and controlling the robot to perform grasping and mounting. To enhance the context awareness, some systems introduce two-dimensional image recognition techniques to assist in identifying specific objects or to perform coarse localization. However, in the prior art, image recognition and point cloud processing are mostly used as independent processes which are connected in front and back, so that deep fusion and unified understanding of two modal information at a characteristic level cannot be realized. The technical route has the fundamental defect that the understanding of the system to the assembly environment is unilateral and rigid because the abundant semantic information of the image and the accurate geometric information of the point cloud are in a splitting state. The three problems are that firstly, in a complex and unstructured construction site, a point cloud registration method which only depends on geometric features is easy to be interfered by a similar structure, the calculation accuracy of the installation pose is insufficient and the robustness is poor, secondly, path planning lacks understanding of scene function attributes (such as where an installation base surface is and where the path planning is a dangerous area), only geometric obstacle avoidance can be carried out, the planned path is often not in line with process logic or low efficiency, thirdly, the assembly execution process is open-loop, the online quality checking and correcting capability based on real-time semantic perception is lacking, and once deviation or environmental change occurs, assembly failure is extremely easy to be caused. Therefore, an integrated intelligent assembly system capable of realizing the deep fusion of image semantics and point cloud geometric information and driving the whole process from accurate positioning, intelligent planning to closed-loop execution is urgently needed. The invention provides a building decoration assembly system and a building decoration assembly method based on image point cloud processing. Disclosure of Invention The invention provides a building decoration assembly system and a method based on image point cloud processing, which solve the fundamental defects of low precision, poor adaptability and dependence on manual intervention caused by uncoupling of sensing, planning and executing links in the traditional assembly technology by constructing a full-flow closed-loop system taking image semantic information as a unified driving core. The invention provides a building decoration assembly system based on image point cloud processing, which comprises: The scene understanding module is used for carrying out pixel-level image semantic segmentation on the color images acquired at the construction site and outputting an image semantic segmentation map containing structural base planes, preset connection points and site obstacle category information; The fusion registration module is used for receiving the image semantic segmentation map and the construction site three-dimensional point cloud, carrying out semantic annotation and feature enhancement on the construction site three-dimensional point cloud according to the image semantic segmentation map, generating a semantic enhancement point cloud, carrying out iterative registration based on semantic consistency on the decoration component digital model and the semantic enhancement point cloud, and outputting the accurate installation pose of the decoration component; The intelligent planning module is used for planning a collision-free assembly track from the initial position to the accurate installation pose of the decorative component according to the accurate installation pose and the image semantic segmentation graph; The closed loop execution module is used for driving the robot to execute assembly operation along the collision-free assembly track, and carrying out pose deviation detection and dynamic motion compensation based on the image and the point cloud acquired in real time in the assembly process. Preferably, the scene understanding module includes: The multi-scale feature coding unit is used for extracting multi-level apparent features from the color image by ado