Search

EP-4672169-B1 - MATERIAL SEGMENTATION METHOD AND SYSTEM FOR BUILDING COLLAPSE SCENE

EP4672169B1EP 4672169 B1EP4672169 B1EP 4672169B1EP-4672169-B1

Inventors

  • LIANG, Kang
  • KONG, Yanan
  • MA, HOUXUE

Dates

Publication Date
20260513
Application Date
20240926

Claims (9)

  1. A material segmentation method for building collapse scenarios, comprising: acquiring a target building collapse scenario image to be segmented; and inputting the target building collapse scenario image to be segmented into a trained building collapse scenario material segmentation model to obtain a material segmentation result; wherein the acquisition of the trained building collapse scenario material segmentation model comprises: training a first deep semantic segmentation model using a first road scenario material data set to obtain a road scenario material segmentation model, the first road scenario material data set comprising a plurality of road scenario material segmentation images and corresponding material labels; processing a road scenario semantic segmentation data set using the road scenario material segmentation model to obtain a labeled road scenario material segmentation image; adding the labeled road scenario material segmentation image into the first road scenario material data set to obtain a second road scenario material data set; mapping road scenario material segmentation images in the second road scenario material data set to obtain a building scenario material data set; adding collapsed area material objects and corresponding material labels to an image of the building scenario material data set to obtain a building collapse scenario material data set; training a second deep semantic segmentation model using the building collapse scenario material data set to obtain a basic building collapse scenario material segmentation model; and performing domain adaptive learning on the basic building collapse scenario material segmentation model using labeled images in target scenario sequence images to obtain the trained building collapse scenario material segmentation model; wherein the first deep semantic segmentation model and the second deep semantic segmentation model are SegFormer models; and/or the first road scenario material data set is a McubeS data set; and/or the road scenario semantic segmentation data set is selected from at least one of a CamVid data set, a CityScape data set and a Kitti data set; and wherein mapping road scenario material segmentation images in the second road scenario material data set to obtain a building scenario material data set comprises: for each road scenario material segmentation image in the second road scenario material data set, transforming building scenarios on both sides into front-view building scenarios by perspective projection transformation, to obtain labeled front-view building images that constitute the building scenario material data set.
  2. The material segmentation method for building collapse scenarios according to claim 1, wherein performing domain adaptive learning on the basic building collapse scenario material segmentation model using labeled images in target scenario sequence images comprises: selecting typical images from the target scenario sequence images for labeling and data enhancement to obtain a prototype image; according to the prototype image, retrieving images with similar scenario structures from the building collapse scenario material data set to form a domain migration training set; performing fine-tuning training on the basic building collapse scenario material segmentation model using the domain migration training set to obtain an advanced model; and iteratively executing loop steps until a preset condition is met, to obtain the trained building collapse scenario material segmentation model, the loop steps comprising: processing the target scenario sequence images using the advanced model to obtain a labeled target scenario image; retrieving images with similar scenario structures from the building collapse scenario material data set according to the target scenario image to obtain an updated domain migration training set; and performing fine-tuning training on the advanced model using the updated domain migration training set to obtain an updated advanced model.
  3. The material segmentation method for building collapse scenarios according to claim 2, wherein the preset condition is to iteratively execute the loop steps 3 to 5 times.
  4. The material segmentation method for building collapse scenarios according to claim 1, wherein adding collapsed area material objects and corresponding material labels to an image of the building scenario material data set to obtain a building collapse scenario material data set comprises: constructing a collapsed area on the image of the building scenario material data set, the collapsed area referring to parts that remain upright following a natural collapse or human demolition of a building; using a texture data set to paste irregular texture block images on a building area where an image of a collapsed area is constructed, the texture block image comprising at least one of bricks, rubble, adobe, steel bars and window frames; and using a COCO data set to randomly superimpose material objects on remaining parts of the building, simulating internal objects that are revealed after the collapse of external walls of the building.
  5. The material segmentation method for building collapse scenarios according to claim 4, wherein constructing a collapsed area on the image of the building scenario material data set comprises: acquiring a building bounding rectangle of the image of the building scenario material data set, selecting a point from any two of left, top, and right sides of the building bounding rectangle to form a connecting line or a broken line, simulating a projection line of a cross-section on the image, and setting an area defined by the connecting line or the broken line and three sides of the building bounding rectangle to be white or sky blue to simulate a building cut in the building scenario; alternatively, starting from an upper edge of a building image area, randomly using structural elements of different sizes and shapes to perform morphological erosion on continuous edge points of the building image area, and replacing eroded areas with an average pixel value of a surrounding area of the building.
  6. The material segmentation method for building collapse scenarios according to claim 2, wherein the data enhancement comprises: performing at least one of cropping, flipping, translating and brightness and color transformation on the typical image; and adding collapsed area material objects and corresponding material labels to the typical image: obtaining images of partially collapsed buildings using truncation and erosion operations, then superimposing texture block images and material objects.
  7. The material segmentation method for building collapse scenarios according to claim 2, wherein retrieving images with similar scenario structures from the building collapse scenario material data set comprises: calculating mIoU of a labeled image of the prototype image/target scenario image and a labeled image of each picture in the building collapse scenario material data set, mIoU being a mean intersection over union for each material category; and taking the top 1% of images from the building collapse scenario material data set with maximum mIoU values as the images with similar scenario structures.
  8. A material segmentation system for building collapse scenarios, comprising a memory and a processor, wherein the memory is used for storing instructions, and the instructions are used for controlling the processor to implement the material segmentation method for building collapse scenarios according to any one of claims 1 to 7.
  9. A demolition machine, comprising the material segmentation system for building collapse scenarios according to claim 8.

Description

TECHNICAL FIELD The invention belongs to the technical field of demolition machinery, and relates to a material segmentation method and system for building collapse scenarios. BACKGROUND During demolition operations by demolition machinery, different drill bits are used according to the material of the object being worked on, and working parameters need to be adjusted accordingly. Currently, most demolition machinery is operated via manual remote control, which largely relies on human judgment and requires long-term supervision. In scenarios where the materials are diverse and frequent changes of drill bits and adjustment of parameters are needed, such as in building demolition, the automation level of demolition machinery is significantly limited, making it impossible to realize fully intelligent automatic control. With the further improvement of technologies such as artificial intelligence, computer vision, and deep learning algorithms, the intelligent and automated operation of demolition machinery becomes an essential trend. Thus, there is a need for a material segmentation method and system that is suitable for demolition machinery and tailored for building collapse scenarios. Liang Yupeng et al ("Multimodal Material Segmentation", 2022 IEEE/CVF Conference On Computer Vision And Pattern Recognition (CVPR), IEEE, 18 June 2022) disclose a new dataset and novel method for multimodal material segmentation. The new MCubeS dataset consists of 500 sets of RGB, polarization, and NIR images of outdoor scenes captured from a vantage point similar to a car. Bang Seongdeok et al ("Image Augmentation To Improve Construction Resource Detection Using Generative Adversarial Networks, Cut-And-Paste, And Image Transformation Techniques", Automation In Construction, Elsevier, Amsterdam, NL, 24 March 2020) disclose an image augmentation method to construct a large-size dataset for improving construction resource detection. The method consists of three techniques: removing-and-inpainting, cut-and-paste, and image-variation. The removing-and-inpainting technique arbitrarily removes objects from images and re-constructs the removed regions via generative adversarial networks (GAN). The cut-and-paste technique extracts objects from the original dataset and places them into the reconstructed images via the previous technique. The image-variation technique applies three image transformation techniques, intensity-, blur- and scale-variation, to the images. Zhang Jiaxin et al ("Automatic generation of synthetic datasets from a city digital twin for use in the instance segmentation of building facades", Journal of Computational Design and Engineering, 26 August 2022) discloses a system that can auto-generate synthetic datasets from a CDT for the instance segmentation of building facades. The system can produce synthetic images of street views from multiple viewpoints under different atmospheric effects and also generate pixel-level instance annotation for synthetic building facades. SUMMARY To address the difficulties in sample labeling and the weak scenario adaptability encountered in material area segmentation using deep learning technology, the invention constructs samples from building collapse scenarios and employs the model domain adaptive learning method to improve the efficiency of model construction and the accuracy of material segmentation, effectively improving the precision and efficiency of demolition operations. This technology can be widely applied in fields such as mining, tunneling, building demolition, cement industry, and rescue operations, ensuring the safety of personnel in harsh working environments and significantly increasing work efficiency. Objective: To overcome the shortcomings in the prior art, the invention provides a material segmentation method and system for building collapse scenarios, involving the control of an electric rotary system on both horizontal and inclined surfaces. Technical Scheme: In order to solve the above technical problems, the technical scheme adopted by the invention is as follows. In a first aspect, a material segmentation method for building collapse scenarios is provided as defined by the appended independent claim 1. In some embodiments, performing domain adaptive learning on the basic building collapse scenario material segmentation model using labeled images in target scenario sequence images comprises: selecting typical images from the target scenario sequence images for labeling and data enhancement to obtain a prototype image; according to the prototype image, retrieving images with similar scenario structures from the building collapse scenario material data set to form a domain migration training set;performing fine-tuning training on the basic building collapse scenario material segmentation model using the domain migration training set to obtain an advanced model; anditeratively executing loop steps until a preset condition is met, to obtain the trained building co