Search

EP-3985615-B1 - POINT CLOUD DATA PROCESSING DEVICE, POINT CLOUD DATA PROCESSING METHOD, AND PROGRAM

EP3985615B1EP 3985615 B1EP3985615 B1EP 3985615B1EP-3985615-B1

Inventors

  • IWAMI KAZUCHIKA

Dates

Publication Date
20260506
Application Date
20200514

Claims (12)

  1. A point cloud data processing apparatus (1) comprising: an image data acquisition unit (13) that acquires image data (5) of an object; a point cloud data acquisition unit (15) that acquires point cloud data (7) representing pieces of three-dimensional information of a number of points (Q) on an outer surface of said object, wherein the image data acquired by the image data acquisition unit (13) and the point cloud data acquired by the point cloud data acquisition unit (15) have a corresponding positional relationship to each other, a positional relationship between the image data acquisition unit (13) and the point cloud data acquisition unit (15) is predetermined; a recognition unit (23) that recognizes the object on the basis of the image data, and acquires a region of the object in the image data and attribute information for identifying the object, the attribute information being form information or name of the object for distinguishing the object from another object; and an attribute assigning unit (25) that selects, from the point cloud data, point cloud data that belongs to the region of the object, and assigns the acquired attribute information to the selected point cloud data.
  2. The point cloud data processing apparatus (1) according to claim 1, wherein the recognition unit (23) acquires, in a case where the image data includes pieces of image data of a plurality of objects, a region of each object among the objects and attribute information for identifying the object on a per object basis, and the attribute assigning unit (25) selects, from the point cloud data, point cloud data that belongs to the region of each object among the objects on a per object basis, and assigns the attribute information of the object to the point cloud data selected on a per object basis.
  3. The point cloud data processing apparatus (1) according to claim 1 or 2, wherein the recognition unit (23) recognizes, in a case where the image data includes image data of the object having a plurality of partial regions, a region of each partial region among the partial regions, and acquires the region of the partial region and attribute information for identifying the partial region, and the attribute assigning unit (25) selects, from the point cloud data, point cloud data that belongs to the region of each partial region among the partial regions on a per partial region basis, and assigns the attribute information of the partial region to the point cloud data selected on a per partial region basis.
  4. The point cloud data processing apparatus (1) according to any one of claims 1 to 3, further comprising a point cloud data complementing unit (27) that detects a missing part of the point cloud data that belongs to the region of the object on the basis of at least the region of the object recognized by the recognition unit (23), and interpolates point cloud data of the detected missing part, wherein the attribute assigning unit (25) assigns the attribute information to the complemented point cloud data.
  5. The point cloud data processing apparatus (1) according to any one of claims 1 to 4, wherein the image data acquisition unit (13) acquires a captured image acquired by image capturing of the object as the image data.
  6. The point cloud data processing apparatus (1) according to any one of claims 1 to 4, further comprising: an image data generation unit (35) that generates the image data on the basis of the point cloud data, wherein the image data acquisition unit (13) acquires the image data generated by the image data generation unit.
  7. The point cloud data processing apparatus (1) according to any one of claims 1 to 6, wherein the recognition unit (23) is formed of a recognizer subjected to machine learning or deep learning.
  8. The point cloud data processing apparatus (1) according to any one of claims 1 to 7, further comprising a three-dimensional model generation unit (29) that generates a three-dimensional model of the object on the basis of the point cloud data that is assigned the attribute information.
  9. The point cloud data processing apparatus (1) according to any one of claims 1 to 8, wherein the image data and the point cloud data are respectively acquired by devices having the same optical axis.
  10. The point cloud data processing apparatus (1) according to any one of claims 1 to 9, wherein the image data and the point cloud data are respectively acquired by devices for which a positional relationship between the devices is known.
  11. A point cloud data processing method comprising: a step of acquiring image data (5) of an object; a step of acquiring point cloud data representing pieces of three-dimensional information of a number of points (Q) on an outer surface of said object, wherein the image data acquired by an image data acquisition unit (13) and the point cloud data acquired by a point cloud data acquisition unit (15) have a corresponding positional relationship to each other, a positional relationship between the image data acquisition unit (13) and the point cloud data acquisition unit (15) is predetermined; a step of recognizing the object on the basis of the image data, and acquiring a region of the object in the image data, and attribute information for identifying the object, the attribute information being form information or name of the object for distinguishing the object from another object; and a step of selecting, from the point cloud data, point cloud data that belongs to the region of the object, and assigning the acquired attribute information to the selected point cloud data.
  12. A program for causing a computer to perform a point cloud data process comprising: a step of acquiring image data (5) of an object; a step of acquiring point cloud data representing pieces of three-dimensional information of a number of points on an outer surface of said object, wherein the image data acquired by an image data acquisition unit (13) and the point cloud data acquired by a point cloud data acquisition unit (15) have a corresponding positional relationship to each other, a positional relationship between the image data acquisition unit (13) and the point cloud data acquisition unit (15) is predetermined; a step of recognizing the object on the basis of the image data, and acquiring a region of the object in the image data and attribute information for identifying the object, the attribute information being form information or name of the object for distinguishing the object from another object; and a step of selecting, from the point cloud data, point cloud data that belongs to the region of the object, and assigning the acquired attribute information to the selected point cloud data.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a point cloud data processing apparatus, a point cloud data processing method, and a program and specifically relates to a technique for classifying point cloud data and assigning attribute information. 2. Description of the Related Art A technique using, for example, a laser scanner is known in which reflection on the surface of an object is used to acquire point cloud data constituted by a large number of points representing three-dimensional information of the surface of the object. The acquired point cloud data is used in various forms. For example, when the points that constitute the acquired point cloud data are grouped on a per object basis, a three-dimensional model of the object can be generated from the point cloud data. JP2013-97489A describes a technique in which point cloud data of an object is acquired, and a two-dimensional geometric shape of the object created in advance is used to perform grouping. US 2019/087976 A1 discloses a system for determining gripping positions on objects using a 3D imaging device that captures distance images. An image processing unit generates CAD models and trains detection and segmentation models based on 3D data. Gripping locations are identified using these models. LAHOUD JEAN ET AL: "2D-Driven 3D Object Detection in RGB-D Images" discloses a method for 3D object detection using RGB-D images, where 2D object detection in RGB images is used to define 3D frustums, reducing the search space for 3D bounding box fitting in the point cloud. Classification is performed at the object level using features from both 2D and 3D data, but no per-point classification or assignment of attributes based on 2D image recognition is disclosed. CHARLES R QI ET AL: "Frustum PointNets for 3D Object Detection from RGB-D Data" discloses PointNet, a deep learning architecture that directly processes 3D point cloud data for object classification and semantic segmentation. It operates solely on 3D data without using 2D image input. Classification is based on learned geometric features, and point-wise labels are predicted, but no attribute information is assigned based on 2D image data. SUMMARY OF THE INVENTION In JP2013-97489A, the two-dimensional geometric shape of the target object is used to group the point cloud; however, it is not always the case that the two-dimensional geometric shape of the target object is available to the user. For example, there may be a case where the two-dimensional geometric shape of the object is not present and a case where although the two-dimensional geometric shape of the object is present, the form of the object has been changed before point cloud data of the object is acquired and there is a difference between the two-dimensional geometric shape and the object. The present invention has been made in view of the above-described circumstances, and an object thereof is to provide a point cloud data processing apparatus, a point cloud data processing method, and a program for enabling easy and efficient grouping of point cloud data without using a two-dimensional geometric shape of the target object. The invention is set out in the appended set of claims. According to the present invention, a region of an object and attribute information for identifying the object are acquired on the basis of image data, and point cloud data that belongs to the region of the identified object is selected and assigned the attribute information. Therefore, even in a case where a two-dimensional geometric shape of the object is not prepared in advance, it is possible to classify the point cloud data easily and efficiently by using the acquired image data of the object and assign the attribute information. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a schematic diagram illustrating a form in which image data and point cloud data are acquired;Fig. 2 is a schematic diagram for explaining a laser scanner and an image capturing device mounted in a three-dimensional measuring device;Fig. 3 is a schematic diagram for explaining that a captured image and point cloud data have a corresponding positional relationship;Fig. 4 is a block diagram illustrating an example functional configuration of a point cloud data processing apparatus;Figs. 5A and 5B are diagrams for explaining recognition of an object A by a recognition unit performing segmentation;Fig. 6 is a diagram for explaining a point Q that is assigned attribute information;Fig. 7 is a flowchart for explaining a point cloud data processing method using the point cloud data processing apparatus;Fig. 8 is a block diagram illustrating an example functional configuration of the point cloud data processing apparatus;Figs. 9A and 9B are diagrams for explaining complementing of point cloud data;Fig. 10 is a block diagram illustrating an example functional configuration of the point cloud data processing apparatus; andFig. 11 is a diagram illustratin