US-12620137-B2 - Systems and method for adaptive filtering in patch generation for video-based point cloud coding

Abstract

Provided are a method, apparatus, and non-transitory computer readable medium for adaptive filtering in patch generation for video-based point cloud coding. The method includes projecting points included in a connected component (CC) to a projection plane associated with the CC, and removing one or more outlier points to generate an updated CC; determining a first surface by selecting one or more points associated with the updated CC; adaptively filtering the one or more points selected for the first surface, by adjusting a grid size parameter based on one or more characteristics of the CC; determining a second surface by selecting one or more points associated with the updated CC; adding a patch corresponding to the filtered first surface and the second surface to a list of patches associated with an original point cloud; and coding the original point cloud based on the list of patches.

Inventors

• Wen Gao
• Xiang Zhang
• Shan Liu

Assignees

• Tencent America LLC

Dates

Publication Date

20260505

Application Date

20221108

Claims (19)

1. 1 . A method comprising: projecting a plurality of points included in a connected component (CC) onto a projection plane associated with the CC; removing one or more outlier points from the plurality of points based on the projected points to generate an updated CC; determining a first surface by selecting one or more points associated with the updated CC, based on the projected points and a first selection criteria; adaptively filtering the one or more points selected for the first surface, by adjusting a grid size parameter based on one or more characteristics of the CC including at least one of statistics associated with a depth of the points in the CC, statistics associated with a geometry of the points in the CC, statistics associated with a color of the points in the CC, and statistics associated with other attributes of the points in the CC; determining a second surface by selecting one or more points associated with the updated CC, based on the projected points and a second selection criteria; adding a patch corresponding to the filtered points for the first surface and the second surface to a list of patches associated with an original point cloud; and coding the original point cloud based on the list of patches.
2. 2 . The method of claim 1 , wherein the first surface is a near surface, the second surface is a far surface, and the first selection criteria comprises selecting one point having a smallest depth among two or more points associated with the updated CC, when the two or more points have a same projected coordinate.
3. 3 . The method of claim 1 , wherein the first surface is a far surface, the second surface is a near surface, and the first selection criteria comprises selecting one point having a largest depth among two or more points associated with the updated CC, when the two or more points have a same projected coordinate.
4. 4 . The method of claim 1 , wherein the adaptively filtering comprises: determining an adjustment value based on the one or more characteristics of the CC; adjusting the grid size parameter based on the adjustment value; and filtering the one or more points selected for the first surface using a grid in the projection plane associated with the CC, wherein the grid is based on the adjusted grid parameter.
5. 5 . The method of claim 4 , wherein the adjustment value is based on an adjustment threshold value and a range between a smallest depth value and a largest depth value associated with the plurality of points included in the CC.
6. 6 . The method of claim 4 , wherein the adjustment value is based on a plurality of adjustment threshold values and a range between a smallest depth value and a largest depth value associated with the plurality of points included in the CC.
7. 7 . The method of claim 1 , further comprising: generating a list of CCs, each CC associated with the projection plane, and including a raw point from the original point cloud and at least one neighbor point that neighbors the raw point in the original point cloud, such that the raw point and the at least one neighbor point have the same projection plane; projecting the plurality of points included in each CC onto the projection plane associated with the CC, and removing one or more outlier points from the plurality of points based on the projected points, to generate a list of updated CC; determining the first surface for each CC in the list of updated CC by selecting one or more points associated with the updated CC, based on the projection of the one or more points and the first selection criteria; adaptively filtering the one or more points selected for each first surface, by adjusting the grid size parameter based on one or more characteristics of the CC; determining the second surface for each CC in the list of updated CC by selecting one or more points associated with the updated CC, based on the projected points and the second selection criteria; and adding a patch corresponding to the filtered points for each first surface and each second surface to the list of patches associated with the original point cloud.
8. 8 . The method of claim 7 , wherein when a CC among the list of CCs is associated with a number of points that is less than a threshold number, then the CC is removed from the list of CCs.
9. 9 . The method of claim 7 , wherein coding the original point cloud based on the list of patches comprises: identifying a plurality of points associated with a filtered first surface and a second surface corresponding to each patch in the list of patches; reconstructing a point cloud based on the identified plurality of points; building a K-D tree using the reconstructed point cloud; searching the K-D tree for a nearest neighbor point to each point in the original point cloud; and classifying each point in the original point cloud as a raw point when a distance between the point and the nearest neighbor point is greater than a distance threshold.
10. 10 . An apparatus comprising: a memory storing program code; and at least one processor configured to execute the program code and operate as instructed by the program code, the program code including: projecting code configured to cause at least one of the at least one processor to project a plurality of points included in a connected component (CC) onto a projection plane associated with the CC; removing code configured to cause at least one of the at least one processor to remove one or more outlier points from the plurality of points based on the projected points to generate an updated CC; determining code configured to cause at least one of the at least one processor to determine a first surface by selecting one or more points associated with the updated CC, based on the projection of the one or more points and a first selection criteria; filtering code configured to cause at least one of the at least one processor to adaptively filter the one or more points selected for the first surface, by adjusting a grid size parameter based on one or more characteristics of the CC including at least one of statistics associated with a depth of the points in the CC, statistics associated with a geometry of the points in the CC, statistics associated with a color of the points in the CC, and statistics associated with other attributes of the points in the CC; determining code configured to cause at least one of the at least one processor to determine a second surface by selecting one or more points associated with the updated CC, based on the projected points and a second selection criteria; adding code configured to cause at least one of the at least one processor to add a patch corresponding to the filtered points for the first surface and the second surface to a list of patches associated with an original point cloud; and coding code configured to cause at least one of the at least one processor to code the original point cloud based on the list of patches.
11. 11 . The apparatus of claim 10 , wherein the first surface is a near surface, the second surface is a far surface, and the program code further includes first selection criteria code configured to cause the at least one processor to select one point having a smallest depth among two or more points associated with the updated CC, when the two or more points have a same projected coordinate.
12. 12 . The apparatus of claim 10 , wherein the first surface is a far surface, the second surface is a near surface, and the program code further includes first selection criteria code configured to cause the at least one processor to select one point having a largest depth among two or more points associated with the updated CC, when the two or more points have a same projected coordinate.
13. 13 . The apparatus of claim 10 , wherein the filtering code comprises: determining code configured to cause at least one of the at least one processor to determine an adjustment value based on the one or more characteristics of the CC; adjusting code configured to cause at least one of the at least one processor to adjust the grid size parameter based on the adjustment value; and filtering code configured to cause at least one of the at least one processor to filter the one or more points selected for the first surface using a grid in the projection plane associated with the CC, wherein the grid is based on the adjusted grid parameter.
14. 14 . The apparatus of claim 13 , wherein the adjustment value is based on an adjustment threshold value and a range between a smallest depth value and a largest depth value associated with the plurality of points included in the CC.
15. 15 . A non-transitory computer readable medium that stores computer readable program code which, when executed by a processor, causes the processor to at least: project a plurality of points included in a connected component (CC) to a projection plane associated with the CC; remove one or more outlier points from the plurality of points based on the projected points to generate an updated CC; determine a first surface by selecting one or more points associated with the updated CC, based on the projected points and a first selection criteria; adaptively filter the one or more points selected for the first surface, by adjusting a grid size parameter based on one or more characteristics of the CC including at least one of statistics associated with a depth of the points in the CC, statistics associated with a geometry of the points in the CC, statistics associated with a color of the points in the CC, and statistics associated with other attributes of the points in the CC; determine a second surface by selecting one or more points associated with the updated CC, based on the projection of the one or more points and a second selection criteria; add a patch corresponding to the filtered points for the first surface and the second surface to a list of patches associated with an original point cloud; and code the original point cloud based on the list of patches.
16. 16 . The non-transitory computer readable medium of claim 15 , wherein the first surface is a near surface, the second surface is a far surface, and the program code further cause the processor to at least select one point having a smallest depth among two or more points associated with the updated CC, when the two or more points have a same projected coordinate.
17. 17 . The non-transitory computer readable medium of claim 15 , wherein the first surface is a far surface, the second surface is a near surface, and the program code further cause the processor to at least select one point having a largest depth among two or more points associated with the updated CC, when the two or more points have a same projected coordinate.
18. 18 . The non-transitory computer readable medium of claim 15 , wherein the program code which causes the processor to adaptively filter, further cause the processor to at least: determine an adjustment value based on the one or more characteristics of the CC; adjust the grid size parameter based on the adjustment value; and filter the one or more points selected for the first surface using a grid in the projection plane associated with the CC, wherein the grid is based on the adjusted grid parameter.
19. 19 . The non-transitory computer readable medium of claim 18 , wherein the adjustment value is based on an adjustment threshold value and a range between a smallest depth value and a largest depth value associated with the plurality of points included in the CC.

Description

CROSS-REFERENCE TO RELATED APPLICATION This application is based on and claims priority to U.S. Provisional Application No. 63/286,434, filed on Dec. 6, 2021, in the U.S. Patent and Trademark Office, the disclosure of which is incorporated by reference herein in its entirety. BACKGROUND 1. Field The disclosure relates to Point Cloud Coding (PCC), and more specifically, systems and methods for adaptive filtering in patch generation for Video-based Point Cloud Coding (V-PCC). 2. Description of Related Art A point cloud is used in many applications. For example, a point cloud is used in autonomous driving vehicles for object detection and localization, in geographic information systems (GIS) for mapping, and in cultural heritage preservation to visualize and archive cultural heritage objects and collections, etc. Point clouds contain a set of high-dimensional points. For example, each point includes three-dimensional (3D) position information, and additional attributes such as color, reflectance, etc. Point clouds may be captured, for example, using multiple cameras and depth sensors, or laser imaging, detection, and ranging (LIDAR). To realistically represent a scene, point clouds may comprise thousands to billions of points. Compression technologies are needed to reduce the amount of data required to represent a point cloud for faster transmission or reduction of storage. ISO/IEC MPEG (JTC 1/SC 29/WG 11) has created an ad-hoc group (MPEG-PCC) to standardize the compression techniques for static or dynamic point clouds. SUMMARY According to an aspect of the disclosure, there is provided a method that includes projecting a plurality of points included in a connected component (CC) to a projection plane associated with the CC; removing one or more outlier points from the plurality of points based on the projection to generate an updated CC; determining a first surface by selecting one or more points associated with the updated CC, based on the projection of the one or more points and a first selection criteria; adaptively filtering the one or more points selected for the first surface, by adjusting a grid size parameter based on one or more characteristics of the CC; determining a second surface by selecting one or more points associated with the updated CC, based on the projection of the one or more points and a second selection criteria; adding a patch corresponding to the filtered first surface and the second surface to a list of patches associated with an original point cloud; and coding the original point cloud based on the list of patches. According to other aspects of the disclosure, an apparatus and computer readable medium consistent with the method are also provided. BRIEF DESCRIPTION OF THE DRAWINGS The above and other aspects, features and advantages of certain embodiments of the present disclosure will be more apparent from the following description taken in conjunction with the accompanying drawings, in which: FIG. 1 is a diagram illustrating components of one or more devices, according to various embodiments; FIG. 2 is a flow diagram illustrating a method of patch generation in Video-based Point Cloud Coding (V-PCC), according to various embodiments; FIG. 3 is a flowchart illustrating a method of generating a near surface s-near, a far surface s-far, and a list of patches, based on a CC, according to various embodiments; FIG. 4 is a flowchart illustrating another method of generating a near surface s-near, a far surface s-far, and a list of patches, based on a CC, according to various embodiments; FIG. 5 illustrates an sample reconstructed point cloud; FIG. 6 is a flowchart illustrating a method of generating a near surface s-near, a far surface s-far, and a list of patches, based on a CC, by performing adaptive filtering, according to various embodiments; and FIG. 7 is a flowchart illustrating another method of generating a near surface s-near, a far surface s-far, and a list of patches, based on a CC, by performing adaptive filtering, according to various embodiments. DETAILED DESCRIPTION According to various embodiment, adaptive filtering methods are provided, which can be applied to a patch generation process, such as patch generation in TMC2 or grided patch generation. The methods can be applied to any generated depth image including the near surface depth image and the far surface depth image. According to various embodiment, the method may adjust a parameter T adaptively based on characteristics of a connected component. The characteristics that may be taken into account may include but not limited to the following: Statistics of a depth of points in the connected component. For example, a depth range, a minimum/maximum depth value, a mean depth value, a depth variation, etc. Statistics of a geometry of the points in the connected component. For example, a bounding box, a geometry distribution, etc. Statistics of a color of the points in the connected component. For example, a color simila