US-12626444-B1 - Systems and methods for generating a splat representation of a three-dimensional asset without an iterative rendering phase

US12626444B1US 12626444 B1US12626444 B1US 12626444B1US-12626444-B1

Abstract

A splat generation system and associated methods implement a modified splatting pipeline that quantifies loss at each splat generation iteration without rendering the splats generated at each iteration or comparing the resulting visualization against reference images of a 3D asset. The system receives the 3D asset and generates a splat representation of the 3D asset. The system compares geometric similarity between different sets of generated splats in different regions of the splat representation and different sets of 3D primitives in corresponding regions of the 3D asset. The system adjusts a set of splats in response to the geometric similarity between the set of splats and a set of 3D primitives in corresponding regions not satisfying a threshold, and reconstructs the 3D asset with an acceptable amount of loss based on the adjustments made to the splats on a region-by-region basis.

Inventors

Stephen Lesser
Joseph Nordling

Assignees

Illuscio, Inc.

Dates

Publication Date: 20260512
Application Date: 20250602

Claims (20)

1 . A method comprising: receiving a three-dimensional (3D) asset that is defined with a plurality of 3D primitives; generating a splat representation of the 3D asset, wherein the splat representation is defined with a plurality of splats; comparing geometric similarity between different sets of splats from the plurality of splats in different regions of the splat representation and different sets of 3D primitives from the plurality of 3D primitives in corresponding regions of the 3D asset over a first set of training iterations that do not adjust colors associated with the plurality of splats; adjusting a position or a geometry of a set of splats from the different sets of splats without adjusting the colors associated with the set of splats at each training iteration of the first set of training iterations in response to the geometric similarity between the set of splats and a set of 3D primitives from the different sets of 3D primitives in a region of the 3D asset that corresponds to a region with the set of splats in the splat representation not satisfying a threshold; reconstructing the 3D asset with an acceptable amount of geometric loss with the splat representation based on said adjusting of the set of splats across the first set of training iterations; and performing a color adjustment iteration after the first set of training iterations, wherein performing the color adjustment iteration comprises adjusting the colors of the set of splats to match colors of corresponding 3D primitives from the set of 3D primitives with an acceptable amount of color loss.
2 . The method of claim 1 further comprising: partitioning the different regions of the splat representation into a first plurality of voxels and the corresponding regions of the 3D asset into a corresponding second plurality of voxels; selecting a first voxel from the first plurality of voxels that encompasses the set of splats and a corresponding first voxel from the corresponding second plurality of voxels that encompasses the set of 3D primitives; and wherein comparing the geometric similarity comprises comparing a geometric complexity of one or more surfaces formed by the set of splats against a geometric complexity of one or more surfaces formed by the set of 3D primitives.
3 . The method of claim 1 , wherein comparing the geometric similarity comprises: determining a distribution of normals associated with the set of splats; determining a distribution of normals associated with the set of 3D primitives; and calculating a deviation between the distribution of normals associated with the set of splats and the distribution of normals associated with the set of 3D primitives.
4 . The method of claim 1 , wherein comparing the geometric similarity comprises: determining an amount by which normals across one or more surfaces formed by the set of splats deviate from normals across one or more surfaces formed by the set of 3D primitives.
5 . The method of claim 1 , wherein comparing the geometric similarity comprises: determining a deviation between one or more surfaces formed by the set of splats and one or more surfaces formed by the set of 3D primitives; and wherein adjusting the position or the geometry of the set of splats comprises: modifying one or more of a position or a geometry of the set of splats to reduce the deviation.
6 . The method of claim 1 , wherein adjusting the position or the geometry of the set of splats comprises one or more of: repositioning one or more of the set of splats, reorienting one or more of the set of splats, adding one or more new splats to the set of splats, or removing one or more splats from the set of splats.
7 . The method of claim 1 , wherein adjusting the position or the geometry of the set of splats comprises: defining an optimized set of splats from the set of splats with greater geometric similarity with the set of 3D primitives than the set of splats.
8 . The method of claim 1 , wherein adjusting the position or the geometry of the set of splats comprises: performing one or more optimization iterations, wherein performing the one or more optimization iterations comprises: producing a different modified set of splats by adjusting one or more splats from the set of splats at each optimization iteration of the one or more optimization iterations; comparing geometric similarity between the different modified set of splats and the set of 3D primitives at each optimization iteration of the one or more optimization iterations; and halting the one or more optimization iterations in response to the geometric similarity between the different modified set of splats produced at a current optimization iteration and the set of 3D primitives satisfying the threshold.
9 . The method of claim 1 , wherein performing the color adjustment iteration comprises: illuminating the splat representation with one or more light sources in response to the geometric similarity between the different sets of splats and the different sets of 3D primitives satisfying the threshold after said adjusting; and defining view-dependent coloring for each particular splat of the splat representation based on light from the one or more light sources reaching the particular splat and one or more absorption, scattering, emission, or material properties of the particular splat.
10 . The method of claim 1 , wherein generating the splat representation comprises: defining each particular splat of the plurality of splats with a position in a 3D space, one or more shape parameters, and an absorption, scattering, emission, or material parameter that is inherited from one or more 3D primitives from the plurality of 3D primitives that are represented by the particular splat in the splat representation.
11 . The method of claim 1 , wherein comparing the geometric similarity comprises: computing an amount of loss in the region with the set of splats based on different amounts by which a geometry created by the set of splats differs from a geometry created by the set of 3D primitives.
12 . The method of claim 1 , wherein comparing the geometric similarity comprises: determining different amounts by which one or more surfaces formed by the set of splats differ from one or more surfaces formed by the set of 3D primitives at corresponding points in the region with the set of splats and the region of the 3D asset.
13 . The method of claim 1 , wherein generating the splat representation comprises: receiving a set of reference images that capture different views of the 3D asset; and generating the plurality of splats to recreate different surfaces as captured in the different views of the 3D asset.
14 . The method of claim 1 , wherein generating the splat representation comprises: directly replacing the plurality of 3D primitives with the plurality of splats by generating each splat of the plurality of splats with a position and a shape that encompasses a region of 3D space that is spanned by one or more 3D primitives of the plurality of 3D primitives.
15 . A splat generation system comprising: one or more hardware processors configured to: receive a three-dimensional (3D) asset that is defined with a plurality of 3D primitives; generate a splat representation of the 3D asset, wherein the splat representation is defined with a plurality of splats; compare geometric similarity between different sets of splats from the plurality of splats in different regions of the splat representation and different sets of 3D primitives from the plurality of 3D primitives in corresponding regions of the 3D asset over a first set of training iterations that do not adjust colors associated with the plurality of splats; adjust a position or a geometry of a set of splats from the different sets of splats without adjusting the colors associated with the set of splats at each training iteration of the first set of training iterations in response to the geometric similarity between the set of splats and a set of 3D primitives from the different sets of 3D primitives in a region of the 3D asset that corresponds to a region with the set of splats in the splat representation not satisfying a threshold; reconstruct the 3D asset with an acceptable amount of geometric loss with the splat representation based on said adjusting of the set of splats across the first set of training iterations; and perform a color adjustment iteration after the first set of training iterations, wherein performing the color adjustment iteration comprises adjusting the colors of the set of splats to match colors of corresponding 3D primitives from the set of 3D primitives with an acceptable amount of color loss.
16 . The splat generation system of claim 15 , wherein the one or more hardware processors are further configured to: partition the different regions of the splat representation into a first plurality of voxels and the corresponding regions of the 3D asset into a corresponding second plurality of voxels; select a first voxel from the first plurality of voxels that encompasses the set of splats and a corresponding first voxel from the corresponding second plurality of voxels that encompasses the set of 3D primitives; and wherein comparing the geometric similarity comprises comparing a geometric complexity of one or more surfaces formed by the set of splats against a geometric complexity of one or more surfaces formed by the set of 3D primitives.
17 . The splat generation system of claim 15 , wherein comparing the geometric similarity comprises: determining a distribution of normals associated with the set of splats; determining a distribution of normals associated with the set of 3D primitives; and calculating a deviation between the distribution of normals associated with the set of splats and the distribution of normals associated with the set of 3D primitives.
18 . The splat generation system of claim 15 , wherein comparing the geometric similarity comprises: determining an amount by which normals across one or more surfaces formed by the set of splats deviate from normals across one or more surfaces formed by the set of 3D primitives.
19 . The splat generation system of claim 15 , wherein comparing the geometric similarity comprises: determining a deviation between one or more surfaces formed by the set of splats and one or more surfaces formed by the set of 3D primitives; and wherein adjusting the position or the geometry of the set of splats comprises: modifying one or more of a position or a geometry of the set of splats to reduce the deviation.
20 . A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a splat generation system, cause the splat generation system to perform operations comprising: receiving a three-dimensional (3D) asset that is defined with a plurality of 3D primitives; generating a splat representation of the 3D asset, wherein the splat representation is defined with a plurality of splats; comparing geometric similarity between different sets of splats from the plurality of splats in different regions of the splat representation and different sets of 3D primitives from the plurality of 3D primitives in corresponding regions of the 3D asset over a first set of training iterations that do not adjust colors associated with the plurality of splats; adjusting a position or a geometry of a set of splats from the different sets of splats without adjusting the colors associated with the set of splats at each training iteration of the first set of training iterations in response to the geometric similarity between the set of splats and a set of 3D primitives from the different sets of 3D primitives in a region of the 3D asset that corresponds to a region with the set of splats in the splat representation not satisfying a threshold; reconstructing the 3D asset with an acceptable amount of geometric loss with the splat representation based on said adjusting of the set of splats across the first set of training iterations; and performing a color adjustment iteration after the first set of training iterations, wherein performing the color adjustment iteration comprises adjusting the colors of the set of splats to match colors of corresponding 3D primitives from the set of 3D primitives with an acceptable amount of color loss.

Description

BACKGROUND Gaussian splatting includes obtaining a set of images that capture a three-dimensional (3D) asset from different viewpoints, reconstructing the 3D model by determining the relative positions of the cameras used to capture the set of images, and iteratively generating and refining different sets of splats until a particular set of splats matches the visual detail in the set of images with an acceptable loss. Gaussian splatting is computationally expensive because the set of splats generated at each iteration is rendered and the resulting visualization is compared against the set of images in order to determine the amount of loss and/or refinements or changes to make in a subsequent iteration. It may take hundreds or thousands of iterations depending on the complexity of the 3D asset in order to arrive at the final set of splats that reproduce the 3D asset with the acceptable loss with the rendering and visualization comparison steps being the more computational expensive and time consuming steps of the splatting pipeline. Consequently, the splat generation process may take tens of minutes or hours to produce an acceptable splat representation of a single 3D asset. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 illustrates an example of generating a splat representation of a three-dimensional (3D) asset based on 3D comparison that replaces the iterative rendering phase in the splatting pipeline in accordance with some embodiments presented herein. FIG. 2 illustrates an example of partitioning the 3D space of a splat representation into voxels for a voxel-by-voxel optimization of the splat representation in accordance with some embodiments presented herein. FIG. 3 illustrates an example of a 3D comparison performed in corresponding voxels of the splat representation and the original 3D asset in accordance with some embodiments presented herein. FIG. 4 illustrates an example of optimizing splats on a voxel-by-voxel basis for a splat representation of a 3D asset in accordance with some embodiments presented herein. FIG. 5 illustrates an example of defining the view-dependent color for the splats after generating and optimizing the splats to match the geometric complexity of an original 3D asset in accordance with some embodiments. FIG. 6 presents a process for the modified splatting pipeline that separates the splat generation phase from the view-dependent color definition phase in order to efficiently create a splat representation of a 3D asset in accordance with some embodiments presented herein. FIG. 7 illustrates example components of one or more devices, according to one or more embodiments described herein. DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS The following detailed description refers to the accompanying drawings. The same reference numbers in different drawings may identify the same or similar elements. Provided are systems and associated methods for generating a splat representation of a three-dimensional (3D) asset without the iterative rendering phase of the splatting pipeline. The systems and methods modify the splatting pipeline to quantify loss at each splat generation iteration based on a 3D comparison that replaces the need to render the splats generated at each iteration or to compare the resulting visualization against reference images of the 3D asset. The 3D comparison may be efficiently computed from values or distributions that are derived from the positional data associated with the generated splats and the original 3D primitives of the 3D asset. In some embodiments, the 3D comparison may include a comparison of the surface normal distribution between the generated splats and the corresponding set of 3D primitives from the 3D asset that are represented or reconstructed by the generated splats. FIG. 1 illustrates an example of generating a splat representation of a 3D asset based on 3D comparison that replaces the iterative rendering phase in the splatting pipeline in accordance with some embodiments presented herein. Splat generation system 100 receives (at 102) a 3D asset and a request to convert the 3D asset into a splat format (e.g., a 3D Gaussian splat representation). For instance, the 3D asset may be defined as a mesh model with a connected set of meshes or polygons that construct the 3D shape and visual characteristics of one or more objects or environments represented by the 3D asset. Alternatively, the 3D model may be defined as a point cloud with a distributed and disconnected set of points that construct the 3D shape and visual characteristics of the one or more objects or environments represented by the 3D model. Splat generation system 100 generates (at 104) a set of images that capture the 3D asset from different perspectives. In some embodiments, the set of images may be generated based on a placement of virtual cameras around the 3D asset with the set of images capturing the rendered visualization of the 3D asset from each virtual camera position. Splat generati