CN-121999103-A - Three-dimensional Gaussian splatter generation method based on density sampling, electronic equipment and program product

CN121999103ACN 121999103 ACN121999103 ACN 121999103ACN-121999103-A

Abstract

The invention provides a three-dimensional Gaussian splatter generation method based on density sampling, electronic equipment and a program product, wherein the three-dimensional Gaussian splatter generation method based on the density sampling comprises the steps of extracting visual features from a multi-view image through a pre-trained visual encoder, and fusing and compressing the visual features into a potential representation Z; and sampling from the probability density distribution to obtain a center point position of three-dimensional Gaussian splatter, and performing differential rendering based on the center point position to obtain a rendered image. And on the condition that the potential representation Z is adopted, predicting probability density distribution of the three-dimensional space by using a hierarchical decoder, and further driving adaptive sampling of the Gaussian center. The method realizes the accurate allocation of computing resources, performs end-to-end optimization through differential rendering, and achieves the purposes of improving the quality and the fidelity of the generated result while guaranteeing the efficiency.

Inventors

YAN RUNJIE
GUO YUANCHEN
WANG PENG
CAO YANPEI

Assignees

北京哇嘶嗒科技有限公司

Dates

Publication Date: 20260508
Application Date: 20260211

Claims (10)

1. A method for three-dimensional gaussian splatter generation based on density sampling, comprising: Extracting visual features from the multi-view image by a pre-trained visual encoder and fusion-compressing the visual features into a potential representation Z; Predicting, by a hierarchical decoder, a probability density distribution in three-dimensional space, said probability density distribution being factorized by an octree structure, on the condition of said potential representation Z; Sampling from the probability density distribution to obtain a center point position of three-dimensional Gaussian splatter, and performing differential rendering based on the center point position to obtain a rendered image; calculating the rendering loss between the rendering image and the real image, calculating the marginal contribution of each center point position to the rendering loss through a differentiable gradient estimation method, and back-propagating the rendering loss to the parameters of the layered decoder according to the marginal contribution so as to optimize the probability density distribution.
2. The density sampling based three-dimensional gaussian splatter generation method according to claim 1, wherein said octree structure has L levels, said probability density distribution is represented by a factorized joint probability formula, which is: , in order to maintain the spatial probability density distribution, Is a coordinate point in a three-dimensional space, Is the total number of layers of the octree, For the octree level at which it is currently located, For full path encoding from the root node to the layer i node, Encoding a parent node path to a level i node, Splitting probabilities for nodes.
3. The density sampling based three-dimensional gaussian splatter generation method of claim 2, wherein optimizing the loss function of the probability density distribution comprises cross entropy loss, formulated as: , Wherein, the In order for the cross-entropy loss to occur, In the true data, the path The probability of the occurrence of the presence of a defect, Is a conditional probability predicted by a layered decoder with a parameter θ.
4. The density sampling based three-dimensional gaussian splatter generation method of claim 3, wherein the differentiable gradient estimation formula is: , Wherein, the To measure the marginal contribution of each anchor to rendering loss, For the purpose of the gradient operator, In order to render the loss, As a result of the desired value(s), For the index of the sample point, For the total number of center points obtained for a single sample, Is the first Marginal contributions of the gaussian center points to rendering losses, In order to score the gradient of the function, The center point position of the sampled 3D gaussian splatter is the j-th.
5. The density sampling based three dimensional gaussian splatter generation method of claim 4, wherein calculating said marginal contribution comprises: in the forward rendering process, calculating residual variables of a real image and a rendered image; Calculating the color change caused by removing the single Gaussian splash; based on the residual variable and the color variation, a marginal contribution of the gaussian splatter to rendering loss is calculated.
6. The density sampling based three dimensional gaussian splatter generation method according to claim 5, wherein the formula for calculating the marginal contribution of the gaussian splatter to rendering loss based on the residual and the color variance is: , Wherein, the In order to make a marginal contribution, As a residual variable, the residual variable is, In order to provide the amount of color change, Is a pixel point.
7. The density sampling based three-dimensional gaussian splatter generation method according to claim 1, wherein said extracting visual features from multi-view images by a pre-trained visual encoder comprises: Mapping the unordered potential token set contained in the potential representation Z to a deterministic three-dimensional sequence point set to generate an ordered potential sequence.
8. The density sampling based three-dimensional gaussian splatter generation method according to claim 7, wherein the formula for mapping the unordered set of potential tokens contained in the potential representation Z to the deterministic set of three-dimensional sequence points is: , the optimal allocation scheme is represented by the formula, For an allocation function based on the optimal transmission theory, As a potential token of interest, In the absence of a set of potential token, For the Sobol sequence point, A set of deterministic three-dimensional sequence points is a set of Sobol sequences.
9. An electronic device, the electronic device comprising: At least one processor, and A memory communicatively coupled to the at least one processor, wherein, The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the density sampling based three-dimensional gaussian splatter generation method of any of claims 1-8.
10. A computer program product comprising computer program/instructions which, when executed by a processor, implements the density sampling based three-dimensional gaussian splatter generation method of any of claims 1 to 8.

Description

Three-dimensional Gaussian splatter generation method based on density sampling, electronic equipment and program product Technical Field The invention belongs to the technical field of three-dimensional rendering, and particularly relates to a three-dimensional Gaussian splatter generation method based on density sampling, electronic equipment and a program product. Background 3D Gaussian Splatting (3 DGS) as an emerging three-dimensional representation method, real-time high quality rendering is achieved by differential rasterization. In recent years, researchers have begun to explore the application of 3DGS to generative modeling in image reconstruction, but existing methods typically employ a fixed number of gaussian primitives and cannot accommodate the complexity requirements of different scenarios. The prior art includes the TRELLIS method, which uses a sparse voxel structure, assigns a fixed number of Gaussian primitives to each voxel. UniLat 3A 3D method is that a regular grid structure is adopted, and each grid unit comprises a preset number of gauss. GaussianCube method of mapping the structured grid to a target gaussian set by optimal transmission. These methods have a problem that adaptive density control is not possible. In addition, the existing three-dimensional Gaussian splatter generation technology has the fixed structure limitation that the existing method binds Gaussian primitives to a predefined structure (such as a fixed grid or a voxel), and the number of the Gaussian primitives cannot be adaptively distributed according to the local geometric complexity, so that the calculation resources are wasted in a simple area, and the detail is insufficient in a complex area. The inventors found during the course of this example that the prior art had the following: (1) The uniform distribution causes resource waste, namely the fixed structure can not adjust the density according to the complexity of the scene; (2) Lack of micro-density control-traditional densification/pruning operations cannot be incorporated into end-to-end training; (3) Training is inefficient, and spread training converges slowly due to permutation ambiguity. Disclosure of Invention Aiming at the problems in the prior art, the invention provides a three-dimensional Gaussian splatter generation method based on density sampling, electronic equipment and a program product. In a first aspect, an embodiment of the present disclosure provides a method for generating three-dimensional gaussian splatter based on density sampling, including: Extracting visual features from the multi-view image by a pre-trained visual encoder and fusion-compressing the visual features into a potential representation Z; Predicting, by a hierarchical decoder, a probability density distribution in three-dimensional space, said probability density distribution being factorized by an octree structure, on the condition of said potential representation Z; Sampling from the probability density distribution to obtain a center point position of three-dimensional Gaussian splatter, and performing differential rendering based on the center point position to obtain a rendered image; calculating the rendering loss between the rendering image and the real image, calculating the marginal contribution of each center point position to the rendering loss through a differentiable gradient estimation method, and back-propagating the rendering loss to the parameters of the layered decoder according to the marginal contribution so as to optimize the probability density distribution. Optionally, the octree structure has L levels, and the probability density distribution is represented by a factored joint probability formula, where the joint probability formula is: , in order to maintain the spatial probability density distribution, Is a coordinate point in a three-dimensional space,Is the total number of layers of the octree,For the octree level at which it is currently located,For full path encoding from the root node to the layer i node,Encoding a parent node path to a level i node,Splitting probabilities for nodes. Optionally, optimizing the loss function of the probability density distribution includes cross entropy loss, which is formulated as: , Wherein, the In order for the cross-entropy loss to occur,In the true data, the pathThe probability of the occurrence of the presence of a defect,Is a conditional probability predicted by a layered decoder with a parameter θ. Alternatively, the differentiable gradient estimation formula is: , Wherein, the To measure the marginal contribution of each anchor to rendering loss,For the purpose of the gradient operator,In order to render the loss,As a result of the desired value(s),For the index of the sample point,For the total number of center points obtained for a single sample,Is the firstMarginal contributions of the gaussian center points to rendering losses,In order to score the gradient of the function,The center point