CN-116188682-B - Real-time rendering method and device based on dynamic perceptron graph

CN116188682BCN 116188682 BCN116188682 BCN 116188682BCN-116188682-B

Abstract

The invention discloses a real-time rendering method and device based on a dynamic perceptron map, which are used for representing a volume video by predicting a frame-by-frame perceptron map so as to realize real-time rendering of a dynamic three-dimensional scene. The realization method comprises the steps of predicting a perceptron map of each frame through a two-dimensional deep neural convolution network to represent a three-dimensional scene of each frame given a multi-view video, wherein the perceptron map is a two-dimensional map, each pixel stores network parameters of a multi-layer perceptron, and a neural representation of the geometry and appearance of the three-dimensional scene is constructed based on the perceptron map. For any three-dimensional point in the three-dimensional scene, selecting corresponding network parameters from the perceptron map for predicting voxel density and color of the three-dimensional point. According to the invention, through the dynamic perceptron map, the efficient modeling of the nerve volume video is realized, and the rendering speed cost of the dynamic three-dimensional scene is improved.

Inventors

ZHOU XIAOWEI
BAO HUJUN
Peng sida
YAN BINZHI

Assignees

浙江大学

Dates

Publication Date: 20260512
Application Date: 20221227

Claims (4)

1. The real-time rendering method based on the dynamic perception map is characterized by comprising the following steps of: (1) For a target dynamic scene, acquiring a multi-view video by using a synchronous multi-view camera; (2) For video frames of multi-view video, predicting a dynamic perceptron map based on a two-dimensional depth convolution network, wherein the perceptron map refers to a two-dimensional characteristic lattice diagram, corresponding to two-dimensional images on each video frame, each image pixel in the lattice diagram stores network parameters of a multi-layer perceptron in the form of parameter vectors, and the parameter vectors are remodeled into scene information corresponding to the multi-layer perceptron back and forth when the parameter vectors need to be inquired; (3) The method comprises the steps of representing three-dimensional scenes of each frame in a nerve volume video based on a perceptron graph, realizing real-time rendering of dynamic scenes, wherein three mutually orthogonal perceptron graphs represent the three-dimensional scenes, specifically, the three-dimensional planes where the three perceptron graphs are located are mutually perpendicular, projecting arbitrary three-dimensional points onto the three mutually orthogonal perceptron graph planes respectively, regressing voxel density and color respectively, finally adding to obtain the voxel density and color of the three-dimensional points, and selecting corresponding network parameters through the perceptron graphs to predict the voxel density and color of the three-dimensional points.
2. The method for real-time rendering based on dynamic perceptron map of claim 1, wherein in step (2), the two-dimensional depth convolution network-based prediction dynamic perceptron map is specifically implemented by selecting a group of pictures from an input view angle for any video frame in order to obtain a volume video from a multi-view video, obtaining feature vectors by using a two-dimensional convolution encoding network, inputting the feature vectors into a two-dimensional convolution decoding network, and outputting the perceptron map of the video frame.
3. A real-time rendering device based on a dynamic perceptron map, comprising a memory and one or more processors, the memory having executable code stored therein, wherein the processor, when executing the executable code, implements a real-time rendering method based on a dynamic perceptron map as defined in any one of claims 1-2.
4. A computer readable storage medium having stored thereon a program, which when executed by a processor, implements a real-time rendering method based on a dynamic perceptual engine map as defined in any one of claims 1-2.

Description

Real-time rendering method and device based on dynamic perceptron graph Technical Field The invention belongs to the field of rendering, and particularly relates to a real-time rendering method and device based on a dynamic perception map. Background The invention provides a dynamic perceptron graph for representing a dynamic three-dimensional scene, and realizes real-time rendering of the dynamic scene. In the related art, most methods focus on rendering of static scenes. Some neural network-based rendering methods achieve dynamic scene rendering by introducing a variable of a time dimension, but these methods have a slow rendering speed. While these methods may accelerate through pre-calculation techniques, they also result in increased storage costs. The invention aims at representing a three-dimensional scene by a group of small neural networks and realizing efficient rendering of the dynamic scene under the condition of low storage cost. Disclosure of Invention The invention aims at overcoming the defects of the prior art, and provides a real-time rendering method and a device based on a dynamic perception map, and predicting a frame-by-frame perceptron map based on the two-dimensional depth convolution network, wherein the perceptron map is used for representing the geometry and appearance of a three-dimensional scene frame by frame, so that the real-time rendering of the volume video is realized. The invention aims at realizing the following technical scheme that in a first aspect, the invention provides a real-time rendering method based on a dynamic perceptron map, which comprises the following steps: (1) For a target dynamic scene, acquiring a multi-view video by using a synchronous multi-view camera; (2) For video frames of multi-view video, a dynamic perceptron map is predicted based on a two-dimensional depth convolution network. The dynamic perceptron map is a two-dimensional image on each video frame, and each image pixel records parameters of a perceptron network; (3) And representing each frame of three-dimensional scene in the nerve volume video based on the perceptron graph, and realizing real-time rendering of the dynamic scene. In step (2), the two-dimensional depth convolution network-based prediction dynamic perceptron map is specifically that, in order to obtain a volume video from a multi-view video, for any video frame, a group of pictures are selected from an input view, a two-dimensional convolution encoding network is used to obtain feature vectors, then the feature vectors are input into a two-dimensional convolution decoding network, and the perceptron map of the video frame is output. Further, in the step (3), each frame of three-dimensional scene in the neural volume video is represented based on the perceptron map, specifically, for any three-dimensional point in the three-dimensional scene, corresponding network parameters are selected from the perceptron map and used for predicting voxel density and color of the three-dimensional point. Further, three orthogonal perceptron graphs are used for representing a three-dimensional scene, specifically, two-dimensional planes where the three perceptron graphs are located are mutually perpendicular, for any three-dimensional point, the three-dimensional points are respectively projected onto the three orthogonal perceptron graph planes, voxel density and color are respectively regressed, and finally, the voxel density and the color of the three-dimensional point are obtained through addition. In a second aspect, the present invention provides a real-time rendering device based on a dynamic perceptron map, including a memory and one or more processors, where the memory stores executable codes, and the processors implement the real-time rendering method based on the dynamic perceptron map when executing the executable codes. In a third aspect, the present invention provides a computer readable storage medium having a program stored thereon, which when executed by a processor, implements the method for real-time rendering based on a dynamic perceptron map. The method has the beneficial effects that the dynamic perception map is provided, so that the real-time rendering of the dynamic scene is realized. In order to obtain the perceptron map of each frame, the invention predicts the perceptron map based on a two-dimensional depth convolution network. The invention also provides a mutually orthogonal perceptron graph, and the modeling capability of the three-dimensional scene is improved by adding signals which represent the three-dimensional scene as three orthogonal directions. Drawings FIG. 1 is a pictorial view of the present invention for generating a single perceptron from a multi-view two-dimensional picture. Fig. 2 is a diagram of a perceptron based on mutual orthogonality for regression of continuous voxel density and color schematic drawing, as proposed by the present invention. Fig. 3 is a block diagram of a real-time renderi