CN-122027777-A - Display method, system, electronic device and storage medium

CN122027777ACN 122027777 ACN122027777 ACN 122027777ACN-122027777-A

Abstract

The application provides a display method, a display system, electronic equipment and a storage medium. And carrying out three-dimensional reconstruction on the human body image based on a Gaussian sputtering technology, generating three-dimensional Gaussian point cloud avatar data of the object user, carrying out rasterization rendering on the three-dimensional Gao Sidian cloud avatar data based on the binocular viewpoint position information, and generating three-dimensional image data matched with the viewing angle of the watching user. And displaying the stereoscopic image data. The application can solve the problem that the existing 3D display technology is difficult to be widely popularized and applied in common consumer level scenes because of depending on specialized acquisition hardware.

Inventors

SHI WENZHE
LU PING
HU JIDONG
QIAO XIUQUAN
HUANG YAKUN
ZHANG CHONGLI

Assignees

北京兴云数科技术有限公司

Dates

Publication Date: 20260512
Application Date: 20260129

Claims (13)

1. A display method, comprising: Acquiring a human body image of an object user and binocular viewpoint position information of a watching user; Performing three-dimensional reconstruction on the human body image based on a Gaussian sputtering technology, generating three-dimensional Gaussian point cloud avatar data of the object user, performing rasterization rendering on the three-dimensional Gao Sidian cloud avatar data based on the binocular viewpoint position information, and generating three-dimensional image data matched with the viewing angle of the watching user; and displaying the stereoscopic image data.
2. The method of claim 1, wherein generating three-dimensional gaussian point cloud avatar data for the subject user based on three-dimensional reconstruction of the human body image using gaussian sputtering techniques comprises: Extracting whole body posture parameters corresponding to the whole body of the human body and local posture parameters corresponding to the local position of the human body from the human body image of the object user; Inputting the whole body attitude parameters into a pre-trained whole body Gaussian sputtering avatar model to carry out whole body three-dimensional reconstruction, and generating a whole body three-dimensional Gaussian point cloud; inputting the local attitude parameters into a pre-trained local Gaussian sputtering avatar model to perform local three-dimensional reconstruction, and generating a local three-dimensional Gaussian point cloud; and carrying out space alignment and fusion on the whole three-dimensional Gaussian point cloud and the local three-dimensional Gaussian point cloud, and generating three-dimensional Gaussian point cloud avatar data of the object user.
3. The method of claim 2, wherein spatially aligning and fusing the entire three-dimensional gaussian point cloud with the local three-dimensional gaussian point cloud to generate three-dimensional gaussian point cloud avatar data for the subject user, comprising: defining a target space corresponding to the local position of the human body in a standard coordinate system corresponding to the whole three-dimensional Gaussian point cloud; removing the Gaussian points in the target space in the whole three-dimensional Gaussian point cloud; And mapping the local three-dimensional Gaussian point cloud to the standard coordinate system, and fusing the local three-dimensional Gaussian point cloud with the rest of the whole three-dimensional Gaussian point cloud to form three-dimensional Gaussian point cloud avatar data of the object user.
4. The method according to claim 2, characterized in that before inputting whole body pose parameters in the human body image into a pre-trained whole body gaussian sputtering avatar for whole body three-dimensional reconstruction, the method comprises: Acquiring a first training sample, wherein the first training sample comprises a sample whole image corresponding to the whole human body and a sample whole pose parameter of the sample whole image; And generating the input of the whole three-dimensional Gaussian point cloud by taking the whole pose parameters of the sample in the first training sample as the whole Gaussian sputtering avatar model, and training the whole Gaussian sputtering avatar model by taking the whole image of the sample in the first training sample as a supervisory signal for carrying out raster rendering based on the whole three-dimensional Gaussian point cloud.
5. The method of claim 4, wherein during training of the whole gaussian sputtering avatar, the method further comprises: acquiring the accumulated gradient of each Gaussian point in the whole three-dimensional Gaussian point cloud, wherein the accumulated gradient refers to the accumulated update amount of the Gaussian attribute of the corresponding Gaussian point in the iteration of the whole Gaussian sputtering avatar model, and the Gaussian attribute comprises at least one of position, color, transparency, rotation angle and scaling; Splitting a Gaussian point with the accumulated gradient higher than a first threshold value in the whole three-dimensional Gaussian point cloud into at least two new Gaussian points, wherein the initial accumulated gradient of the split Gaussian points is zero; And removing a specified number of Gaussian points in the whole three-dimensional Gaussian point cloud according to the sequence from low to high of training contribution degree after the total number of Gaussian points of the whole three-dimensional Gaussian point cloud exceeds a second threshold, wherein the training contribution degree is determined based on at least one of accumulated gradient of the corresponding Gaussian points, rasterization rendering position error and rasterization rendering transparency.
6. A method according to claim 3, characterized in that before inputting the local pose parameters into a pre-trained local gaussian sputtering avatar for local three-dimensional reconstruction, the method further comprises: Acquiring a second training sample, wherein the second training sample comprises a sample local image corresponding to the local position of the human body and sample local posture parameters of the sample local image; and generating the input of the local three-dimensional Gao Sidian cloud by taking the sample local posture parameters in the second training sample as the local Gaussian sputtering avatar model, and training the local Gaussian sputtering avatar model by taking the sample local image in the second training sample as a supervision signal for carrying out rasterization rendering based on the local three-dimensional Gaussian point cloud.
7. The method of claim 1, wherein rasterizing rendering the three-dimensional Gao Sidian cloud avatar data based on the binocular viewpoint position information generates stereoscopic image data that matches the viewing user's perspective, comprising: determining a left eye viewpoint and a right eye viewpoint based on the binocular viewpoint position information; Distributing the rendering weight of the left eye viewpoint and the rendering weight of the right eye viewpoint according to the horizontal rotation angle of the head of the watching user relative to the front direction, wherein the rendering weight of the right eye viewpoint is larger than the rendering weight of the left eye viewpoint when the horizontal rotation angle is positive and larger than a third threshold value, the rendering weight of the left eye viewpoint is larger than the rendering weight of the right eye viewpoint when the horizontal rotation angle is negative and larger than a fourth threshold value, the rendering weight of the left eye viewpoint is larger than the rendering weight of the right eye viewpoint when the horizontal rotation angle is negative and larger than the fourth threshold value, the head corresponding to the positive value rotates leftwards when the horizontal rotation angle is positive, and the head corresponding to the negative value rotates rightwards when the horizontal rotation angle is negative; and carrying out rasterization rendering on the three-dimensional Gaussian point cloud avatar data of the object user based on the viewpoint positions of the left eye viewpoint and the right eye viewpoint and the corresponding rendering weights, and generating the stereoscopic image data matched with the viewing angle of the viewing user.
8. The method of claim 1, wherein the subject user and the viewing user are both users participating in the same conference and the subject user and the viewing user are the same user or different users, displaying the stereoscopic image data, comprising: the stereoscopic image data is displayed by a conference application that serves the viewing user.
9. The display system is characterized by comprising an image acquisition device, a processing device and a display device, wherein: the image acquisition device is configured to acquire a human body image of a subject user; The processing device is configured to acquire a human body image acquired by the image acquisition device, to perform three-dimensional reconstruction on the human body image of the object user based on a Gaussian sputtering technology, to generate three-dimensional Gaussian point cloud avatar data of the object user, to acquire binocular viewpoint position information of a watching user, to perform raster rendering on the three-dimensional Gao Sidian cloud avatar data based on the binocular viewpoint position information, to generate stereoscopic image data matched with the viewing angle of the watching user, and to send the stereoscopic image data to the display device; the display device is configured to display the stereoscopic image data.
10. The system of claim 9, wherein the system further comprises a controller configured to control the controller, The processing device is further configured to control the display device to lock the stereoscopic image data from a last previous frame in response to the subject user leaving a preset acquisition area of the image acquisition device.
11. The system of claim 9, further comprising a wearable device, wherein: The processing device is further configured to obtain physical properties of a virtual object interacting with the stereoscopic image data, wherein the physical properties include at least one of hardness, coefficient of friction, and quality; The wearable device is configured to generate haptic feedback based on the received haptic feedback information.
12. An electronic device comprising a processor, and a memory configured to store computer-executable instructions that, when executed, cause the processor to perform the method of any of claims 1-8.
13. A computer readable storage medium for storing computer executable instructions which, when executed by a processor, implement the method of any one of claims 1-8.

Description

Display method, system, electronic device and storage medium Technical Field The present application relates to the field of 3D display technologies, and in particular, to a display method, a system, an electronic device, and a storage medium. Background 3D display technology (such as holographic display, naked eye 3D and the like) can provide realistic space sense and presence sense for an observer by presenting stereoscopic images with depth information, so that more natural and visual interaction experience is brought. At present, the technology is primarily applied to scenes with high requirements on immersion, such as teleconferencing, virtual display, interactive entertainment, education training and the like. However, to achieve high quality three-dimensional reconstruction and stereo rendering, existing solutions typically rely on specialized hardware devices, such as multi-view camera arrays, depth sensors, or dedicated optical marker systems for data acquisition. Such devices are not only costly, but also complex to deploy, and have high requirements on the use environment and the expertise of the operators. These factors significantly restrict the popularization and landing of 3D display technology in everyday applications. Disclosure of Invention The application aims to provide a display method, a display system, electronic equipment and a storage medium, and aims to solve the problem that the existing 3D display technology is difficult to widely popularize and apply in common consumer-level scenes due to dependence on specialized acquisition hardware. In order to achieve the above object, embodiments of the present application are realized as follows: in a first aspect, a display method is provided, including: Acquiring a human body image of an object user and binocular viewpoint position information of a watching user; Performing three-dimensional reconstruction on the human body image based on a Gaussian sputtering technology, generating three-dimensional Gaussian point cloud avatar data of the object user, performing rasterization rendering on the three-dimensional Gao Sidian cloud avatar data based on the binocular viewpoint position information, and generating three-dimensional image data matched with the viewing angle of the watching user; and displaying the stereoscopic image data. In a second aspect, a display system is provided, including an image acquisition device, a processing device, and a display device, wherein: the image acquisition device is configured to acquire a human body image of a subject user; The processing device is configured to acquire a human body image acquired by the image acquisition device, to perform three-dimensional reconstruction on the human body image of the object user based on a Gaussian sputtering technology, to generate three-dimensional Gaussian point cloud avatar data of the object user, to acquire binocular viewpoint position information of a watching user, to perform raster rendering on the three-dimensional Gao Sidian cloud avatar data based on the binocular viewpoint position information, to generate stereoscopic image data matched with the viewing angle of the watching user, and to send the stereoscopic image data to the display device; the display device is configured to display the stereoscopic image data. In a third aspect, an embodiment of the application provides an electronic device comprising a processor and a memory configured to store computer-executable instructions that, when executed, cause the processor to perform the method of the first aspect. In a fourth aspect, there is provided a computer readable storage medium for storing computer executable instructions which, when executed by a processor, implement the method of the first aspect. In the scheme provided by the embodiment of the application, the human body image of the object user is firstly acquired through the common image acquisition equipment in the market, and the binocular viewpoint position information of the watching user is synchronously acquired by using the same equipment or another common camera which is cooperatively arranged. Then, three-dimensional reconstruction is carried out on the acquired human body image based on a Gaussian sputtering technology, and three-dimensional Gaussian point cloud avatar data of an object user are generated. And then, carrying out view-angle-related rasterization rendering on the three-dimensional Gao Sidian cloud avatar data by combining the real-time acquired position information of the two eye viewpoints of the watching user, and generating stereoscopic image data conforming to the stereoscopic vision principle of human eyes. Finally, the stereoscopic image data is output to a display device for presentation, so that the holographic image of the target user is restored. Through the flow, the embodiment of the application can realize 3D display with real depth perception by only utilizing common image acquisition equipment (the