CN-122023614-A - Digital human data generation method and system based on 3D and 2D fusion

CN122023614ACN 122023614 ACN122023614 ACN 122023614ACN-122023614-A

Abstract

The invention discloses a digital human data generation method and system based on 3D and 2D fusion, wherein the method comprises the steps of obtaining a 2D image to be processed; the method comprises the steps of identifying character feature data corresponding to a 2D image based on a two-dimensional image identification algorithm, screening 3D model data in a preset model database according to the character feature data, and generating a corresponding digital human model based on a multi-modal neural network according to the 2D image and the 3D model data. Therefore, the invention can realize the accurate automatic reconstruction from a single 2D image to a complete digital human model, improve the digital human generation efficiency and the authenticity, and reduce the model distortion risk caused by manual modeling or inaccurate feature matching.

Inventors

LI CHAO
Li Menchang
WU SHUANGDI
Fang Enyuan
Qian Feiwan

Assignees

广州中长康达信息技术有限公司

Dates

Publication Date: 20260512
Application Date: 20260211

Claims (10)

1. A method for generating digital data based on 3D and 2D fusion, the method comprising: acquiring a 2D image to be processed; identifying character feature data corresponding to the 2D image based on a two-dimensional image identification algorithm; screening 3D model data from a preset model database according to the character characteristic data; And generating a corresponding digital human model based on a multi-modal neural network according to the 2D image and the 3D model data.
2. The method for generating digital data based on 3D and 2D fusion according to claim 1, wherein the 2D image is image data of a user including an own image of the user.
3. The method for generating digital person data based on 3D and 2D fusion according to claim 1, wherein the identifying person feature data corresponding to the 2D image based on a two-dimensional image identification algorithm includes: dividing the 2D image based on the trained face region division algorithm model to obtain a face image; Inputting the face image into a user parameter identification model to obtain predicted user parameters corresponding to the 2D image, wherein the predicted user parameters comprise at least one of user age, user gender, user occupation, user current scene type and user height; Screening a target character feature recognition model from a plurality of candidate character feature recognition models according to the predicted user parameters, wherein the target character feature recognition model is obtained by training a training data set comprising a plurality of training image data and corresponding character feature labels; And inputting the 2D image into the target character recognition model to obtain output character feature data.
4. The method for generating digital data based on 3D and 2D fusion of claim 3, wherein the screening the target character recognition model from the plurality of candidate character recognition models based on the predicted user parameters comprises: For each candidate character feature recognition model, calculating a first similarity between a user parameter label in model training data of the candidate character feature recognition model and the predicted user parameter; Calculating an average value of image similarity between each predicted input image and the 2D image in the historical prediction record of the candidate character feature recognition model to obtain second similarity; calculating a weighted sum value of the first similarity and the second similarity to obtain the model priority of the candidate character feature recognition model; And determining the candidate character feature recognition model with the highest model priority to obtain a target character feature recognition model.
5. The 3D and 2D fusion-based digital person data generation method of claim 1, wherein the character feature data includes at least one of a figure feature, a skull feature, an arm feature, a crotch feature, a clothing feature, a makeup feature, and a hairstyle feature.
6. The method for generating digital data based on 3D and 2D fusion according to claim 1, wherein the step of screening 3D model data from a preset model database according to the character feature data comprises: for each candidate 3D model, calculating a data similarity between the modeled person description data and the person feature data for that candidate 3D model; determining a reference generated image based on the character feature data and the generated model corresponding to the candidate 3D model; Calculating an image similarity between the reference generated image and the 2D image; Calculating a weighted sum value of the data similarity and the image similarity to obtain a model matching degree corresponding to the candidate 3D model; And determining the candidate 3D model with the highest model matching degree as corresponding 3D model data.
7. The method for generating digital data based on 3D and 2D fusion of claim 6, wherein the determining a reference generated image based on the character feature data and the generated model corresponding to the candidate 3D model comprises: determining a generation algorithm model corresponding to the candidate 3D model, wherein the generation algorithm model is used for directly generating the candidate 3D model or the similarity between the generated 3D model and the candidate 3D model is larger than a preset threshold; inputting the character characteristic data into the generation algorithm model to obtain an output reference 3D model; determining shooting angle information of the 2D image; and framing the reference 3D model based on the same angle as the shooting angle information to obtain a reference generated image.
8. The method for generating digital person data based on 3D and 2D fusion according to claim 1, wherein the generating a corresponding digital person model based on a multi-modal neural network from the 2D image and the 3D model data comprises: The 2D image and the 3D model data are input into a trained digital person generation model to obtain an output digital person model, the digital person generation is obtained through training of a training data set comprising a plurality of training 3D model data and corresponding 2D image labels and digital person data labels, and the digital person model comprises three-dimensional physical parameter data and three-dimensional image data corresponding to a plurality of preset body parts.
9. A digital data generation system based on 3D and 2D fusion, the system comprising: the acquisition module is used for acquiring the 2D image to be processed; The identification module is used for identifying character characteristic data corresponding to the 2D image based on a two-dimensional image identification algorithm; The screening module is used for screening 3D model data from a preset model database according to the character characteristic data; And the generation module is used for generating a corresponding digital human model based on the multi-modal neural network according to the 2D image and the 3D model data.
10. A digital data generation system based on 3D and 2D fusion, the system comprising: a memory storing executable program code; a processor coupled to the memory; the processor invokes the executable program code stored in the memory to perform the digital person data generation method based on 3D and 2D fusion as claimed in any one of claims 1 to 8.

Description

Digital human data generation method and system based on 3D and 2D fusion Technical Field The invention relates to the technical field of data processing, in particular to a digital human data generation method and system based on 3D and 2D fusion. Background With the rapid popularization of digital people technology in the fields of virtual social contact, film and television special effects and meta-universe, enterprises and developers pay more and more attention to realizing the rapid reconstruction of high-fidelity digital people models, wherein how to generate complete digital people through multi-mode fusion to reduce distortion risks becomes a key technical problem. The prior art generally selects 3D templates from a model library by manual selection or simple tag matching, and then completes digital human generation by basic texture mapping or shallow network to meet the preliminary avatar requirements. The existing solution is difficult to realize high accuracy of feature matching and authenticity of a final digital human model due to lack of character feature depth extraction of a 2D image, intelligent screening mechanism of a preset model database and fusion modeling of the 2D image and 3D model data by a multi-mode neural network, the common mode is easily influenced by feature extraction deviation and subjectivity of template selection, digital human face detail distortion, inaccurate body type proportion or unnatural texture are caused, reconstruction quality is unstable frequently or repeated reworking is required, and automatic generation efficiency and consistency from a single 2D image to a complete digital human model are severely restricted. It can be seen that the prior art has defects and needs to be solved. Disclosure of Invention The invention aims to solve the technical problem of providing a digital person data generation method and a system based on 3D and 2D fusion, which can realize accurate automatic reconstruction from a single 2D image to a complete digital person model, improve the digital person generation efficiency and the authenticity, and reduce the model distortion risk caused by manual modeling or inaccurate feature matching. To solve the above technical problems, the first aspect of the present invention discloses a method for generating digital data based on 3D and 2D fusion, the method comprising: acquiring a 2D image to be processed; identifying character feature data corresponding to the 2D image based on a two-dimensional image identification algorithm; screening 3D model data from a preset model database according to the character characteristic data; And generating a corresponding digital human model based on a multi-modal neural network according to the 2D image and the 3D model data. As an optional implementation manner, in the first aspect of the present invention, the 2D image is image data including an own image of the user captured by the user. As an optional implementation manner, in the first aspect of the present invention, the identifying, based on a two-dimensional image identification algorithm, person feature data corresponding to the 2D image includes: dividing the 2D image based on the trained face region division algorithm model to obtain a face image; Inputting the face image into a user parameter identification model to obtain predicted user parameters corresponding to the 2D image, wherein the predicted user parameters comprise at least one of user age, user gender, user occupation, user current scene type and user height; Screening a target character feature recognition model from a plurality of candidate character feature recognition models according to the predicted user parameters, wherein the target character feature recognition model is obtained by training a training data set comprising a plurality of training image data and corresponding character feature labels; And inputting the 2D image into the target character recognition model to obtain output character feature data. In a first aspect of the present invention, the screening the target character recognition model from the plurality of candidate character recognition models according to the predicted user parameter includes: For each candidate character feature recognition model, calculating a first similarity between a user parameter label in model training data of the candidate character feature recognition model and the predicted user parameter; Calculating an average value of image similarity between each predicted input image and the 2D image in the historical prediction record of the candidate character feature recognition model to obtain second similarity; calculating a weighted sum value of the first similarity and the second similarity to obtain the model priority of the candidate character feature recognition model; And determining the candidate character feature recognition model with the highest model priority to obtain a target character feature recognition model. As