US-12620190-B2 - Method and apparatus for transferring facial expression of digital human, electronic device, and storage medium

US12620190B2US 12620190 B2US12620190 B2US 12620190B2US-12620190-B2

Abstract

Method and apparatus for transferring facial expression of digital human, electronic device, and storage medium which relates to the fields of augmented reality technologies, virtual reality technologies, computer vision technologies, deep learning technologies, or the like, and can be applied to scenarios, such as metaverse, a virtual digital human, or the like, An implementation includes: selecting an identification of a target reference model matched with an object model from a preset reference model library; the reference model library including a plurality of reference models; acquiring an expression library of the target reference model based on the identification of the target reference model; and transferring a last frame of an expression in the expression library of the target reference model into the object model to obtain a last frame of an expression of the object model.

Inventors

Lei Wang
Xiaodong Zhang
Shiyan Li

Assignees

BEIJING BAIDU NETCOM SCIENCE TECHNOLOGY CO., LTD.

Dates

Publication Date: 20260505
Application Date: 20240619
Priority Date: 20231016

Claims (20)

1 . A computer-implemented method for transferring an expression of a target reference model to an object model of a digital human, comprising: selecting an identification of the target reference model matched with the object model from a preset reference model library, the reference model library comprising a plurality of reference models; acquiring an expression library of the target reference model based on the identification of the target reference model, wherein the expression library of the target reference model comprises a plurality of expressions which are stored in a form of point cloud data; and transferring a last frame of an expression in the expression library of the target reference model into the object model to obtain a last frame of an expression of the object model, wherein transferring the last frame of the expression in the expression library of the target reference model into the object model to obtain the last frame of the expression of the object model comprises: acquiring a first size ratio of the expression based on point cloud data of a key feature corresponding to a last frame of the expression of the target reference model and point cloud data of the key feature of the target reference model in a natural state, wherein the first size ratio of the expression comprises a ratio of the length of the point cloud of the key feature corresponding to the last frame of the expression to the corresponding length of the point cloud of the key feature in the natural state, a ratio of the width of the point cloud of the key feature corresponding to the last frame of the expression to the corresponding width of the point cloud of the key feature in the natural state, and a ratio of the height of the point cloud of the key feature corresponding to the last frame of the expression to the corresponding height of the point cloud of the key feature in the natural state; and acquiring point cloud data of a key feature corresponding to a last frame of the expression to be transferred of the object model based on the first size ratio of the expression and point cloud data of the key feature of the object model in a natural state, so as to obtain the last frame of the expression of the object model.
2 . The method according to claim 1 , wherein the selecting an identification of a target reference model matched with an object model from a preset reference model library comprises: selecting the identification of the target reference model matched with the object model from the reference model library based on attribute information of the object model and attribute information of each reference model in the reference model library.
3 . The method according to claim 1 , wherein the selecting an identification of a target reference model matched with an object model from a preset reference model library comprises: showing attribute information of each reference model in the reference model library to a user; and receiving the user selected target reference model matched with the object model.
4 . The method according to claim 1 , wherein the selecting an identification of a target reference model matched with an object model from a preset reference model library comprises: acquiring a first feature curve of a preset part from the object model; acquiring a second feature curve of the preset part from each reference model; calculating an offset distance of the preset part between the object model and each reference model based on the first feature curve and each second feature curve; and selecting the identification of the target reference model matched with the object model from the reference model library based on the offset distance of the preset part between the object model and each reference model.
5 . The method according to claim 4 , wherein the calculating an offset distance of the preset part between the object model and each reference model based on the first feature curve and the second feature curve comprises: acquiring coordinates of each point on the first feature curve and each second feature curve; calculating a distance of points with same point identifications on the first feature curve and each second feature curve based on the coordinates of the points on the first feature curve and each second feature curve; adding the distances of the points with the point identifications on the first feature curve and each second feature curve to obtain a point distance sum; and obtaining the offset distance of the preset part between the object model and each reference model based on the point distance sum and a number of the points comprised on the first feature curve.
6 . The method according to claim 5 , wherein the selecting the identification of the target reference model matched with the object model from the reference model library based on the offset distance of the preset part between the object model and each reference model comprises: selecting an identification of the reference model with the minimum offset distance of the preset part from the multiple reference models in the reference model library as the identification of the target reference model matched with the object model.
7 . The method according to claim 5 , wherein the selecting the identification of the target reference model matched with the object model from the reference model library based on the offset distance of the preset part between the object model and each reference model comprises: if two or more preset parts are comprised, calculating a comprehensive offset distance between the object model and the reference models based on pre-configured weights of the preset parts and the offset distances corresponding to the preset parts; and selecting the identification of the target reference model matched with the object model from the reference model library based on the comprehensive offset distance between the object model and each reference model.
8 . The method according to claim 1 , wherein before acquiring the first size ratio of the expression based on point cloud data of the key feature corresponding to the last frame of the expression of the target reference model and point cloud data of the key feature of the target reference model in the natural state, the method further comprises: integrally registering the target reference model with the object model.
9 . The method according to claim 8 , after the acquiring point cloud data of a key feature corresponding to a last frame of the expression to be transferred of the object model based on the size ratio of the expression and point cloud data of the key feature of the object model in a natural state, and before the obtaining the last frame of the expression of the object model, further comprising: transferring the point cloud data of the key feature corresponding to the last frame of the expression to be transferred of the object model onto the object model; acquiring a third feature curve corresponding to the last frame of the expression and a fourth feature curve corresponding to the last frame of the expression on the target reference model after the transferring to the object model; adjusting the third feature curve by taking the fourth feature curve as a constraint; and smoothing a joint between the point cloud of the key feature corresponding to the last frame of the expression after the transferring to the object model and the original point cloud of the object model.
10 . The method according to claim 1 , after the transferring a last frame of an expression in the expression library of the target reference model into the object model to obtain a last frame of an expression of the object model, further comprising: if the expression library further comprises an intermediate frame of the expression, transferring the intermediate frame of the expression in the expression library of the target reference model into the object model to obtain an intermediate frame of the expression of the object model.
11 . The method according to claim 10 , wherein the transferring the intermediate frame of the expression in the expression library of the target reference model into the object model to obtain an intermediate frame of the expression of the object model comprises: acquiring a second size ratio of the intermediate frame of the expression relative to the last frame of the expression based on point cloud data of a key feature corresponding to the intermediate frame of the expression in the expression library of the target reference model, the point cloud data of the key feature of the target reference model in the natural state and the point cloud data of the key feature corresponding to the last frame of the expression of the target reference model; and obtaining an intermediate frame of the expression of the object model based on the second size ratio of the intermediate frame of the expression relative to the last frame and the last frame of the expression of the object model.
12 . The method according to claim 11 , wherein the acquiring a second size ratio of the intermediate frame of the expression relative to the last frame of the expression based on point cloud data of a key feature corresponding to the intermediate frame of the expression in the expression library of the target reference model, the point cloud data of the key feature of the target reference model in the natural state and the point cloud data of the key feature corresponding to the last frame of the expression of the target reference model comprises: obtaining the fourth feature curve corresponding to the last frame of the expression of the target reference model, a fifth feature curve corresponding to the intermediate frame of the expression, and a sixth feature curve in the natural state based on the point cloud data of the key feature corresponding to the intermediate frame of the expression in the expression library of the target reference model, the point cloud data of the key feature of the target reference model in the natural state, and the point cloud data of the key feature corresponding to the last frame of the expression of the target reference model; and acquiring the second size ratio of the intermediate frame of the expression relative to the last frame based on coordinates of key points on the fourth feature curve, the fifth feature curve and the sixth feature curve.
13 . The method according to claim 11 , wherein the obtaining an intermediate frame of the expression of the object model based on the second size ratio of the intermediate frame of the expression relative to the last frame and the last frame of the expression of the object model comprises: obtaining a corresponding reference feature curve of the intermediate frame of the expression of the object model based on the second size ratio of the intermediate frame of the expression relative to the last frame and the corresponding third feature curve of the last frame of the expression of the object model; transferring the point cloud data of the key feature corresponding to the intermediate frame of the expression of the target reference model onto the object model; acquiring a seventh feature curve corresponding to the intermediate frame of the expression after the transferring to the object model; and adjusting the seventh feature curve with the reference feature curve as a constraint.
14 . An electronic device, comprising: at least one processor; and a memory connected with the at least one processor communicatively; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a method for transferring an expression of a target reference model to an object model of a digital human, the method for transferring the expression of the target reference model to the object model of the digital human comprising: selecting an identification of the target reference model matched with the object model from a preset reference model library; the reference model library comprising a plurality of reference models; acquiring an expression library of the target reference model based on the identification of the target reference model, wherein the expression library of the target reference model comprises a plurality of expressions which are stored in a form of point cloud data; and transferring a last frame of an expression in the expression library of the target reference model into the object model to obtain a last frame of an expression of the object model, wherein transferring the last frame of the expression in the expression library of the target reference model into the object model to obtain the last frame of the expression of the object model comprises: acquiring a first size ratio of the expression based on point cloud data of a key feature corresponding to a last frame of the expression of the target reference model and point cloud data of the key feature of the target reference model in a natural state, wherein the first size ratio of the expression comprises a ratio of the length of the point cloud of the key feature corresponding to the last frame of the expression to the corresponding length of the point cloud of the key feature in the natural state, a ratio of the width of the point cloud of the key feature corresponding to the last frame of the expression to the corresponding width of the point cloud of the key feature in the natural state, and a ratio of the height of the point cloud of the key feature corresponding to the last frame of the expression to the corresponding height of the point cloud of the key feature in the natural state; and acquiring point cloud data of a key feature corresponding to a last frame of the expression to be transferred of the object model based on the first size ratio of the expression and point cloud data of the key feature of the object model in a natural state, so as to obtain the last frame of the expression of the object model.
15 . The electronic device according to claim 14 , wherein the selecting an identification of a target reference model matched with an object model from a preset reference model library comprises: selecting the identification of the target reference model matched with the object model from the reference model library based on attribute information of the object model and attribute information of each reference model in the reference model library.
16 . The electronic device according to claim 14 , wherein the selecting an identification of a target reference model matched with an object model from a preset reference model library comprises: showing attribute information of each reference model in the reference model library to a user; and receiving the user selected target reference model matched with the object model.
17 . The electronic device according to claim 14 , wherein the selecting an identification of a target reference model matched with an object model from a preset reference model library comprises: acquiring a first feature curve of a preset part from the object model; acquiring a second feature curve of the preset part from each reference model; calculating an offset distance of the preset part between the object model and each reference model based on the first feature curve and each second feature curve; and selecting the identification of the target reference model matched with the object model from the reference model library based on the offset distance of the preset part between the object model and each reference model.
18 . The electronic device according to claim 17 , wherein the calculating an offset distance of the preset part between the object model and each reference model based on the first feature curve and the second feature curve comprises: acquiring coordinates of each point on the first feature curve and each second feature curve; calculating a distance of points with same point identifications on the first feature curve and each second feature curve based on the coordinates of the points on the first feature curve and each second feature curve; adding the distances of the points with the point identifications on the first feature curve and each second feature curve to obtain a point distance sum; and obtaining the offset distance of the preset part between the object model and each reference model based on the point distance sum and a number of the points comprised on the first feature curve.
19 . The electronic device according to claim 18 , wherein the selecting the identification of the target reference model matched with the object model from the reference model library based on the offset distance of the preset part between the object model and each reference model comprises: selecting an identification of the reference model with the minimum offset distance of the preset part from the multiple reference models in the reference model library as the identification of the target reference model matched with the object model.
20 . A non-transitory computer readable storage medium with computer instructions stored thereon, wherein the computer instructions are used for causing a computer to perform a method for transferring an expression of a target reference model to an object model of a digital human, the method for transferring the expression of the target reference model to the object model of the digital human comprising: selecting an identification of the target reference model matched with the object model from a preset reference model library; the reference model library comprising a plurality of reference models; acquiring an expression library of the target reference model based on the identification of the target reference model, wherein the expression library of the target reference model comprises a plurality of expressions which are stored in a form of point cloud data; and transferring a last frame of an expression in the expression library of the target reference model into the object model to obtain a last frame of an expression of the object model, wherein transferring the last frame of the expression in the expression library of the target reference model into the object model to obtain the last frame of the expression of the object model comprises: acquiring a first size ratio of the expression based on point cloud data of a key feature corresponding to a last frame of the expression of the target reference model and point cloud data of the key feature of the target reference model in a natural state, wherein the first size ratio of the expression comprises a ratio of the length of the point cloud of the key feature corresponding to the last frame of the expression to the corresponding length of the point cloud of the key feature in the natural state, a ratio of the width of the point cloud of the key feature corresponding to the last frame of the expression to the corresponding width of the point cloud of the key feature in the natural state, and a ratio of the height of the point cloud of the key feature corresponding to the last frame of the expression to the corresponding height of the point cloud of the key feature in the natural state; and acquiring point cloud data of a key feature corresponding to a last frame of the expression to be transferred of the object model based on the first size ratio of the expression and point cloud data of the key feature of the object model in a natural state, so as to obtain the last frame of the expression of the object model.

Description

The present application claims priority to Chinese Patent Application No. 202311339427.X, entitled “Method and apparatus for transferring facial expression of digital human, electronic device, and storage medium”, filed on Oct. 16, 2023, the disclosure of which is incorporated herein by reference in its entirety. FIELD OF THE DISCLOSURE The present disclosure relates to the fields of computer technologies and artificial intelligence technologies, and particularly to the fields of augmented reality technologies, virtual reality technologies, computer vision technologies, deep learning technologies, or the like, which can be applied to scenarios, such as metaverse, a virtual digital human, or the like. In particular, the present disclosure relates to method and apparatus for transferring facial expression of digital human, electronic device, and storage medium. BACKGROUND OF THE DISCLOSURE As a key design link of digital human figure driving, binding can realize application of an expression of a digital human to a model of the digital human. In the prior art, binding is usually done by a professional designer. The form of binding includes blendshape deformation and skeleton skinning, and a blendshape deformation and a skeleton skinning can be combined to realize optimization in certain scenarios. For different digital human figures, binding work also requires different investments. It should be noted that a time period thereof is not short, and is usually 1 to 2 weeks or more. This time cost may be further greatly increased under a high quality expression driving effect requirement of an ultra-realistic digital human. SUMMARY OF THE DISCLOSURE The present disclosure provides method and apparatus for transferring facial expression of digital human, electronic device, and storage medium. According to an aspect of the present disclosure, there is provided a method for transferring facial expression of digital human, including: screening an identification of a target reference model matched with an object model from a preset reference model library, the reference model library including a plurality of reference models;acquiring an expression library of the target reference model based on the identification of the target reference model; andtransferring a last frame of an expression in the expression library of the target reference model into the object model to obtain a last frame of an expression of the object model. According to another aspect of the present disclosure, there is provided an electronic device, including: at least one processor; anda memory connected with the at least one processor communicatively;wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a method for transferring facial expression of digital human, the method for transferring facial expression of digital human including:screening an identification of a target reference model matched with an object model from a preset reference model library, the reference model library including a plurality of reference models;acquiring an expression library of the target reference model based on the identification of the target reference model; andtransferring a last frame of an expression in the expression library of the target reference model into the object model to obtain a last frame of an expression of the object model. According to still another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium with computer instructions stored thereon, wherein the computer instructions are used for causing a computer to perform a method for transferring facial expression of digital human, the method for transferring facial expression of digital human including: screening an identification of a target reference model matched with an object model from a preset reference model library, the reference model library including a plurality of reference models;acquiring an expression library of the target reference model based on the identification of the target reference model; andtransferring a last frame of an expression in the expression library of the target reference model into the object model to obtain a last frame of an expression of the object model. It should be understood that the statements in this section are not intended to identify key or critical features of the embodiments of the present disclosure, nor limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS The drawings are used for better understanding the present solution and do not constitute a limitation of the present disclosure. In the drawings, FIG. 1 is a schematic diagram according to a first embodiment of the present disclosure; FIG. 2 is a schematic diagram according to a second embodiment of the present disclosure; FIG. 3 is a schematic diagram of an eye