US-20260127826-A1 - Systems and Methods for 3D Facial Modeling

Abstract

In an embodiment, a 3D facial modeling system includes a plurality of cameras configured to capture images from different viewpoints, a processor, and a memory containing a 3D facial modeling application and parameters defining a face detector, wherein the 3D facial modeling application directs the processor to obtain a plurality of images of a face captured from different viewpoints using the plurality of cameras, locate a face within each of the plurality of images using the face detector, wherein the face detector labels key feature points on the located face within each of the plurality of images, determine disparity between corresponding key feature points of located faces within the plurality of images, and generate a 3D model of the face using the depth of the key feature points.

Inventors

• Kartik Venkataraman

Assignees

• ADEIA IMAGING LLC

Dates

Publication Date

20260507

Application Date

20251230

Claims (1)

1. 1 . A method for animating a three-dimensional (3D) model of a face based on frames of video containing the face, comprising: obtaining a video sequence comprising a first image and a second image containing a human face; locating the human face within the first image using a face detector, where the face detector is configured to label key feature points of the human face; selecting key feature points associated with eyes of the human face in the first image; determining a depth value for at least one of the eyes; tracking a difference in pose of the human face between the first image and the second image; and animating the 3D model based on the tracked difference and the depth value.

Description

RELATED APPLICATIONS This application is a continuation of U.S. application Ser. No. 18/491,599 filed Oct. 20, 2023 and published on Feb. 8, 2024 as US 2024/0046571, which is a continuation of U.S. application Ser. No. 17/652,078 filed Feb. 22, 2022 and issued on Nov. 28, 2023 as U.S. Pat. No. 11,830,141, which is a continuation of U.S. application Ser. No. 16/865,776 filed May 4, 2020 and issued on Feb. 22, 2022 as U.S. Pat. No. 11,257,289, which is a continuation of U.S. application Ser. No. 15/823,473 filed Nov. 27, 2017 and issued on May 5, 2020 as U.S. Pat. No. 10,643,383, the disclosures of which are incorporated by reference herein in their entireties. FIELD OF THE INVENTION The present invention relates generally to image processing and facial modeling, and more specifically to building three-dimensional (3D) models of a face using color and depth information. BACKGROUND OF THE INVENTION Image data describing an image can be associated with a depth map which describes the relative depth of pixels in the described image. Depth maps can be generated using camera systems such as array cameras. Meshes can be used in modeling 3D objects. A mesh is made of vertices, edges, and faces that define the shape of an object. Meshes can be deformed, or modified, by manipulating the characteristics of the vertices, edges and/or faces. Facial recognition can be performed on a face by comparing key facial features from an image of a face to a database of faces annotated with key facial features. SUMMARY OF THE INVENTION In an embodiment, a 3D facial modeling system includes a plurality of cameras configured to capture images from different viewpoints, a processor, and a memory containing a 3D facial modeling application and parameters defining a face detector, wherein the 3D facial modeling application directs the processor to obtain a plurality of images of a face captured from different viewpoints using the plurality of cameras, locate a face within each of the plurality of images using the face detector, wherein the face detector labels key feature points on the located face within each of the plurality of images, determine disparity between corresponding key feature points of located faces within the plurality of images, and generate a 3D model of the face using the depth of the key feature points. In another embodiment, the plurality of images is a stereo pair of images. In a further embodiment, the plurality of cameras includes a stereo pair of cameras. In still another embodiment, a first camera in the stereo pair of cameras is a wide angle camera, and a second camera in the stereo pair of cameras is a telephoto camera. In a still further embodiment, generating a 3D model of the face includes providing a convolutional neural network with the disparity between corresponding key feature points of located faces within the plurality of images. In yet another embodiment, generating a 3D model of the face further includes providing the convolutional neural network with at least one image from the plurality of images. In a yet further embodiment, the 3D facial modeling application further directs the processor to calculate the depth of at least one key feature point based on the disparity. In another additional embodiment, calculating the depth of at least one key feature point further includes using calibration data. In a further additional embodiment, wherein the 3D facial modeling application further directs the processor to animate the 3D model based on a second plurality of images of the face captured from different viewpoints using the plurality of cameras. In another embodiment again, generating a 3D model of the face includes matching a template mesh to the key feature points. In a further embodiment again, the template mesh is deformed to fit the key feature points. In still yet another embodiment, the 3D facial modeling application further directs the processor to generate a depth map for at least one image in the plurality of images. In a still yet further embodiment, the 3D facial modeling application further directs the processor to update the depth map based on the 3D model of the face. In still another additional embodiment, the 3D facial modeling application further directs the processor to locate a second face in the plurality of images, and generate a 3D model of the second face. In a still further additional embodiment, a method for generating a 3D model of a face using a 3D facial modeling system includes obtaining a plurality of images of a face captured from different viewpoints using a plurality of cameras, locating a face within each of the plurality of images using a face detector, wherein the face detector labels key feature points on the located face within each of the plurality of images, determining the disparity between corresponding key feature points of located faces within the plurality of images, and generating a 3D model of the face using the depth of the key feature points. In sti