KR-102963100-B1 - DEEP LEARNING METHOD FOR DETECTING ERROR AND CORRECTION OF 3D FACE-SCAN MESH BASED ON UV-SPACE TRIANGLE REPROJECTION

Abstract

A machine learning method for error detection in data modeled from a mesh is disclosed. The machine learning method comprises the steps of preparing normal and texture data by normal mapping the mesh, selecting a triangle in UV space by normal mapping, generating transformed data in which the normal and texture data are normalized into a rectangle by reprojecting the selected triangle and three adjacent triangles into a predefined rectangle, and inputting the transformed data into a deep learning network. The present invention provides an effective learning method for verifying errors in a 3D model image and can generate an AI model with excellent performance.

Inventors

• 정진수
• 유균형

Assignees

• 한국전자기술연구원

Dates

Publication Date

20260511

Application Date

20241210

Claims (12)

1. In a machine learning method for error detection of data modeled as a mesh, Step of preparing normal and texture data by normal mapping the mesh; A step of selecting a triangle in UV space by the above normal mapping; A step of generating transformed data in which the normal and texture data are normalized to the rectangle by reprojecting the selected triangle and three adjacent triangles onto a predefined rectangle; and A machine learning method comprising the step of inputting the above-mentioned transformed data into a deep learning network.
2. In paragraph 1, A machine learning method comprising the above transformation data, 3-channel normal data reprojected into the rectangle, 3-channel texture data, 1-channel triangle mask data identifying a triangle, and 1-channel position mask data representing a region of the mesh.
3. In paragraph 2, The above deep learning network is a machine learning method, which is a U-net.
4. In paragraph 3, The above mesh is a machine learning method in which a mesh is a 3D modeled mesh obtained by scanning a face.
5. In paragraph 4, A machine learning method for learning the classification of valid or incorrect normal data from a feature map extracted at the end of the contracting path of the above U-net.
6. In paragraph 5, The step of inputting the above-mentioned transformed data into a deep learning network is, The method includes the step of deleting the normal data of the selected triangle of the transformed data and inputting it into the deep learning network. A machine learning method for learning the deleted normal data from the output of the last layer of the expansion path of the above U-net.
7. A method for determining and restoring errors in a 3D face scan mesh using an AI model trained by the machine learning method of paragraph 6, A step of inputting transformation data into the AI model, wherein one triangle and three adjacent triangles are reprojected into a predefined rectangle from the normal mapped normal and texture data of the 3D face scan mesh; and A method for error determination and restoration of a 3D face scan mesh, comprising the step of estimating validity or error from a feature map extracted from the deepest layer (bottleneck) of the AI model.
8. In Paragraph 7, A step of deleting the normal data of the target triangle estimated as an error by the above AI model and inputting it into the above AI model; A step of estimating normal data of the target triangle by the AI model; and A method for error determination and restoration of a 3D face scan mesh, comprising the step of generating a restored image by inserting the above-mentioned estimated normal data.
9. In a machine learning computing system for error detection of data modeled as a mesh, An input unit that receives normal and texture data obtained by normal mapping the mesh; and Select a triangle in UV space based on the above normal mapping, and By reprojecting the selected triangle and three adjacent triangles onto a predefined rectangle, transform data is generated in which the normal and texture data are normalized to the rectangle, and A computing system comprising: a processor that inputs the above-mentioned transformed data into a deep learning network to perform machine learning.
10. In Paragraph 9, A computing system comprising the above transformation data, 3-channel normal data reprojected into the rectangle, 3-channel texture data, 1-channel triangle mask data identifying triangles, and 1-channel position mask data representing parts of the mesh.
11. In Paragraph 10, The deep learning network mentioned above is U-net, and The above mesh is a computing system that is a mesh modeled in 3D by scanning a face.
12. In Paragraph 11, The above processor is, Performing an operation to learn the classification of valid or erroneous normal data from the feature map extracted at the bottleneck at the end of the contracting path of the above U-net, A computing system that deletes the normal data of the selected triangle of the transformed data and inputs it into the deep learning network, and performs an operation to learn the deleted normal data from the output of the last layer of the expansion path of the U-net.

Description

Deep Learning Method for Detecting Error and Correction of 3D Face-Scan Mesh Based on UV-Space Triangle Reprojection The present invention relates to a machine learning method capable of detecting errors in mesh data, and specifically, to a deep learning method capable of improving error detection performance through learning by reprojecting triangles in UV space that have been normal-mapped to the mesh. Normal mapping is a method of expressing detailed undulations or textures within a polygon by representing the values of texels applied to the polygon surface as normal vectors. Referring to FIG. 1, U and V vector values are provided along axes named T (Tangent) and B (Binormal) that correspond to object space represented by XYZ location coordinates (or local coordinates). In the illustrated example, vertices at three locations P1, P2, and P3 have texture coordinate values ( U1 , V1 ), ( U2 , V2), and ( U3 , V3 ), and the two edges of the triangle E1 and E2 can be represented as ΔU1 , ΔU2 , ΔV1 , and ΔV2 . The xyz local space and the tangent space or UV space can be converted to each other using linear determinants. When generating texture and normal data in a defined format within a mesh-based UV space using 3D reconstruction technology on images of faces scanned by a multi-view camera, highly detailed and realistic data that is nearly photographic can be obtained. However, errors may exist in some data due to errors in the captured data or errors in merging multi-view data. Although interpolation using adjacent pixel data is generally used to correct these errors, there was a problem in that it was difficult to create natural-looking data that is similar to data created based on actual captured data. Existing deep learning-based image segmentation and inpainting can produce results quickly by learning from large amounts of data, but there were some tricky or inefficient aspects when applied to tasks such as finding or correcting errors within texture and normal data defined in the UV space of a 3D mesh. First, in the case of photorealistic texture and normal data, the image resolution is very high, some spaces contain empty data, and the ratio of valid texture and normal data to error data is very high, so it is inefficient to train the network by feeding the entire dataset as input at once. In addition, after the scanning and 3D reconstruction process, it is impossible to know where errors will occur, and because the range and shape of the error data generated in UV space are irregular, a large amount of actual scan data had to be secured to build the dataset for training the deep learning network. FIG. 1 is a diagram illustrating conventional normal mapping; FIG. 2 is a block diagram showing the configuration of a computing system according to an embodiment of the present invention; FIG. 3 is a block diagram showing a specific configuration of a computing system according to an embodiment of the present invention; FIG. 4 is a block diagram showing a software module stored in the storage unit of the computing system of FIG. 3; FIG. 5 is a diagram illustrating the input data transformation of a deep learning network according to an embodiment of the present invention; FIG. 6 is a diagram illustrating the learning of a deep learning network according to an embodiment of the present invention; FIG. 7 is a drawing for explaining two embodiments utilizing a model trained with a deep learning network according to one embodiment of the present invention; FIG. 8 is a flowchart illustrating a machine learning method for error detection of data modeled with a mesh according to an embodiment of the present invention; and, Figure 9 is a flowchart illustrating a method for determining and restoring errors in a 3D face scan mesh using an AI model trained by a machine learning method. The present invention will be described in more detail below with reference to the drawings. Furthermore, in describing the present invention, detailed descriptions of related known functions or configurations are omitted if it is determined that such detailed descriptions would unnecessarily obscure the essence of the invention. Additionally, the terms described below are defined considering their functions in the present invention, and these may vary depending on the intentions or relationships of the user or operator. Therefore, their definitions should be based on the content throughout this specification. FIG. 2 is a block diagram showing the configuration of a computing system according to one embodiment of the present invention. Referring to FIG. 2, a machine learning computing system (100) for error detection of data modeled as a mesh includes an input unit (110) and a processor (120). The input unit (110) receives normal and texture data that has been normal-mapped from a mesh. The input unit (110) may be configured as a communication unit connected to another device of the computing system (100) or a camera scanner. Alternativel