CN-121580796-B - 3D-GAN small model incremental training and optimizing method based on retrieval enhancement

CN121580796BCN 121580796 BCN121580796 BCN 121580796BCN-121580796-B

Abstract

The invention discloses a 3D-GAN small model incremental training and optimizing method based on retrieval enhancement, and relates to the technical field of three-dimensional generation countermeasure. According to the method, after a 3D model generation task is received, at least one 3D sample model similar to the generation task in geometric structure and/or texture characteristics is searched in real time from a 3D sample library to serve as a reference sample, then a high-dimensional characteristic representation of the reference sample is migrated to a 3D-GAN small model with the total parameter amount smaller than 10 hundred million through knowledge distillation technology, meanwhile, an adaptive loss adjustment strategy is introduced in the migration process to dynamically monitor the incremental training process of the small model and evaluate fitting risks, then the weight of a loss function is automatically adjusted according to a risk evaluation result, after the incremental training is completed, hardware adaptation optimization for a target chip is conducted on the optimized small model, and a final 3D-GAN small model which is used for light-weight deployment on the target chip is generated, so that the problems of high efficiency-quality paradox and high deployment cost can be effectively solved.

Inventors

WU WENJUAN
LIU YONGGANG
JIAO MIN

Assignees

中国人民大学

Dates

Publication Date: 20260512
Application Date: 20251118

Claims (9)

1. The 3D-GAN small model incremental training and optimizing method based on retrieval enhancement is characterized by comprising the following steps of: Constructing and maintaining a 3D sample library, wherein the 3D sample library comprises 3D sample models of various types generated by a 3D-GAN large model with the total parameter of not less than 10 hundred million or artificially designed and high-dimensional characteristic representations of the 3D sample models; after receiving a 3D model generation task, according to task content and a generation target carried in the 3D model generation task, retrieving at least one 3D sample model similar to the 3D model generation task in geometric structure and/or texture characteristics from the 3D sample library in real time as a reference sample; Migrating the high-dimensional characteristic representation of the reference sample into a 3D-GAN small model with a total parameter of less than 10 hundred million by a knowledge distillation technology so as to enhance the extraction and reconstruction capability of the 3D-GAN small model on detail characteristics; Introducing an adaptive loss adjustment strategy to dynamically monitor an incremental training process of the 3D-GAN small model and evaluate the fitting risk in the migration process through the knowledge distillation technology, and then automatically adjusting the weight of a loss function according to a risk evaluation result to balance the reduction degree of generated details and the diversity of the generated result, wherein the adaptive loss adjustment strategy comprises the steps of periodically calculating the diversity index of the generated result in the incremental training process, wherein the diversity index comprises the difference degree between generated samples and/or the repetition rate between the generated samples and a training data set, judging that the fitting risk exists if the index value of any index in the diversity index is lower than the corresponding preset index threshold according to a plurality of preset index thresholds which are in one-to-one correspondence, and automatically reducing the weight of the distillation loss function and/or the weight of the reconstruction loss function when the fitting risk is judged to exist so as to encourage the 3D-GAN small model to generate more diversified output; And after the incremental training is completed, performing hardware adaptation optimization for the optimized 3D-GAN small model facing to a target chip, and generating a final 3D-GAN small model for light deployment on the target chip.
2. The 3D-GAN small model incremental training and optimization method of claim 1, wherein retrieving in real time at least one 3D sample model from the 3D sample library that is similar in geometry and/or texture features to the 3D model generation task based on task content and generation targets carried in the 3D model generation task, comprises: extracting task semantic descriptions and/or low-dimensional feature vectors used as initial input items of the model from the 3D model generation task to serve as task content and generation targets; calculating to obtain similarity between the task semantic description and/or the low-dimensional feature vector and the corresponding high-dimensional feature representation for each 3D sample model in the 3D sample library; And screening up to K3D sample models from the 3D sample library according to a preset similarity threshold value and/or a Top-K sorting strategy based on similarity to serve as at least one 3D sample model similar to the 3D model generating task in terms of geometric structure and/or texture characteristics, wherein K represents a positive integer.
3. The 3D-GAN small model incremental training and optimization method of claim 1, wherein retrieving in real time at least one 3D sample model from the 3D sample library that is similar in geometry and/or texture features to the 3D model generation task based on task content and generation targets carried in the 3D model generation task, comprises: for each 3D sample model in the 3D sample library, dimension-reducing the corresponding high-dimensional feature representation to obtain a corresponding first vector, and adding the first vector to a vector library; extracting task semantic descriptions and/or low-dimensional feature vectors used as initial input items of the model from the 3D model generation task to serve as task content and generation targets; Converting the task semantic description and/or the low-dimensional feature vector into a second vector, and adopting an ANN (automatic analysis network) searching algorithm to search M third vectors which are most similar to the second vector from the vector library, wherein M represents a positive integer; And screening M3D sample models which are in one-to-one correspondence with the M third vectors from the 3D sample library to serve as at least one 3D sample model which is similar to the 3D model generating task in geometric structure and/or texture characteristics.
4. The 3D-GAN small model incremental training and optimization method of claim 1 wherein migrating the high-dimensional feature representation of the reference sample into a 3D-GAN small model with a total of less than 10 billions of parameters by knowledge distillation techniques comprises: Constructing a distillation loss function which is represented by high-dimensional features of the reference sample as a 'teacher signal' and is represented by non-input layer features of a 3D-GAN small model with a total parameter of less than 10 hundred million as a 'student signal', wherein the distillation loss function comprises content loss for restricting visual fidelity of a generated model, style loss for restricting style features and structural loss for restricting a topological structure of the 3D model, and the non-input layer features comprise middle layer features and/or output layer features; By minimizing the distillation loss function, the 3D-GAN small model is forced to learn and approximate the detailed features of the reference sample.
5. The 3D-GAN small model incremental training and optimization method according to claim 1, wherein performing hardware adaptation optimization for the optimized 3D-GAN small model toward a target chip comprises: converting the optimized computational graph structure of the 3D-GAN small model into an intermediate representation format supported by a target chip; and replacing and optimizing the computation intensive operators in the optimized 3D-GAN small model based on the intermediate representation format by utilizing a special operator library provided by the target chip so as to improve the reasoning efficiency of the final 3D-GAN small model on the target chip.
6. The 3D-GAN small model incremental training and optimizing device based on retrieval enhancement is characterized by comprising a 3D sample library operation and maintenance unit, a reference sample retrieval unit, a knowledge distillation migration unit, an automatic loss adjustment unit and a hardware adaptation optimizing unit which are sequentially connected in a communication mode; the 3D sample library operation and maintenance unit is used for constructing and maintaining a 3D sample library, wherein the 3D sample library comprises various 3D sample models and high-dimensional characteristic representations of the 3D sample models, wherein the 3D sample models are generated by a 3D-GAN large model with the total parameter amount not less than 10 hundred million or are obtained through manual design; the reference sample retrieval unit is used for retrieving at least one 3D sample model similar to the 3D model generation task in geometric structure and/or texture characteristics from the 3D sample library in real time as a reference sample according to task content and a generation target carried in the 3D model generation task after receiving the 3D model generation task; The knowledge distillation migration unit is used for migrating the high-dimensional characteristic representation of the reference sample into a 3D-GAN small model with the total parameter amount less than 10 hundred million through a knowledge distillation technology so as to enhance the extraction and reconstruction capability of the 3D-GAN small model on detail characteristics; The automatic loss adjustment unit is used for dynamically monitoring an incremental training process of the 3D-GAN small model and evaluating the fitting risk in the migration process through the knowledge distillation technology, and then automatically adjusting the weight of a loss function according to a risk evaluation result so as to balance the reduction degree of the generated details and the diversity of the generated result, wherein the adaptive loss adjustment strategy comprises the steps of periodically calculating the diversity index of the generated result in the incremental training process, wherein the diversity index comprises the difference degree between generated samples and/or the repetition rate between the generated samples and a training data set, judging that the fitting risk exists according to a plurality of preset index thresholds which are in one-to-one correspondence with the diversity index, and automatically reducing the weight of the distillation loss function and/or the weight of the reconstructed loss function when judging that the fitting risk exists so as to encourage the 3D-GAN small model to generate more diversified output; the hardware adaptation optimization unit is used for carrying out hardware adaptation optimization on the optimized 3D-GAN small model facing to a target chip after incremental training is completed, and generating a final 3D-GAN small model for light deployment on the target chip.
7. The computer equipment is characterized by comprising a storage module, a processing module and a receiving and transmitting module which are sequentially connected in a communication mode, wherein the storage module is used for storing a computer program, the receiving and transmitting module is used for receiving and transmitting messages, and the processing module is used for reading the computer program and executing the 3D-GAN small model incremental training and optimizing method according to any one of claims 1-5.
8. A computer readable storage medium having instructions stored thereon which, when executed on a computer, perform the 3D-GAN small model incremental training and optimization method of any of claims 1-5.
9. A computer program product comprising a computer program or instructions which, when executed by a computer, implement the 3D-GAN small model incremental training and optimization method of any of claims 1-5.

Description

3D-GAN small model incremental training and optimizing method based on retrieval enhancement Technical Field The invention belongs to the technical field of three-dimensional generation countermeasure, and particularly relates to a 3D-GAN small model incremental training and optimizing method based on retrieval enhancement. Background Three-dimensional generation countermeasure network (3D Generative Adversarial Networks, abbreviated as 3D-GAN) technology is the leading direction in the fields of computer vision and computer graphics, and has great application potential in digital entertainment, industrial design, automatic driving simulation and other scenes. However, the current 3D-GAN technology in practice has a deep dilemma of "efficiency" and "quality" which are difficult to be compatible, which severely restricts the industrial application thereof, especially for small and medium enterprises with limited budgets. First, the prior art has a fundamental bottleneck in terms of model representation methods and rendering efficiency. Mainstream 3D representation methods such as Voxel Grid (Voxel Grid) and neural implicit representation (Neural Implicit Representation) each have significant drawbacks. Voxel grid representation methods rely on discrete three-dimensional grids, which when resolution is increased to pursue finer generation quality, require exponentially increasing memory overhead, resulting in models that cannot be trained and deployed on conventional hardware. In order to avoid the memory bottleneck, although the neural implicit expression method (such as NeRF) can realize continuous high-quality expression, the rendering process needs to perform point-by-point inquiry and evaluation on points in the space, so that the calculation amount is huge, the rendering speed is extremely slow, the real-time interaction standard of 5 frames per second (< 5 fps) is generally difficult to reach, and the performance requirements of applications such as real-time generation and virtual reality cannot be met. Second, current solutions are costly and present supply chain risks in terms of hardware deployment and ecological adaptation. Most advanced 3D-GAN models are developed and optimized around foreign GPUs (Graphics Processing Unit, graphics processors), and lack optimization of the underlying hardware adaptations of chips such as homemade rising (assnd). The method leads users to face two major problems when in deployment, namely, the users need to carry out additional and costly transplanting adaptation work, and the users depend on foreign GPU hardware schemes for a long time, so that the high purchase and operation cost pushes up the whole deployment cost, and the technical application threshold is high. Furthermore, in terms of model weight reduction and precision balance, the prior art path fails to effectively solve the core contradiction. To reduce deployment costs, industry has attempted to employ miniaturized models (i.e., small models) with parameters of less than 10 billion. However, due to the limited capacity of the model, the small model has a natural short plate in the capability of extracting and reconstructing detail characteristics, and the precision, the richness and the fidelity of the generated model are obviously different from those of a superior large model (namely, a large model) with hundreds of billions or even billions of parameters. The sharp contradiction between low cost and high precision makes the existing small model scheme difficult to meet the core requirements of small and medium enterprises on technology. In summary, an innovative solution is needed in the existing 3D-GAN technology to overcome the bottleneck of memory and computing efficiency, achieve efficient adaptation with the domestic computing infrastructure, and fundamentally improve the generation quality of the small parameter model, so as to break the "efficiency-quality" paradox which plagues the development of industry for a long time. Disclosure of Invention The invention aims to provide a retrieval enhancement-based 3D-GAN small model incremental training and optimizing method, a retrieval enhancement-based 3D-GAN small model incremental training and optimizing device, computer equipment, computer readable storage media and computer program products, which are used for solving the problem that the conventional 3D-GAN technology severely restricts industrialized application due to the double dilemma that the efficiency and the quality are difficult to be compatible in practice. In order to achieve the above purpose, the present invention adopts the following technical scheme: in a first aspect, a 3D-GAN small model incremental training and optimization method based on search enhancement is provided, including: Constructing and maintaining a 3D sample library, wherein the 3D sample library comprises 3D sample models of various types generated by a 3D-GAN large model with the total parameter of not less t