CN-121996889-A - Diffusion model calculation method and device based on least square fitting and hardware unit

Abstract

The invention provides a diffusion model calculation method, a device and a hardware unit based on least square fitting, wherein the method comprises the steps of generating differential calculation reference data through the least square fitting method based on cached previous frame input data, a previous frame calculation result and original input data of a current time step; the method comprises the steps of carrying out structured sparse calculation according to original input data, differential calculation reference data and previous frame input data of a current time step to generate average error correction, carrying out calculation on the average error correction and the differential calculation reference data based on a linear layer of a diffusion model to generate current frame data, constructing a high-similarity differential calculation reference through least square fitting, improving quality and stability of the reference data, combining a dynamic threshold value and the structured sparse calculation, realizing remarkable reduction of calculation complexity on the premise of ensuring model precision, and continuously realizing high-efficiency and rapid calculation with low delay and low energy consumption in a modern diffusion model with increased inter-frame difference.

Inventors

• HAN MENG
• ZHU JIANFENG
• LIU LEIBO

Assignees

• 清华大学

Dates

Publication Date

20260508

Application Date

20251229

Claims (11)

1. 1. A diffusion model calculation method based on least squares fitting, the method comprising: Generating differential calculation reference data by a least square fitting method based on the cached previous frame input data, the previous frame calculation result and the original input data of the current time step; Carrying out structured sparse calculation according to the original input data, the differential calculation reference data and the input data of the previous frame of the current time step to generate an average error correction; and calculating the average error correction amount and the differential calculation reference data based on a linear layer of the diffusion model to generate current frame data, wherein the current frame data comprises current frame input data and a current frame calculation result.
2. 2. The diffusion model calculation method based on least squares fitting according to claim 1, wherein the generating differential calculation reference data by least squares fitting method based on the buffered previous frame input data, the previous frame calculation result, and the original input data of the current time step comprises: Generating a least square fitting coefficient according to the input data of the previous frame and the original input data of the current time step through the constructed optimized objective function; And generating differential calculation reference data according to the least square fitting coefficient, the input data of the previous frame and the calculation result of the previous frame.
3. 3. The least squares fitting based diffusion model calculation method according to claim 1, wherein the performing structured sparse calculation based on the original input data, the differential calculation reference data, and the previous frame input data of the current time step, generating an average error correction amount, comprises: generating a data differential matrix according to the differential calculation reference data and the original input data of the current time step; performing grouping column-by-column structured sparse calculation on the data differential matrix to generate a sparse discrimination index; And carrying out grouping progressive structured sparse calculation on the data differential matrix based on the sparse discrimination index, the original input data of the current time step and the input data of the previous frame, and generating the average error correction.
4. 4. The method for computing a diffusion model based on least squares fitting according to claim 3, wherein the generating the average error correction amount based on the sparse discriminant indicator, the original input data of the current time step, and the input data of the previous frame includes: generating a dynamic threshold according to the original input data of the current time step and the input data of the previous frame; based on the sparse discrimination index and a dynamic threshold, carrying out structured sparse discrimination on the data differential matrix to generate a sparse mask; And based on the sparse mask, carrying out grouping and line-by-line structured sparse calculation on the data differential matrix to generate the average error correction.
5. 5. The least squares fitting based diffusion model calculation method of claim 1, wherein the differential calculation reference data comprises differential calculation reference input data and differential calculation reference similarity results; The linear layer based on the diffusion model calculates the average error correction amount and the differential calculation reference data to generate current frame data, and the method comprises the following steps: Calculating reference input data according to the average error correction amount and the difference, and generating current frame input data; and calculating the difference calculation reference similarity result and the average error correction based on a linear layer of the diffusion model to generate a current frame calculation result.
6. 6. The least squares fitting based diffusion model calculation method of claim 1, further comprising: and determining the current frame input data as the previous frame input data, determining the current frame calculation result as the previous frame calculation result, determining the input data of the next time step as the original input data of the current time step, continuously executing the buffer-based previous frame input data, the previous frame calculation result and the original input data of the current time step, and generating differential calculation reference data through a least square fitting method.
7. 7. A hardware unit, characterized in that it is applied to the least squares fitting-based diffusion model calculation method according to any one of claims 1 to 6, and comprises a plurality of vector subtractors, a plurality of vector multipliers, a vector absolute value device, a plurality of addition trees, a maximum tree, a plurality of adders, a plurality of multipliers, comparators and dividers.
8. 8. A least squares fitting based diffusion model calculation apparatus applied to the hardware unit of claim 7, the apparatus comprising: The reference data calculation module is used for generating differential calculation reference data through a least square fitting method based on the cached previous frame input data, the previous frame calculation result and the original input data of the current time step; The structured sparse calculation module is used for carrying out structured sparse calculation according to the original input data of the current time step, the differential calculation reference data and the input data of the previous frame to generate an average error correction; And the linear calculation module is used for calculating the average error correction quantity and the differential calculation reference data based on a linear layer of the diffusion model to generate current frame data, wherein the current frame data comprises current frame input data and a current frame calculation result.
9. 9. A computer readable medium having stored thereon a computer program, which when executed by a processor implements the least squares fitting based diffusion model calculation method according to any of claims 1 to 6.
10. 10. A computer device comprising a memory for storing information including program instructions and a processor for controlling execution of the program instructions, wherein the program instructions when loaded and executed by the processor implement the least squares fitting based diffusion model calculation method of any one of claims 1 to 6.
11. 11. A computer program product comprising computer programs/instructions which, when executed by a processor, implement the least squares fitting based diffusion model calculation method of any one of claims 1 to 6.

Description

Diffusion model calculation method and device based on least square fitting and hardware unit Technical Field The present invention relates to the field of computer technologies, and in particular, to a method, an apparatus, and a hardware unit for computing a diffusion model based on least squares fitting. Background In recent years, the generated neural network has been widely used in the fields of image generation, video generation, multi-modal modeling, and the like. Among them, diffusion models have become one of the mainstream generation schemes because of the outstanding expression in terms of generation quality and diversity. With the development of deep learning technology, the infrastructure of the diffusion model also gradually evolves from a convolutional neural network based on U-Net to a diffusion transducer model based on transducer (DiT). However, the DiT model needs to repeatedly perform forward computation on a plurality of time steps in the reasoning process, so that the reasoning delay and the energy consumption of the model are obviously improved, and the application requirements of real-time or low-power consumption scenes are difficult to meet. In order to alleviate the above-mentioned problems, an acceleration method based on differential calculation is proposed in the prior art. The method utilizes the similarity of input data between adjacent time steps, takes the input of the previous time step and the output of the corresponding layer as a reference, and approximately obtains the current output by calculating the difference between the current input and the previous input. Since the data change of adjacent time steps in the traditional multi-step diffusion is small, the numerical value of most elements in the difference is close to zero, the method can reduce the operation amount by means of low-precision calculation or sparse calculation, and therefore the calculation efficiency is improved. However, the above-described differential calculation method is highly dependent on the high similarity between the input data of the adjacent time steps. With the development of diffusion model technology, the methods of few-step iteration, model distillation and the like have greatly reduced the sampling steps from more than the traditional hundred steps to less than twenty steps, so that the data difference between adjacent time steps is obviously increased. Under the condition, the difference calculation method directly based on the previous time step is excessively large in difference between the reference and the current input, so that the difference no longer has obvious sparsity or low-value characteristics, the acceleration effect of the difference is rapidly reduced, even the difference is frequently degraded into full-scale calculation, and the effective acceleration ratio cannot be maintained in a modern diffusion model with reduced iteration steps. Therefore, the prior art still lacks a high-efficiency acceleration scheme capable of effectively mining the data correlation among iterations, maintaining the calculation efficiency and having the hardware expandability under the condition that the time step difference is obviously increased. Disclosure of Invention The invention aims to provide a diffusion model calculation method based on least square fitting, which is used for constructing a high-similarity differential calculation benchmark through least square fitting, effectively improving the quality and stability of benchmark data, combining dynamic threshold and structured sparse calculation, realizing remarkable reduction of calculation complexity on the premise of ensuring model precision, and continuously realizing high-efficiency and rapid calculation with low delay and low energy consumption in a modern diffusion model with increased inter-frame difference. It is a further object of the invention to provide a hardware unit. It is another object of the present invention to provide a diffusion model calculation device based on least squares fitting. It is yet another object of the present invention to provide a computer readable medium. It is a further object of the invention to provide a computer device. In order to achieve the above object, the present invention discloses a diffusion model calculation method based on least squares fitting, comprising: Generating differential calculation reference data by a least square fitting method based on the cached previous frame input data, the previous frame calculation result and the original input data of the current time step; Carrying out structured sparse calculation according to the original input data, the differential calculation reference data and the input data of the previous frame of the current time step to generate an average error correction; And calculating the average error correction amount and the differential calculation reference data based on the linear layer of the diffusion model to generate current frame