CN-121996412-A - Method and device for generating execution sequence load of large model, method for simulation training of large model and electronic equipment

CN121996412ACN 121996412 ACN121996412 ACN 121996412ACN-121996412-A

Abstract

The application relates to the technical field of artificial intelligence and discloses a generation method for a large model execution sequence load, which comprises the steps of constructing a program control flow diagram corresponding to a large model training program; the method comprises the steps of carrying out data flow analysis on a program control flow graph based on large model execution parameters, eliminating deterministic conditional branches in the program control flow graph, statically traversing large model execution paths after eliminating deterministic conditional branches in the program control flow graph, determining target paths of non-deterministic conditional branches in the program control flow graph, obtaining an execution path sequence of the program control flow graph, and generating a large model execution sequence load based on the execution path sequence of the program control flow graph. By combining static analysis and heuristic path selection during compiling, the accuracy of executing sequence load by a large model generated during complex conditional branching is improved. The application also discloses a generating device for the execution sequence load of the large model, a method for simulation training of the large model and electronic equipment.

Inventors

LI LUN
ZHAO JUNFANG
WANG YAHAN
CUI JIARUI
CHEN WENGUANG

Assignees

中国信息通信研究院
清华大学

Dates

Publication Date: 20260508
Application Date: 20251230

Claims (10)

1. A method for generating a large model execution sequence load, comprising: constructing a program control flow diagram corresponding to the large model training program; Carrying out data flow analysis on the program control flow graph based on the execution parameters of the large model, and eliminating deterministic condition branches in the program control flow graph; After deterministic conditional branches in a program control flow diagram are eliminated, statically traversing a large model execution path, determining a target path of an uncertain conditional branch in the program control flow diagram, and obtaining an execution path sequence of the program control flow diagram; and generating a large model execution sequence load based on the execution path sequence of the program control flow graph.
2. The generating method according to claim 1, wherein the deterministic conditional branches comprise constant conditional branches and/or cyclic conditional branches; performing data flow analysis on the program control flow graph based on the large model execution parameters, eliminating deterministic conditional branches in the program control flow graph, including: Under the condition that the deterministic conditional branch comprises a constant conditional branch, carrying out data flow analysis on a program control flow graph based on large model execution parameters, and eliminating the constant conditional branch in the program control flow graph through constant propagation and constant folding; In the case where the deterministic conditional branch includes a loop conditional branch, performing data flow analysis on the program control flow graph based on the large model execution parameters, and eliminating the loop conditional branch in the program control flow graph by expanding a loop whose number of loops in the program control flow graph is constant.
3. The method of generating according to claim 2, wherein, Performing data flow analysis on the program control flow graph based on large model execution parameters, and eliminating constant condition branches in the program control flow graph through constant propagation and constant folding, wherein the method comprises the following steps: Determining a constant value in a large model training program based on the large model execution parameters; Propagating the constant values to each node in the program control flow graph, and compiling and calculating expressions consisting of the constant values; Based on a deterministic result obtained by compiling calculation, eliminating constant condition branches in a program control flow graph; Performing data flow analysis on the program control flow graph based on the execution parameters of the large model, and expanding the loops with the constant circulation times in the program control flow graph to eliminate the circulation condition branches in the program control flow graph, wherein the method comprises the following steps: Identifying a circulation structure in the program control flow graph, and determining the circulation times of the circulation structure based on the execution parameters of the large model; And copying the circulation volume of the circulation structure for circulation times, and converting circulation control dependence into sequence dependence so as to eliminate circulation condition branches in the program control flow diagram.
4. A method of generating as claimed in any one of claims 1 to 3, wherein statically traversing the large model execution path, determining the target path of the unconfirmed conditional branch in the program control flow graph comprises: Statically traversing a large model execution path, and determining an undetermined conditional branch in a program control flow graph; The target path in the non-deterministic conditional branch is determined using a branch selection algorithm.
5. The method of generating of claim 4, wherein determining the target path in the non-deterministic conditional branch using a branch selection algorithm comprises: calculating the dominance relation of the program control flow graph; Determining a key merging node corresponding to a conditional branch node in the uncertain conditional branch based on a dominance relation of a program control flow graph, wherein the key merging node is a necessary node from the conditional branch node to a program outlet, and the key merging node is a nearest node from the conditional branch node; calculating path weights of execution paths from the conditional branch nodes to the critical junction nodes; And selecting a target path according to the path weight.
6. The generation method according to claim 5, wherein calculating path weights of execution paths from the conditional branch node to the critical junction node comprises: determining the operation weight of each basic block on the execution path, wherein the operation weight is associated with the operation type and the operation scale of the basic block; Path weights are calculated based on the operational weights.
7. The generation method according to claim 5, wherein selecting the target path according to the path weight comprises: under the condition that the weight difference of the plurality of path weights is larger than or equal to a preset weight difference threshold value, selecting an execution path with the largest path weight as a target path; And selecting an execution path containing the most calculation operations as a target path under the condition that the weight difference of the path weights is smaller than a preset weight difference threshold value.
8. A method for large model simulation training, comprising: Simulation training of a large model using a large model execution sequence load according to any of claims 1 to 7.
9. A generation apparatus for large model execution sequence loads, comprising: the construction module is configured to construct a program control flow diagram corresponding to the large model training program; the elimination module is in communication connection with the construction module and is configured to perform data flow analysis on the program control flow graph based on the large model execution parameters and eliminate deterministic conditional branches in the program control flow graph; The determining module is in communication connection with the eliminating module and is configured to eliminate deterministic conditional branches in the program control flow graph, then statically traverse the execution path of the large model, determine target paths of non-deterministic conditional branches in the program control flow graph and obtain an execution path sequence of the program control flow graph; and the generating module is in communication connection with the determining module and is configured to generate a large model execution sequence load based on the execution path sequence of the program control flow graph.
10. An electronic device comprising a processor, a memory, and a computer program stored on the memory, the processor, when executing the computer program, implementing: The method for generating a large model execution sequence load according to claim 1 to 7, or The method for large model simulation training of claim 8.

Description

Method and device for generating execution sequence load of large model, method for simulation training of large model and electronic equipment Technical Field The application relates to the technical field of artificial intelligence, in particular to a method and a device for generating a large model execution sequence load, a method for large model simulation training and electronic equipment. Background The large language model (Large Language Models, LLMs) is a revolutionary technological breakthrough in the field of artificial intelligence, and by virtue of its powerful natural language understanding, generating and multi-modal interaction capabilities, the working mode and life style of humans are being deeply remodeled. The core driving force for the breakthrough of the technology is derived from massive data-based large-scale parameter training, wherein the model parameters are usually in the order of hundreds of billions or even trillions, and large-scale high-performance computing resources (comprising GPU clusters and special AI accelerators) are required to be used for executing continuous training processes for weeks or even months. Such large-scale, time-consuming and costly training tasks make performance characteristic analysis and optimization of the training process a critical bottleneck problem. The training performance is effectively understood, predicted and optimized, and the method has important significance in reducing training cost, improving resource utilization rate and accelerating model iteration. The large model training simulation tool performs full stack simulation and prediction on the whole training process before inputting actual hardware resources through a software virtualization technology, and provides key decision support for a training system designer. In a typical large model training simulation system, the load of an execution sequence and the configuration of target hardware jointly form the core input of a simulation module, and the key performance index output such as training time, resource utilization rate and the like is obtained through simulation execution. One of the main methods of generating execution sequence loads in the current industry is a Pre-execution generation technology, and the basic principle of the method is that in a training software framework (such as PyTorch, tensorFlow), through a Pre-execution model training process, an actually executed operation instruction sequence is dynamically captured and recorded, and is converted into a load description available to a simulation system. In the process of implementing the embodiments of the present disclosure, it is found that at least the following problems exist in the related art: When the pre-execution method is used for coping with complex conditional branches, the generated execution sequence load has insufficient accuracy. It should be noted that the information disclosed in the above background section is only for enhancing understanding of the background of the application and thus may include information that does not form the prior art that is already known to those of ordinary skill in the art. Disclosure of Invention The following presents a simplified summary in order to provide a basic understanding of some aspects of the disclosed embodiments. This summary is not an extensive overview, and is intended to neither identify key/critical elements nor delineate the scope of such embodiments, but is intended as a prelude to the more detailed description that follows. The embodiment of the disclosure provides a generation method and device for large model execution sequence load, a method for large model simulation training and electronic equipment, so as to improve the accuracy of the generated execution sequence load when complex conditional branches are handled. In some embodiments, the method for generating the large model execution sequence load comprises the steps of constructing a program control flow graph corresponding to a large model training program, carrying out data flow analysis on the program control flow graph based on large model execution parameters, eliminating deterministic conditional branches in the program control flow graph, carrying out static traversal on a large model execution path after eliminating deterministic conditional branches in the program control flow graph, determining a target path of an uncertain conditional branch in the program control flow graph, and obtaining an execution path sequence of the program control flow graph, and generating the large model execution sequence load based on the execution path sequence of the program control flow graph. Optionally, the deterministic conditional branches comprise constant conditional branches and/or loop conditional branches; the method comprises the steps of carrying out data flow analysis on a program control flow graph based on large model execution parameters, eliminating deterministic condition