CN-121984867-A - Data arrangement method and device for operation intention

CN121984867ACN 121984867 ACN121984867 ACN 121984867ACN-121984867-A

Abstract

The invention provides a data arrangement method and device for an operation intention, the method comprises the steps of compiling the operation intention to obtain a multi-level arrangement strategy, wherein the multi-level arrangement strategy comprises at least two of an application layer arrangement strategy, a transmission layer arrangement strategy and a system storage layer arrangement strategy, and data transmission operation is executed based on the multi-level arrangement strategy. According to the method provided by the invention, the operation intention containing global requirements and constraints is obtained and compiled into the multi-level arrangement strategy covering a plurality of layers such as application, transmission and storage, and finally the data transmission operation is executed based on the strategy, so that the full-link resource allocation can be flexibly adjusted according to the operation intention, the problems of downstream data receiving blockage and the like caused by mismatching of the network and the storage performance are effectively solved, the service level target is ensured, the complex constraints are met, and the comprehensive performance and the resource utilization rate of data transmission are obviously improved.

Inventors

ZHANG WEI
Shen linjiang
CUI CHAO

Assignees

浪潮通信信息系统有限公司

Dates

Publication Date: 20260505
Application Date: 20251212

Claims (10)

1. A data arrangement method for a job intention, comprising: Acquiring the operation intention of a data transmission operation; Compiling the operation intention to obtain a multi-level arrangement strategy, wherein the multi-level arrangement strategy comprises at least two of an application layer arrangement strategy, a transmission layer arrangement strategy and a system storage layer arrangement strategy; and executing the data transmission job based on the multi-level arrangement strategy.
2. The job intent-oriented data orchestration method according to claim 1, wherein the performing the data transfer job based on the multi-level orchestration policy comprises: step S1, respectively injecting the multi-level arrangement strategies into levels corresponding to the multi-level arrangement strategies, so that each level performs data transmission according to the level arrangement strategies; Step S2, in the data transmission process, collecting multi-level observation signals; Step S3, evaluating the multi-level observation signals, taking the level observation signals with evaluation results not meeting preset conditions as target observation signals, and adjusting the level arrangement strategy based on the target observation signals to obtain an optimized level arrangement strategy; and S4, taking the optimized hierarchical arrangement strategy as an updated multi-level arrangement strategy, and iteratively executing the steps S1 to S3 until the data transmission operation is completed.
3. The method for organizing data for a job intent according to claim 2, wherein said adjusting the hierarchical organization policy based on the target observation signal results in an optimized hierarchical organization policy, comprising: Under the condition that the target observation signal belongs to a transmission layer, determining a network transmission path to be optimized based on the target observation signal; And carrying out collaborative optimization on the arrangement strategy of the network transmission path to be optimized and the collaborative network transmission path based on the target observation signal to obtain the optimization hierarchy arrangement strategy.
4. The method for organizing data for a job intent of claim 2, wherein said adjusting the hierarchical organization policy based on the target observed signal results in an optimized hierarchical organization policy, further comprising: And under the condition that the target observation signal belongs to a system storage layer, adjusting the transmission layer arrangement strategy based on the target observation signal to obtain the optimized hierarchical arrangement strategy.
5. The data arrangement method for a job intention according to claim 4, wherein the target observation signal includes a storage write throughput of a data receiving end and a transmission rate of a data transmitting end in a case where the target observation signal belongs to a system storage layer; the adjusting the transmission layer arrangement policy based on the target observation signal to obtain the optimized hierarchical arrangement policy includes: And under the condition that the duration time of the transmission rate larger than the storage write throughput meets the preset duration time condition, carrying out hierarchical adjustment on the transmission layer arrangement strategy based on the target observation signal to obtain the optimized hierarchical arrangement strategy.
6. The method for orchestrating data according to claim 5, wherein the step of hierarchically adjusting the transport layer orchestration strategy based on the target observation signal to obtain the optimal hierarchical orchestration strategy comprises: and respectively adjusting the concurrent flow quantity, the priority arrangement strategy and the transmission overhead arrangement strategy in the transmission layer arrangement strategy based on the target observation signal to obtain the optimized hierarchical arrangement strategy.
7. The method of claim 1 to 5, wherein the job intention comprises a data transmission path, a job data item, a job phase priority, a service level objective, and a job global constraint.
8. The job intent-oriented data orchestration method according to claim 7, wherein the service level objectives include at least one of lowest throughput, highest tail latency, and stability indicators; The job global constraints include at least one of a job total cost budget, a latest completion deadline, a job total energy consumption, and a transmission security requirement.
9. A data arrangement device for an operation intention, comprising: An acquisition unit that acquires a job intention of a data transmission job; The compiling unit compiles the operation intention to obtain a multi-level arrangement strategy, wherein the multi-level arrangement strategy comprises at least two of an application layer arrangement strategy, a transmission layer arrangement strategy and a system storage layer arrangement strategy; And the execution unit is used for executing the data transmission job based on the multi-level arrangement strategy.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the data orchestration method according to any one of claims 1 to 8 when the computer program is executed.

Description

Data arrangement method and device for operation intention Technical Field The present invention relates to the field of data transmission technologies, and in particular, to a data arrangement method and apparatus for an operation intention. Background With the explosive growth of global data volume, large-scale data-intensive applications such as cross-regional data entry, artificial intelligence model training data distribution, scientific computing data synchronization, transnational production of media asset content and migration and archiving of mass object storage present unprecedented challenges to the performance of data transmission systems. The bottleneck in end-to-end data transfer is no longer just network bandwidth, but rather involves the full link comprehensive performance of reading from source storage, data preprocessing, network transfer, data verification to destination, decompression, and eventually dropping. The conventional data transmission operation mainly optimizes the data transmission scheme through a single layer, namely, the transmission scheme is optimized for a transmission layer, an application layer or a system storage layer independently. However, the traditional single-layer optimized data transmission scheme can only optimize at a certain specific level of a data path, and cannot effectively solve the end-to-end performance bottlenecks such as downstream data receiving blockage, insufficient CPU computing power, overlarge storage and writing pressure and the like when facing complex data operation. That is, when faced with complex constraints, globally optimal data orchestration cannot be achieved. Disclosure of Invention The invention provides a data arrangement method and device for an operation intention, which are used for solving the defect that global optimization cannot be realized based on single-level optimization when complex data operation is faced in the prior art. The invention provides a data arrangement method facing to operation intention, which comprises the following steps: Acquiring the operation intention of a data transmission operation; Compiling the operation intention to obtain a multi-level arrangement strategy, wherein the multi-level arrangement strategy comprises at least two of an application layer arrangement strategy, a transmission layer arrangement strategy and a system storage layer arrangement strategy; and executing the data transmission job based on the multi-level arrangement strategy. According to the data arrangement method facing to the operation intention, the data transmission operation is executed based on the multi-level arrangement strategy, and the method comprises the following steps: step S1, respectively injecting the multi-level arrangement strategies into levels corresponding to the multi-level arrangement strategies, so that each level performs data transmission according to the level arrangement strategies; Step S2, in the data transmission process, collecting multi-level observation signals; Step S3, evaluating the multi-level observation signals, taking the level observation signals with evaluation results not meeting preset conditions as target observation signals, and adjusting the level arrangement strategy based on the target observation signals to obtain an optimized level arrangement strategy; and S4, taking the optimized hierarchical arrangement strategy as an updated multi-level arrangement strategy, and iteratively executing the steps S1 to S3 until the data transmission operation is completed. According to the data arrangement method facing to the operation intention provided by the invention, the hierarchy arrangement strategy is adjusted based on the target observation signal to obtain an optimized hierarchy arrangement strategy, and the method comprises the following steps: Under the condition that the target observation signal belongs to a transmission layer, determining a network transmission path to be optimized based on the target observation signal; And carrying out collaborative optimization on the arrangement strategy of the network transmission path to be optimized and the collaborative network transmission path based on the target observation signal to obtain the optimization hierarchy arrangement strategy. According to the data arrangement method for the operation intention provided by the invention, the hierarchical arrangement strategy is adjusted based on the target observation signal to obtain an optimized hierarchical arrangement strategy, and the method further comprises the following steps: And under the condition that the target observation signal belongs to a system storage layer, adjusting the transmission layer arrangement strategy based on the target observation signal to obtain the optimized hierarchical arrangement strategy. According to the data arrangement method facing to the operation intention, when the target observation signal belongs to a system storage layer, the target