Search

CN-122019127-A - Intelligent body infinite context processing method and system based on distributed computing

CN122019127ACN 122019127 ACN122019127 ACN 122019127ACN-122019127-A

Abstract

The invention relates to the technical field of intelligent processing of long texts and discloses an intelligent body infinite context processing method and system based on distributed computation, wherein the method comprises the steps of receiving a long text of a user and processing requirements, analyzing and generating a MapReduce execution plan containing a Map stage, a Reduce stage and parameter configuration; the method comprises the steps of intelligently segmenting a long text into independent data fragments with overlapping areas according to a fragmentation strategy, guaranteeing context continuity, scheduling multi-agent parallel processing fragments to generate intermediate results, enabling the intermediate results to be batched according to batch processing parameters in a Reduce stage, sequentially calling agent iterative aggregation, generating a global result through accumulating the contexts, and storing execution process data to support breakpoint continuous transmission. The invention breaks through the context bottleneck of the intelligent agent, realizes the high-efficiency processing of the long text, has processing integrity and reliability, and meets the analysis requirement of the long text in multiple scenes.

Inventors

  • WANG YONG

Assignees

  • 广东华之源信息工程有限公司
  • 佳都科技集团股份有限公司
  • 广州佳都智通科技有限公司
  • 广州华佳软件有限公司

Dates

Publication Date
20260512
Application Date
20251219

Claims (10)

  1. 1. An agent infinite context processing method based on distributed computing, comprising: Receiving a hyper-long text input by a user and processing requirements; Analyzing the processing requirement to generate a MapReduce execution plan, wherein the execution plan comprises a Map stage, a Reduce stage and parameter configuration, and the parameter configuration comprises a slicing strategy, concurrency and batch processing parameters; Intelligently segmenting the long text into a plurality of independent data segmentation tasks according to the segmentation strategy, wherein each segmentation task comprises segmentation input content and a configurable overlapping area for maintaining context continuity; Scheduling a plurality of agents to process all data slicing tasks in parallel based on the concurrency through a Map stage executor to generate a plurality of intermediate results; After all the data slicing tasks are processed, calling the intelligent agents to carry out iterative aggregation according to the batch sequence based on the batch processing parameters by a Reduce stage aggregator so as to generate a final processing result of global aggregation and feeding the final processing result back to a user; and storing the task state, the execution record and the final processing result in the execution process.
  2. 2. The method of claim 1, wherein the slicing strategy comprises an auto-detect mode, a fixed size mode, a chapter-based mode, a semantic boundary-based detect mode, and a slicing parameter comprising a single maximum number of characters, a number of overlapping region characters, and wherein the task directory list is generated after slicing is completed.
  3. 3. The method according to claim 2, wherein the automatic detection mode is divided into sections preferentially, then divided into sections, and finally divided into fixed sizes; the process for slicing based on the semantic boundary detection mode comprises the steps of carrying out semantic boundary detection near a target slicing point, searching according to the priority sequence of a paragraph boundary, a sentence boundary, a comma boundary and a space boundary, carrying out slicing at the boundary if a conforming boundary is found, and carrying out forced slicing at the target slicing point if the conforming boundary is not found.
  4. 4. The method of claim 2, wherein the scheduling, by the Map phase executor, of the plurality of agents to process all data slicing tasks in parallel based on the concurrency, generating a plurality of intermediate results comprises: The Map stage executor receives the task directory list, and triggers a thread pool dynamically adjusted according to a preset concurrency degree to call a plurality of agents for processing in concurrency, wherein the size of the thread pool is adaptively updated according to system resources and an API flow limiting strategy; Each agent processes an independent slicing task with a unique task ID, and the tasks are not mutually influenced; and executing automatic retry on the task with failed processing, wherein the retry times and delay time are configurable, and the completion of the task is judged by checking that the output file exists and is not empty, so that each intermediate result is ensured to be effective.
  5. 5. The method of claim 1 or 4, wherein iteratively aggregating the plurality of intermediate results by a Reduce stage aggregator invoking agents in batch order based on the batch parameters to generate a final aggregated process result comprises: after detecting that all the slicing tasks in the Map stage are successfully executed, collecting all intermediate results generated in the Map stage; dividing the all intermediate results into a plurality of ordered batches according to the batch parameters, wherein the batch parameters at least comprise a maximum character number threshold of batch processing so as to ensure that the total number of characters of each batch does not exceed the threshold; And executing aggregation operation on each batch in turn according to the sequence of the ordered batches, wherein for the first batch, an agent is called to process the content of the batch to generate a first round of aggregation result, for each subsequent batch, the agent is called to process the aggregation result of the previous round and the content of the current batch simultaneously to generate a new round of aggregation result, wherein the aggregation result of the previous round is transmitted to the agent as an accumulated context, and the aggregation result obtained after the last batch is processed is output as the global aggregation result.
  6. 6. The method of claim 1, wherein storing the task state, execution record, and final processing result during execution comprises: If the identification of the execution plan is not found, a new plan state object is created and stored in the shared state container, wherein the plan state object comprises a task state table for recording the states of all the slicing tasks in the execution plan, an atomic counter for generating a unique task identification for the execution plan and a creation time stamp of the execution plan; Responding to the change of the state of the slicing task in the execution plan, acquiring a corresponding plan state object from the shared state container according to the identification of the execution plan, and atomically updating the state of the corresponding slicing task in the task state table in the plan state object; Responding to a state query request, acquiring a corresponding plan state object from the shared state container according to the identification of the execution plan, and querying a completed slicing task list of a specific state based on the task state table; Periodically scanning the shared state container, removing from the shared state container a planned state object having a creation time earlier than a preset expiration threshold.
  7. 7. An agent infinite context processing system based on distributed computing, comprising: the user request layer is used for receiving the ultra-long text input by the user and processing requirements; The plan generation layer is used for analyzing the processing requirements to generate a MapReduce execution plan, wherein the execution plan comprises a Map stage, a Reduce stage and parameter configuration, and the parameter configuration comprises a slicing strategy, concurrency and batch processing parameters; a data slicing layer, configured to intelligently slice the long text into a plurality of independent data slicing tasks according to the slicing policy, where each slicing task includes slicing input content and a configurable overlapping area for maintaining context continuity; The Map stage execution layer is used for scheduling a plurality of agents to process all data slicing tasks in parallel and generating a plurality of intermediate results; the Reduce stage execution layer is used for calling the intelligent agents for iterative aggregation according to the batch sequence after all the data slicing tasks are processed, so as to generate final processing results of global aggregation and feeding the final processing results back to a user; and the result persistence layer is used for storing the task state, the execution record and the final processing result in the execution process.
  8. 8. An electronic device, comprising: a memory and a processor in communication with each other, the memory having stored therein computer instructions that, upon execution, perform the distributed computing-based agent infinite context processing method of any of claims 1-6.
  9. 9. A computer-readable storage medium having stored thereon computer instructions for causing a computer to perform the distributed computing-based agent infinite context processing method of any one of claims 1 to 6.
  10. 10. A computer program product comprising computer instructions for causing a computer to perform the distributed computing-based agent infinite context processing method of any one of claims 1 to 6.

Description

Intelligent body infinite context processing method and system based on distributed computing Technical Field The invention relates to the technical field of intelligent processing of long texts, in particular to an intelligent body infinite context processing method and system based on distributed computing. Background With the deep penetration of artificial intelligence technology in enterprise-level applications, long text processing demands are becoming urgent, covering a plurality of core scenarios such as content analysis, log analysis, knowledge base construction and the like. In the field of content analysis, the text of network novels, academic review papers, legal contracts, judgment books and the like is very few hundred thousand words or even millions of words, chapter abstracts, full text overviews and key information extraction are required to be efficiently generated, in a log analysis scene, a system log file generated by a production environment can reach a scale of several GB, intelligent recognition of abnormal and alarm information is required to be urgently needed, and in the aspect of knowledge base construction, a code base of a large open-source project and massive technical documents accumulated in enterprises are required to be generated by analysis and arrangement. The current mainstream large language models (such as Claude-4.5, GPT-5, deepSeek) have strong natural language processing capability, but have explicit context length limitations (128K-272 Ktokens) and cannot directly process the very long text. The existing solutions mainly comprise a sliding window method, a retrieval enhancement generation (RAG), simple slicing parallel processing, long context model direct processing and a manual summary method, however, the solutions have obvious defects. The long context model is high in use cost, 10-20 times that of the short context model, the API call cost for processing the text with millions of words can reach hundreds of dollars, and the manual processing is more difficult to scale. In terms of performance and quality, the longer the context is, the slower the model reasoning speed is, and the problem of 'middle forgetting' exists, the sliding window method and the simple segmentation can crack semantic integrity, and the RAG method depends on retrieval quality and is easy to miss key information. In efficiency, both the sliding window method and the manual processing are sequentially executed, and the time is long and the resource utilization rate is low. In addition, the prior art lacks a systematic framework, and lacks self-adaptive slicing, distributed task arrangement, intelligent aggregation, fault tolerance guarantee and extensible design, so that the requirements of high efficiency, high quality and low cost of enterprise-level ultra-long text processing are difficult to meet. Disclosure of Invention In order to solve the technical problems of limited context length, high processing cost, easy cracking of semantic integrity, low efficiency and lack of systematic fault tolerance and expandable framework existing in the process of processing the ultra-long text in the prior art, the invention provides an intelligent infinite context processing method and system based on distributed computing, which realize the efficient, low-cost and high-quality processing of the ultra-long text. In a first aspect, the present invention provides an agent infinite context processing method based on distributed computing, including: Receiving a hyper-long text input by a user and processing requirements; Analyzing the processing requirement to generate a MapReduce execution plan, wherein the execution plan comprises a Map stage, a Reduce stage and parameter configuration, and the parameter configuration comprises a slicing strategy, concurrency and batch processing parameters; Intelligently segmenting the long text into a plurality of independent data segmentation tasks according to the segmentation strategy, wherein each segmentation task comprises segmentation input content and a configurable overlapping area for maintaining context continuity; Scheduling a plurality of agents to process all data slicing tasks in parallel based on the concurrency through a Map stage executor to generate a plurality of intermediate results; After all the data slicing tasks are processed, calling the intelligent agents to carry out iterative aggregation according to the batch sequence based on the batch processing parameters by a Reduce stage aggregator so as to generate a final processing result of global aggregation and feeding the final processing result back to a user; and storing the task state, the execution record and the final processing result in the execution process. According to the intelligent agent infinite context processing method based on distributed computing, which is provided by the embodiment of the invention, the limitation of the intelligent agent context length is broken through