CN-116821187-B - Database-based data processing method and device, medium and electronic equipment

CN116821187BCN 116821187 BCN116821187 BCN 116821187BCN-116821187-B

Abstract

The disclosure relates to a data processing method, a device, a medium and electronic equipment based on a database, wherein the method comprises the steps of receiving a query request task to be executed, determining a plurality of cooperative tasks corresponding to the query request task, adding the cooperative tasks into a global task queue shared by a plurality of threads at first, distributing the cooperative tasks into a local task queue corresponding to the threads from the global task queue, determining a target cooperative task to be executed in each thread according to time information of each cooperative task in the local task queue of the thread, interrupting the target cooperative task after the target cooperative task is not executed and the target cooperative task is executed in the current time slice of the thread, adding the target cooperative task into the global task queue, determining a new target cooperative task of the thread according to the global task queue and the local task queue of the thread, and executing the new target cooperative task in the next time slice.

Inventors

LIN YUANJIN
DING WEI

Assignees

抖音视界有限公司
脸萌有限公司

Dates

Publication Date: 20260508
Application Date: 20230704

Claims (10)

1. A database-based data processing method, the method comprising: receiving a query request task to be executed, and determining a plurality of coroutine tasks corresponding to the query request task, wherein the coroutine tasks are firstly added into a global task queue shared by a plurality of threads, then distributed into local task queues of corresponding threads from the global task queue, and the coroutine tasks in the global task queue are ordered according to the execution priority from high to low and distributed according to the execution priority from high to low; In each thread, determining a target cooperative task to be executed according to time information of each cooperative task in a local task queue of the thread; after the target cooperative task is not executed and the target cooperative task is executed in the current time slice of the thread, interrupting the target cooperative task and adding the target cooperative task into a global task queue; And determining a new target coroutine task of the thread according to the global task queue and the local task queue of the thread, and executing the new target coroutine task in the next time slice.
2. The method of claim 1, wherein the target coroutine task is added to a global task queue by: determining the accumulated execution time of the target coroutine task and the quota of the CPU resource corresponding to the target coroutine task; Determining the execution parameters of the target coroutine task according to the accumulated execution time and the quota; And adding the target coroutine task into the global task queue according to the execution parameters.
3. The method of claim 2, wherein determining the execution parameters of the target coroutine task based on the accumulated execution time and the quota comprises: And determining the ratio of the accumulated execution time to the quota as the execution parameter.
4. The method of claim 1, wherein determining a new target coroutine task for the thread based on the global task queue and the local task queue for the thread comprises: If the cooperative task exists in the local task queue of the thread, taking the cooperative task with the earliest distributing time in the local task queue as the new target cooperative task; and if the coroutine task in the local task queue of the thread is empty, applying for the coroutine task from the global task queue to the local task queue, and determining the new target coroutine task from the local task queue after applying for the coroutine task.
5. The method of claim 4, wherein applying for coroutine tasks from the global task queue to the local task queue comprises: Sending a coroutine task request to the global task queue, wherein the coroutine task request comprises the application quantity; and adding the coroutine tasks distributed from the global task queue in response to the coroutine task request to the local task queue, wherein the distributed coroutine tasks are coroutine tasks of the application number sequentially selected based on ordering in the global task queue.
6. The method of claim 1, wherein the determining a plurality of coroutine tasks corresponding to the query request task comprises: Analyzing the query request task and determining a plurality of operators corresponding to the query request task; for each operator, if the input operators of the operators are multiple, dividing the operators and the input operators into different cooperative tasks; if the input operator of the operator is one, dividing the operator and the input operator into the same coroutine task.
7. The method of any of claims 1-6, wherein the global task queue is implemented in a red-black tree structure.
8. A database-based data processing apparatus, the apparatus comprising: The system comprises a receiving module, a processing module and a processing module, wherein the receiving module is used for receiving a query request task to be executed and determining a plurality of coroutine tasks corresponding to the query request task, wherein the coroutine tasks are firstly added into a global task queue shared by a plurality of threads and then distributed into local task queues of corresponding threads from the global task queue, and the coroutine tasks in the global task queue are ordered according to the execution priority from high to low and distributed according to the execution priority from high to low; The first determining module is used for determining target cooperative tasks to be executed in each thread according to the time information of each cooperative task in the local task queue of the thread; the processing module is used for interrupting the target cooperative task and adding the target cooperative task into a global task queue after the target cooperative task is not executed and is executed by the thread in a current time slice; and the second determining module is used for determining a new target cooperative task of the thread according to the global task queue and the local task queue of the thread, and executing the new target cooperative task in the next time slice.
9. A computer readable medium on which a computer program is stored, characterized in that the program, when being executed by a processing device, carries out the steps of the method according to any one of claims 1-7.
10. An electronic device, comprising: A storage device having a computer program stored thereon; processing means for executing said computer program in said storage means to carry out the steps of the method according to any one of claims 1-7.

Description

Database-based data processing method and device, medium and electronic equipment Technical Field The present disclosure relates to the field of computers, and in particular, to a database-based data processing method, apparatus, medium, and electronic device. Background Under the current database architecture, an SQL request is processed by an Optimizer and a planner (Planner) to generate an Execution plan, and finally is passed to an Execution Engine (Execution Engine) to perform specific computation. In the related art, the execution engine mainly comprises a plurality of operators (for example, join operators, shuffle operators) and an executor (Executor) for driving data to flow in the operators, and the execution flow of the operators in the executor has a larger influence on the execution of the SQL statement. Disclosure of Invention This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. In a first aspect, the present disclosure provides a database-based data processing method, the method comprising: Receiving a query request task to be executed, and determining a plurality of coroutine tasks corresponding to the query request task, wherein the coroutine tasks are firstly added into a global task queue shared by a plurality of threads, and then distributed into local task queues of corresponding threads from the global task queue; In each thread, determining a target cooperative task to be executed according to time information of each cooperative task in a local task queue of the thread; after the target cooperative task is not executed and the target cooperative task is executed in the current time slice of the thread, interrupting the target cooperative task and adding the target cooperative task into a global task queue; And determining a new target coroutine task of the thread according to the global task queue and the local task queue of the thread, and executing the new target coroutine task in the next time slice. In a second aspect, the present disclosure provides a database-based data processing apparatus, the apparatus comprising: the receiving module is used for receiving a query request task to be executed and determining a plurality of coroutine tasks corresponding to the query request task, wherein the coroutine tasks are firstly added into a global task queue shared by a plurality of threads and then distributed into local task queues of corresponding threads from the global task queue; The first determining module is used for determining target cooperative tasks to be executed in each thread according to the time information of each cooperative task in the local task queue of the thread; the processing module is used for interrupting the target cooperative task and adding the target cooperative task into a global task queue after the target cooperative task is not executed and is executed by the thread in a current time slice; and the second determining module is used for determining a new target cooperative task of the thread according to the global task queue and the local task queue of the thread, and executing the new target cooperative task in the next time slice. In a third aspect, the present disclosure provides a computer readable medium having stored thereon a computer program which when executed by a processing device performs the steps of the method of the first aspect. In a fourth aspect, the present disclosure provides an electronic device comprising: A storage device having a computer program stored thereon; processing means for executing said computer program in said storage means to carry out the steps of the method of the first aspect. In the technical scheme, the plurality of cooperative tasks corresponding to the query request task to be executed are determined, so that the plurality of cooperative tasks can be scheduled and executed on the existing thread, two-stage scheduling is performed through the global task queue and the local task queue of the thread, and after each cooperative task in the thread is executed for one time slice, interruption is performed to further execute other cooperative tasks, so that parallel execution among the plurality of cooperative tasks can be supported, an accurate and reasonable execution flow is provided for execution of the query request task, and concurrency and efficiency of task execution are improved. Meanwhile, the user mode coroutine simulation thread can be used for realizing the task execution flow, so that the method can be suitable for a scene with fewer threads, realizes resource multiplexing and further widens the application scene of the data processing method. In addition, through the technical scheme, the task execution efficiency is improv