Search

CN-116775598-B - Data table performance detection method, system, computing device and computer readable storage medium

CN116775598BCN 116775598 BCN116775598 BCN 116775598BCN-116775598-B

Abstract

The embodiment of the specification provides a data table performance detection method, a system, a computing device and a computer readable storage medium, wherein the data table performance detection method comprises the steps of obtaining data operation information of a data table to be detected, obtaining time consumption of data reading of the data table to be detected in a data operation process and time consumption of the data operation process by utilizing the data operation information of the data table to be detected, and determining an operation performance index of the data table to be detected based on the time consumption of the data reading and the time consumption of the data operation process.

Inventors

  • LI YONG

Assignees

  • 阿里云计算有限公司

Dates

Publication Date
20260508
Application Date
20230523

Claims (12)

  1. 1. A data table performance detection method, comprising: acquiring data operation information of a data table to be detected; acquiring the time consumption of data reading of the data table to be detected in the data operation process and the time consumption of the data operation process by utilizing the data operation information; And determining an operation performance index of the data table to be detected based on the time consumption of the data reading and the time consumption of the data operation process, and screening the data table to be optimized from a plurality of data tables to be detected according to the operation performance index and an auxiliary diagnosis index, wherein the auxiliary diagnosis index comprises at least one of data inclination, operation times, operation time, a maximum value of data reading quantity of a subtask in the data operation task and a total data reading quantity in operation, and the data table to be optimized represents a data table which has the problem of data inclination and needs to be optimized.
  2. 2. The method of claim 1, further comprising: And screening the data table to be optimized from the plurality of data tables to be detected according to the operation performance index.
  3. 3. The method of claim 2, the method further comprising: acquiring auxiliary diagnosis indexes of the data table to be detected; And selecting the data table to be optimized from a plurality of data tables to be detected according to the operation performance index, including: Screening a data table to be optimized from a plurality of data tables to be detected according to the operation performance index and the auxiliary diagnosis index; The auxiliary diagnosis index comprises at least one of data gradient, operation times, operation time, maximum data reading quantity of sub-tasks in the data operation task and total data reading quantity in operation.
  4. 4. The method according to claim 3, wherein the auxiliary diagnostic index includes a number of operations and a data inclination, and the selecting a data table to be optimized from a plurality of data tables to be detected according to the operation performance index and the auxiliary diagnostic index includes: Screening out data tables with operation times meeting the preset operation times from a plurality of data tables to be detected, and obtaining a candidate set of the data tables to be optimized; Screening out a data table with data gradient meeting a preset gradient condition from the candidate set to obtain an updated candidate set of the data table to be optimized; and determining a data table with the operation performance index meeting the preset operation performance index condition from the updated candidate set, and taking the data table as the data table to be optimized.
  5. 5. A method according to claim 3, the auxiliary diagnostic index comprising a data inclination, the obtaining the auxiliary diagnostic index of the data table to be detected comprising: determining a data operation task of the data table to be detected by utilizing the data operation information of the data table to be detected; Acquiring data reading amounts corresponding to a plurality of subtasks in the data operation task, wherein the data table to be detected is divided into a plurality of data fragments, the plurality of data fragments are stored in a plurality of storage nodes in a scattered manner, and one subtask is used for reading out data of one data fragment from one storage node and performing data operation; and calculating the data gradient of the data table to be detected based on the data reading amounts corresponding to the plurality of subtasks.
  6. 6. The method according to claim 5, wherein calculating the data inclination of the data table to be detected based on the data read amounts corresponding to the plurality of subtasks comprises: Determining a first target subtask with the maximum corresponding data reading amount from the plurality of subtasks; And calculating the ratio of the data reading quantity corresponding to the first target subtask to the average value of the data reading quantities corresponding to the plurality of subtasks to obtain the data gradient of the data table to be detected.
  7. 7. The method of claim 6, the obtaining the time-consuming data reading of the data table to be detected during the data operation and the time-consuming data operation, comprising: determining a second target subtask with the longest time consumption for corresponding data reading from the plurality of subtasks; Acquiring the time consumption of data reading of the second target subtask and the time consumption of the data table to be detected in the data operation process; The determining the operation performance index of the data table to be detected based on the time consumption of the data reading and the time consumption of the data operation process comprises the following steps: And calculating the ratio of the time consumption of the data reading of the second target subtask to the time consumption of the data table to be detected in the data operation process to obtain the operation performance index of the data table to be detected.
  8. 8. A method according to claim 3, wherein the obtaining the data operation information of the data table to be detected comprises: analyzing the database operation log to obtain data operation information of a plurality of data tables to be detected in the database; the method further comprises the steps of: Respectively storing the table identification, the data operation task identification, the operation performance index and the auxiliary diagnosis index of the data table to be detected in each field of a diagnosis data table; and selecting a data table to be optimized from a plurality of data tables to be detected according to the operation performance index and the auxiliary diagnosis index, wherein the data table to be optimized comprises: And screening the data table to be optimized from the plurality of data tables to be detected by utilizing the table identification, the data operation task identification, the operation performance index and the auxiliary diagnosis index of the data table to be detected stored in each field of the diagnosis data table and the preset screening conditions corresponding to the operation performance index and the auxiliary diagnosis index.
  9. 9. A method according to claim 3, further comprising: And reselecting a designated field aiming at a data table to be optimized, wherein the designated field is used for segmenting the data of the data table to obtain a plurality of data fragments, and the plurality of data fragments are stored in a plurality of storage nodes in a scattered manner.
  10. 10. A data table performance detection system comprises a detection node, a database service node, a calculation node and a plurality of storage nodes; The detection node configured to detect an operation performance index of a data table in a database by applying the data table performance detection method according to any one of claims 1 to 9; the database service node is configured to run the database and issue a data operation task of the database to the computing node; the computing node is configured to perform data operation tasks on a data table of the database; The storage node is configured to store data fragments of the data table, the data fragments are obtained by dividing data of the data table by using specified fields of the data table, and a plurality of data fragments of the data table are stored in a plurality of storage nodes in a scattered manner.
  11. 11. A computing device, comprising: A memory and a processor; The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, which when executed by the processor, implement the steps of the data table performance detection method of any one of claims 1-9.
  12. 12. A computer readable storage medium storing computer executable instructions which when executed by a processor perform the steps of the data table performance detection method of any one of claims 1 to 9.

Description

Data table performance detection method, system, computing device and computer readable storage medium Technical Field The embodiment of the specification relates to the technical field of computers, in particular to a data table performance detection method. Background With the development of informatization, the data of the database is continuously increased, and the database is required to have higher performance. However, since the implementation of functions of a database is generally a focus of attention in the development stage of the database, less attention is paid to the operation performance, and as the database is used, it becomes more and more difficult to rapidly operate related data from a database with a large data volume. Based on this, the data table is generally optimized after the database is put into use, so as to improve the operation performance of the database. But optimizing which data tables in the database to improve the operational performance of the database is a difficult problem. Therefore, there is a need for a data table performance detection method to help people accurately detect the data table affecting the database operation performance. Disclosure of Invention In view of this, the embodiments of the present disclosure provide a data table performance detection method. One or more embodiments of the present specification are also directed to a data table performance detection system, a computing device, a computer-readable storage medium, and a computer program that address the technical deficiencies of the prior art. According to a first aspect of embodiments of the present disclosure, a method for detecting performance of a data table is provided, including obtaining data operation information of the data table to be detected, obtaining time consumption of data reading of the data table to be detected in a data operation process and time consumption of the data operation process by using the data operation information of the data table to be detected, and determining an operation performance index of the data table to be detected based on the time consumption of data reading and the time consumption of the data operation process. According to a second aspect of embodiments of the present specification, there is provided a data table performance detection system comprising a detection node, a database service node, a calculation node and a plurality of storage nodes. The detection node is configured to detect operation performance indexes of data tables in a database by applying the data table performance detection method according to any embodiment of the specification, the database service node is configured to run the database and send data operation tasks of the database to the calculation node, the calculation node is configured to execute data operation tasks on the data tables of the database, the storage node is configured to store data fragments of the data tables, the data fragments are obtained by dividing data of the data tables by using specified fields of the data tables, and a plurality of data fragments of the data tables are stored in a plurality of storage nodes in a scattered mode. According to a third aspect of embodiments of the present specification, there is provided a computing device comprising a memory for storing computer executable instructions and a processor for executing the computer executable instructions which when executed by the processor implement the steps of the data table performance detection method described above. According to a fourth aspect of embodiments of the present specification, there is provided a computer-readable storage medium storing computer-executable instructions which, when executed by a processor, implement the steps of the data table performance detection method described above. According to a fifth aspect of embodiments of the present specification, there is provided a computer program, wherein the computer program, when executed in a computer, causes the computer to perform the steps of the above-described data table performance detection method. According to the method, data operation information of the data table to be detected is obtained, and time consumption of data reading of the data table to be detected in a data operation process and time consumption of the data operation process are obtained by utilizing the data operation information of the data table to be detected, so that operation performance indexes of the data table to be detected are determined based on the time consumption of data reading and the time consumption of the data operation process. Therefore, the method determines the influence of data reading on data operation based on the time consumption of data reading and the whole time consumption of the data operation process, so that the operation performance index of the data table to be detected is obtained, people can be helped to accurately find out the data table affecting the ope