CN-115640314-B - Data query method and device

Abstract

The embodiment of the application provides a data query method, which comprises the steps of responding to a query request of a user, determining a target queue of the user in a target computing engine, submitting the query request to the target queue for query, and obtaining a query result returned by the target queue, wherein the target computing engine comprises a plurality of queues. The data query method provided by the embodiment of the application can improve the flexibility of the user and the resource management mode when Kyuubi is used.

Inventors

• JIA DONGDONG

Assignees

• 上海哔哩哔哩科技有限公司

Dates

Publication Date

20260512

Application Date

20221019

Claims (11)

1. 1. A method of querying data, comprising: Responding to a query request of a user, determining a target queue of the user in a target computing engine, wherein the target computing engine comprises a plurality of queues; submitting the query request to the target queue for query; acquiring a query result returned by the target queue; Wherein the query request includes the ID of the user, and the submitting the query request to the target queue for querying includes: submitting the query request to the target queue according to the ID of the user so that the target queue queries according to the ID of the user; The step of submitting the query request to the target queue according to the ID of the user so that the target queue queries according to the ID of the user comprises the following steps: And submitting the query request to the target queue, and binding the user identity of a target thread in the target queue by using the ID of the user so that the target thread can process the query request according to the user identity.
2. 2. The data query method of claim 1, wherein submitting the query request to the target queue for a query comprises: Distributing the query request to a target engine instance corresponding to the target queue; and submitting the query request to the target queue for query through the target engine instance.
3. 3. The data query method of claim 1, further comprising: determining a resource demand parameter of the target queue; Requesting resources of the target computing engine according to the resource demand parameters to create the target queue in the target computing engine.
4. 4. The data query method of claim 1, further comprising: monitoring a target event of the query request; determining a state corresponding to the query request according to the target event; and displaying the state on a preset page in real time.
5. 5. The data query method of claim 4, further comprising: Collecting target information of the query request, wherein the target information comprises user, session, job and stage information; determining progress and resource consumption information corresponding to the query request according to the target information; and the progress and the resource consumption information are returned to the front end for display.
6. 6. The method of any one of claims 1-5, applied to Kyuubi, wherein the target computing engine is Spark.
7. 7. The data query method of any one of claims 1-5, further comprising: Monitoring the memory usage of the target computing engine; storing the result pulled from the target computing engine into a local disk under the condition that the memory usage exceeds a preset threshold; And responding to the request for acquiring the result, acquiring the pulled result from the local disk in batches and returning the pulled result.
8. 8. A data query device, comprising: the determining module is used for responding to the query request of the user and determining a target queue where the user is located in a target computing engine, wherein the target computing engine comprises a plurality of queues; the query module is used for submitting the query request to the target queue for query; The acquisition module is used for acquiring a query result returned by the target queue; Wherein the query request includes the ID of the user, and the submitting the query request to the target queue for querying includes: submitting the query request to the target queue according to the ID of the user so that the target queue queries according to the ID of the user; The step of submitting the query request to the target queue according to the ID of the user so that the target queue queries according to the ID of the user comprises the following steps: And submitting the query request to the target queue, and binding the user identity of a target thread in the target queue by using the ID of the user so that the target thread can process the query request according to the user identity.
9. 9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor is adapted to implement the steps of the data query method of any of claims 1 to 7 when the computer program is executed by the processor.
10. 10. A computer-readable storage medium, in which a computer program is stored, the computer program being executable by at least one processor to cause the at least one processor to perform the steps of the data query method of any one of claims 1 to 7.
11. 11. A computer program product comprising computer instructions which, when executed by a processor, implement the steps of the method of any one of claims 1 to 7.

Description

Data query method and device Technical Field The present application relates to the field of big data technologies, and in particular, to a data query method, a data query device, a computer device, and a storage medium. Background Kyuubi is an open source project contributed by the networkteam to the Apache community, is mainly applied to big data field scenes, comprises big data offline computing, adhoc (i.e. check xun), BI (business intelligence) and other directions, and is a big data processing service which is distributed, supports multiple users, is compatible with JDBC (Java Database Connectivity, java database connection) or ODBC (Open Database Connectivity, open database interconnection). At present, before Kyuubi is used, a new user needs to be configured into a group through bottom layer configuration, when the user queries by Kyuubi, kyuubi performs corresponding resource allocation and authority control according to the group where the user is located, and the flexibility of the user and the resource management mode is low. Disclosure of Invention The application aims to provide a data query method, a data query device, computer equipment and a storage medium, which are used for solving the technical problem that the flexibility of a user and a resource management mode is low when Kyuubi is used at present. One aspect of the embodiment of the application provides a data query method, which comprises the steps of responding to a query request of a user, determining a target queue of the user in a target computing engine, submitting the query request to the target queue for query, and obtaining a query result returned by the target queue, wherein the target computing engine comprises a plurality of queues. Optionally, the step of submitting the query request to the target queue for query comprises the steps of distributing the query request to a target engine instance corresponding to the target queue, and submitting the query request to the target queue for query through the target engine instance. Optionally, the query request includes the ID of the user, and the submitting the query request to the target queue for query includes submitting the query request to the target queue according to the ID of the user for query by the target queue according to the ID of the user. Optionally, submitting the query request to the target queue according to the ID of the user so that the target queue can query according to the ID of the user, including submitting the query request to the target queue, and binding the user identity of a target thread in the target queue according to the ID of the user so that the target thread can process the query request according to the user identity. Optionally, the method further comprises determining a resource demand parameter of the target queue, and requesting resources of the target computing engine according to the resource demand parameter to create the target queue in the target computing engine. Optionally, the method further comprises the steps of monitoring a target event of the query request, determining a state corresponding to the query request according to the target event, and displaying the state on a preset page in real time. Optionally, the method further comprises the steps of collecting target information of the query request, wherein the target information comprises user, session, job and stage information, determining progress and resource consumption information corresponding to the query request according to the target information, and transmitting the progress and the resource consumption information back to the front end for display. Optionally, the method is applied to Kyuubi, and the target computing engine is Spark. Optionally, the method further comprises the steps of monitoring the memory usage of the target computing engine, storing the pulled result from the target computing engine to a local disk under the condition that the memory usage exceeds a preset threshold, and responding to a request for obtaining the result, obtaining the pulled result from the local disk in batches and returning the pulled result. The application further provides a data query device, which comprises a determination module, a query module and an acquisition module, wherein the determination module is used for responding to a query request of a user and determining a target queue where the user is located in a target computing engine, the target computing engine comprises a plurality of queues, the query module is used for submitting the query request to the target queue for query, and the acquisition module is used for acquiring a query result returned by the target queue. An aspect of an embodiment of the present application further provides a computer device, where the computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the