CN-121996321-A - Distributed data preloading and query method, device and storage medium

CN121996321ACN 121996321 ACN121996321 ACN 121996321ACN-121996321-A

Abstract

The application discloses a distributed data preloading and inquiring method, equipment and a storage medium, which comprise the steps of searching data corresponding to an inquiring key in a cache area integrated by an application server, returning an inquiring result if the inquiring result of the inquiring key is hit in the cache area, determining whether a target inquiring key identical to the inquiring key exists in an executing database inquiring if the inquiring result of the inquiring key is not hit in the cache area, initiating an inquiring request based on the inquiring key to a database if the target inquiring key does not exist, and storing the inquiring result returned by the database into the cache area integrated by the application server for responding to inquiring of the same inquiring key. According to the application, the cache is integrated in the application server, so that the resource expense of an independent cache server is eliminated, and the concurrent query in the case of cache miss is merged into a single query, so that the repeated request pressure of the database is reduced fundamentally, and the cache breakdown risk is effectively prevented.

Inventors

WEI XINYU
CHEN QIHAN
Yi Die
DENG DONGSHENG
QIU DENGHUI

Assignees

尚娱软件（深圳）有限公司

Dates

Publication Date: 20260508
Application Date: 20260106

Claims (10)

1. A distributed data preloading and querying method, characterized in that the distributed data preloading and querying method comprises the steps of: Acquiring a historical query record, and analyzing the occurrence frequency and time distribution mode of each query key based on the historical query record; Identifying one or more high-frequency query keys from the historical query record according to the occurrence frequency and the time distribution mode; After a preset low-load period or the application server is started, actively initiating a preloading query request based on the high-frequency query key to a database; receiving a preloading query result returned by the database, and storing the preloading query result into a cache area integrated by the application server in advance; Responding to a subsequent data query request, wherein the data query request comprises a query key; Searching whether a query result matched with the query key exists in the cache area based on the data query request; And if the query result corresponding to the query key is hit in the cache area, directly returning the query result.
2. The distributed data preloading and querying method of claim 1, further comprising, after the step of searching in the cache area based on the data query request for whether there is a query result matching the query key: If the query result corresponding to the query key is not hit in the cache area, determining whether a target query key identical to the query key exists in the database query in execution; And if the target query key does not exist, initiating a real-time query request based on the query key to the database, and storing a real-time query result returned by the database into a cache area integrated by the application server.
3. The distributed data preloading and querying method of claim 2, wherein after the step of determining whether there is a target query key identical to the query key in the executing database query, further comprising: if the target query key exists, performing association registration on the current data query request and the database query which is being executed and corresponds to the target query key; And after the target query key is used for completing the database query execution, distributing the unified query result returned by the database to all the associated registered data query requests.
4. The distributed data preloading and querying method of claim 2, wherein after the step of determining whether there is a target query key identical to the query key in the executing database query, further comprising: If the target query key does not exist, a new entry which is based on the current query key as an identification is newly built in a pre-maintained unfinished query registry; and creating a corresponding database query in execution based on the new entry marker.
5. The distributed data preloading and querying method of claim 1, further comprising, prior to the step of responding to a subsequent data querying request: analyzing target service data to be queried, and extracting a server identifier, a service logic type identifier and a data entity identifier in a server cluster; And combining the server identifier, the service type identifier and the entity identifier into a unique three-dimensional composite main key by adopting a preset coding rule, and taking the unique three-dimensional composite main key as the query key.
6. The method for preloading and querying distributed data as claimed in claim 1 or 2, wherein the step of storing the query results in the cache region comprises: Serializing the query result and converting the query result into a compact structured data block; and adopting a binary large object format, storing the structured data block in association with a corresponding query key, and recording the storage time stamp of the structured data block.
7. The method for preloading and querying distributed data as claimed in claim 6, wherein after the step of storing the structured data block in association with a corresponding query key and recording a storage time stamp thereof, further comprising: Triggering a cleaning process periodically or when the buffer space reaches a threshold value; traversing the cache area, and judging and clearing out an expired query result according to the storage time stamp and a predefined expiration strategy; wherein the expiration policy configures different survival times for preloaded query results and real-time query results, respectively.
8. The distributed data preloading and querying method as claimed in any of claims 1 to 7, the method is characterized in that the distributed data preloading and inquiring method further comprises the following steps: Detecting real-time load indexes of a plurality of candidate server nodes, wherein the load indexes comprise at least one of CPU utilization rate, memory utilization rate and network I/O; Selecting a node with the lowest load from the candidate server nodes as an application server responding to the preloading inquiry request and the inquiry request according to a load balancing algorithm; and directing the preloaded query request and the route of the query request to the application server.
9. A distributed data preloading and querying device, characterized in that it stores a computer program which, when executed by a processor, implements the distributed data preloading and querying method according to any of claims 1-8.
10. A storage medium storing a computer program which when executed by a processor implements the distributed data preloading and querying method of any of claims 1-8.

Description

Distributed data preloading and query method, device and storage medium Technical Field The present application relates to the field of distributed system data management technologies, and in particular, to a distributed data preloading and querying method, device, and storage medium. Background In a distributed system such as a large-scale multiplayer online game and a social platform which needs to process massive real-time interaction data, a business logic server frequently queries a central database to acquire key data such as user state and configuration information, and the like, so that the business logic server is one of the most core and load-intensive operations. In order to ensure the response speed and stability of the system, coping with high concurrent access pressure, the design of a data query architecture is important. Currently, a common solution is to have the business server directly access a relational database (such as MySQL), which is simple to implement, but in actual operation, especially in the face of transient peak requests, a large number of queries of the same or similar content directly impact the database. Because each query needs to independently establish a connection, parse SQL, execute, and return results, the CPU, memory, and I/O resources of the database quickly become bottlenecks, resulting in increased response delays and even possibly causing service unavailability. The fundamental disadvantage is that the lack of merging and buffering mechanisms for concurrent requests does not translate repeated database accesses into a single operation, resulting in unnecessarily large pressure on the database. To solve the database pressure problem, introducing a separate distributed cache middleware (such as Redis) has become a standard architecture. In this mode, the traffic server first queries the independent cache cluster and only accesses the database if there is a cache miss. This approach, while alleviating the direct load of the database, itself introduces significant new challenges. First, deploying and maintaining a set of highly available independent cache clusters requires dedicated server hardware, network configuration, and continuous operational investment. In addition, in many business scenarios, the access load has obvious peaks and valleys, and in the low-load period, the resource utilization rate of the dedicated cache server is often very low, so that the hardware investment and the energy waste are caused. Second, the complexity and maintenance costs of the system architecture increase significantly. The business logic is changed from directly accessing the database to sequentially interacting with the cache layer and the database layer, so that the links are increased, and the fault points are increased. This not only increases network latency and overall system complexity, but also makes deployment, monitoring, and fault diagnosis more difficult. More critical, the architecture is hidden from cache miss risk. For example, when certain hot spot data is out of date or cleared in the cache, a large number of concurrent requests which are instantaneously flushed will simultaneously "penetrate" the cache layer to directly access the database, and this "cache breakdown" phenomenon is very easy to cause instantaneous overload of the back-end database, and may cause a chain reaction to cause system level avalanche. It can be seen that in the conventional mainstream technology path, the solution of directly accessing the database cannot cope with high concurrency, while the solution of introducing independent cache has high cost in terms of resource efficiency, architecture complexity and system risk when relieving the pressure of the database. The foregoing is provided merely for the purpose of facilitating understanding of the technical solutions of the present application and is not intended to represent an admission that the foregoing is prior art. Disclosure of Invention The application mainly aims to provide a distributed data preloading and inquiring method, equipment and a storage medium, and aims to solve the technical problems of resource waste, complex architecture and cache breakdown risk of the existing cache scheme. In order to achieve the above objective, the present application provides a distributed data preloading and querying method, which includes: Acquiring a historical query record, and analyzing the occurrence frequency and time distribution mode of each query key based on the historical query record; Identifying one or more high-frequency query keys from the historical query record according to the occurrence frequency and the time distribution mode; After a preset low-load period or the application server is started, actively initiating a preloading query request based on the high-frequency query key to a database; receiving a preloading query result returned by the database, and storing the preloading query result into a cache area integr