CN-122019412-A - Data multi-level cache collaborative acceleration method and system

CN122019412ACN 122019412 ACN122019412 ACN 122019412ACN-122019412-A

Abstract

The invention discloses a data multi-level cache collaborative acceleration method and system, and mainly relates to the technical field of data cache acceleration. The method comprises the steps of obtaining data access characteristics, inputting a model integrating sliding window statistics and exponential decay to calculate a dynamic heat value, dynamically scheduling data to a high-speed memory, distributed sharing or local file cache level according to the heat value, optimizing a cross-level query path based on an intelligent routing model, maintaining multi-level cache consistency by adopting an asynchronous version control mechanism, performing layered collaborative cleaning and elastic expansion according to real-time heat and capacity utilization rate, and preloading hot spot data by utilizing a prediction model. The invention has the beneficial effects that the cache hit rate and the system throughput are obviously improved through intelligent scheduling and collaborative optimization, the access delay and the storage cost are reduced, and the stability and the data consistency under high concurrency are enhanced.

Inventors

WANG CHENGXUAN
ZHANG NA
CHI YUNQIANG
YAO WEIYAN

Assignees

山东数字人科技股份有限公司

Dates

Publication Date: 20260512
Application Date: 20260130

Claims (10)

1. The data multi-level cache collaborative acceleration method is characterized by comprising the following steps of: acquiring access characteristic information of data to be accessed; Calculating a heat value of data to be accessed based on the access characteristic information; Determining a cache level to be accessed by the data to be accessed in priority according to the heat value and a preset heat threshold interval, wherein the cache level comprises a high-speed memory cache, a distributed shared cache and a local file cache; if the data to be accessed hit in the determined cache hierarchy, returning the data to be accessed and updating the access record of the data to be accessed; if the access path is not hit, selecting an optimal query path according to a pre-established access path decision model, and initiating query to the next cache level until the access path is hit or the lowest cache is reached; if the lowest layer cache is not hit, loading the data from the persistent storage and writing the data into all cache levels; In the data access process, dynamically triggering the transfer of cache data across the levels based on the real-time change of the data heat, and executing collaborative cleaning operation on each cache level; When the data is updated, an asynchronous synchronization mechanism based on version control is adopted to synchronize the updating content to each cache level step by step.
2. The method for collaborative acceleration of data multi-level caching according to claim 1, wherein the method is characterized in that the method is based on the access characteristic information and calculates a heat value of data to be accessed, and specifically comprises the following steps: Constructing and maintaining a sliding time window with a fixed time length, and counting the real-time access times of the data in the sliding time window; Acquiring historical cumulative access times of data to be accessed, and carrying out weighted calculation on the historical cumulative access times by using an exponential decay function, wherein the decay coefficient of the exponential decay function is dynamically set according to the access mode of the data type; Converting the byte size of the data to be accessed into a standardized storage cost factor, the standardized storage cost factor being inversely proportional to the data size; Extracting the proportion of the remaining effective time of the data to be accessed to the initial effective time as a timeliness factor; Normalizing the real-time access times, the historical access times weighted by exponential decay, the normalized storage cost factors and the timeliness factors to obtain corresponding characteristic values respectively; Dynamically adjusting the weight of each characteristic value based on a preset initial weight coefficient and according to the real-time analysis result of the current system load and the data access mode, wherein the adjustment process adopts an online learning algorithm based on gradient descent; According to the dynamically adjusted weight, carrying out linear weighted summation on each normalized characteristic value, and calculating the comprehensive heat value of the data; and inputting the comprehensive heat value into a calibration model based on a neural network for post-processing so as to eliminate heat deviation under different data access modes and output a final dynamic heat value.
3. The method of claim 2, wherein determining the cache hierarchy based on the hotness value comprises: If the heat value reaches a first threshold value, judging that the high-heat data is high-heat data, and preferentially accessing the cache memory; if the heat value reaches the second threshold value but is lower than the first threshold value, judging that the heat data is medium heat data, and preferentially accessing the distributed shared cache; If the heat value is lower than the second threshold value, judging that the low heat data is low heat data, and preferentially accessing the local file cache; Wherein the heat threshold is dynamically adjusted according to the historical access pattern by a machine learning model.
4. A data multi-level cache collaborative acceleration method according to claim 3, wherein the machine learning model is an LSTM prediction model trained based on historical access sequences for dynamically modifying the first and second thresholds according to access trends.
5. The method of claim 1, wherein selecting the optimal query path according to the pre-established access path decision model comprises: Constructing a routing table taking access delay, data heat and hierarchical load as decision factors; predicting a better cache access path by using a decision model based on a predefined rule and real-time weight; And dynamically selecting whether to skip the intermediate cache hierarchy for inquiring according to the prediction result.
6. The method for collaborative acceleration of a data multi-level cache of claim 1, wherein the collaborative cleaning operation comprises: Calculating a cleaning priority score according to the real-time heat value and the storage cost of the data; the data to be cleaned are arranged in ascending order according to the priority score; And performing hierarchical cleaning according to the current utilization rate of each cache level and a preset cleaning proportion, wherein the cache cleaning supercapacity part of the high-speed memory is 15% -25%, the distributed shared cache cleaning is 25% -35%, and the local file cache cleaning is 35% -45%.
7. The method for collaborative acceleration of data multi-level caching according to claim 1, further comprising the step of cache preloading: predicting hot spot data within a future set time period based on a time series analysis model; And when the system load is lower than a set threshold value, asynchronously preloading the predicted hot spot data into a high-speed memory cache or a distributed shared cache.
8. A data multi-level cache collaborative acceleration system for implementing a data multi-level cache collaborative acceleration method according to any one of claims 1-7, comprising: The configuration management module is used for dynamically enabling or disabling each level of cache, and setting the capacity upper limit, expiration time and connection parameters of each cache level; The intelligent scheduling module is used for acquiring data access characteristics, calculating a heat value and selecting a cache access path according to the heat value; the route optimization module is used for constructing and executing an access path decision model and optimizing the cross-level query sequence; The heterogeneous storage module comprises a high-speed memory cache unit, a distributed shared cache unit and a local file cache unit, and is used for storing data with different heat degrees respectively; the consistency guarantee module is used for realizing asynchronous data synchronization among the multi-level caches based on a version control protocol; The elastic management and control module is used for monitoring the utilization rate of each level of buffer, dynamically adjusting the cleaning threshold value, distributing capacity and executing cooperative cleaning; and the prediction preloading module is used for predicting hot spot data based on the machine learning model and triggering cache preloading operation.
9. The data multi-level cache collaborative acceleration system according to claim 8, wherein the cache unit adopts a dual-mode elimination strategy supporting absolute expiration and relative expiration, the distributed shared cache unit adopts an asynchronous notification mechanism based on version control to synchronously update to a local cache unit, and the local file cache unit organizes cache files by adopting a two-level directory structure based on hash mapping.
10. The data multi-level cache collaborative acceleration system of claim 8, further comprising a high concurrency protection module configured to: enabling the mutual exclusion lock when hot spot data access is detected, and setting the upper limit of lock waiting time; adding a random offset to the expiration time of the cached data; An automatic failover is performed when the cache node fails.

Description

Data multi-level cache collaborative acceleration method and system Technical Field The invention relates to the technical field of data cache acceleration, in particular to a multi-level cache collaborative acceleration method and system based on heterogeneous storage hierarchy and scheduling. Background With the rapid development of fields such as accurate medical treatment and digital pathology, the data volume generated by a single biomedical specimen grows exponentially, and the three access challenges of high concurrence, low delay and large capacity are faced in the scenes of scientific research cooperation, AI training, remote diagnosis and the like. Traditional single-cache architecture, such as pure memory cache, is limited by physical capacity and cost, distributed cache has network delay and single-point fault risk, and local file cache has low retrieval efficiency, so that the performance requirements of the complex scene are difficult to meet. In the prior art, although a multi-level cache scheme is presented, a fixed hierarchy access sequence and a static scheduling strategy are generally adopted, and the difference of data access heat, size and timeliness cannot be fully considered, so that cache resources are unreasonably contended for high-frequency small data and low-frequency large data. In addition, the lack of an efficient consistency synchronization mechanism between the multi-level caches often relies on full-scale updates, resulting in a large amount of redundant I/O overhead. Meanwhile, the existing scheme lacks elastic expansion and intelligent scheduling capability when facing access mode dynamic change and burst flow, and is easy to cause problems of cache breakdown, avalanche and the like, and the overall cache efficiency and the resource utilization rate are not optimal. Therefore, there is a need for a multi-level cache acceleration method and system capable of deeply fusing heterogeneous storage media and having intelligent scheduling and dynamic coordination capabilities, so as to achieve an optimal balance between storage cost and access efficiency while ensuring high performance and availability. Disclosure of Invention The invention aims to provide a multi-level cache collaborative acceleration method and system based on heterogeneous storage hierarchy and scheduling, which remarkably improve cache hit rate and system throughput, reduce access delay and storage cost and enhance stability and data consistency under high concurrency through intelligent scheduling and collaborative optimization. The invention aims to achieve the aim, and the aim is achieved by the following technical scheme: In one aspect, the invention provides a data multi-level cache collaborative acceleration method, which comprises the following steps: acquiring access characteristic information of data to be accessed; Calculating a heat value of data to be accessed based on the access characteristic information; Determining a cache level to be accessed by the data to be accessed in priority according to the heat value and a preset heat threshold interval, wherein the cache level comprises a high-speed memory cache, a distributed shared cache and a local file cache; if the data to be accessed hit in the determined cache hierarchy, returning the data to be accessed and updating the access record of the data to be accessed; if the access path is not hit, selecting an optimal query path according to a pre-established access path decision model, and initiating query to the next cache level until the access path is hit or the lowest cache is reached; if the lowest layer cache is not hit, loading the data from the persistent storage and writing the data into all cache levels; In the data access process, dynamically triggering the transfer of cache data across the levels based on the real-time change of the data heat, and executing collaborative cleaning operation on each cache level; When the data is updated, an asynchronous synchronization mechanism based on version control is adopted to synchronize the updating content to each cache level step by step. Preferably, the heat value of the data to be accessed is calculated based on the access characteristic information, specifically: Constructing and maintaining a sliding time window with a fixed time length, and counting the real-time access times of the data in the sliding time window; Acquiring historical cumulative access times of data to be accessed, and carrying out weighted calculation on the historical cumulative access times by using an exponential decay function, wherein the decay coefficient of the exponential decay function is dynamically set according to the access mode of the data type; Converting the byte size of the data to be accessed into a standardized storage cost factor, the standardized storage cost factor being inversely proportional to the data size; Extracting the proportion of the remaining effective time of the data to be accessed to