CN-122018820-A - Data multi-stage storage method and system

CN122018820ACN 122018820 ACN122018820 ACN 122018820ACN-122018820-A

Abstract

The application provides a data multi-level storage method and a system, wherein the method comprises the steps of reading vehicle running data from a vehicle-mounted sensor and a remote information processor; and storing the vehicle running data according to a hierarchical storage configuration rule corresponding to the data storage level of the vehicle running data. According to the application, the storage level of the vehicle driving data is determined through the double-factor data hierarchical model, so that the data storage efficiency is improved, and the waste of storage resources is avoided.

Inventors

JIANG QINGSONG
HE LIYANG
WANG XIAOHONG
REN YI
CHEN ZHIQIANG

Assignees

赛力斯汽车有限公司

Dates

Publication Date: 20260512
Application Date: 20260413

Claims (10)

1. A method of multi-level storage of data, the method comprising: reading vehicle travel data from the vehicle-mounted sensor and the telematics processor; Determining a data storage level corresponding to the vehicle driving data by using a two-factor data classification model based on time and criticality, wherein the two-factor data classification model comprises a time classification sub-model, a criticality classification sub-model and a two-factor mapping sub-model; storing the vehicle running data according to a hierarchical storage configuration rule corresponding to the data storage level of the vehicle running data; The step of determining the data storage level corresponding to the vehicle driving data by using a two-factor data grading model based on time and criticality comprises the following steps: Inputting the data acquisition time corresponding to the vehicle running data into the time grading sub-model, and determining the query level corresponding to the vehicle running data; Inputting the data types related to the vehicle driving data into the criticality grading sub-model, and determining the criticality grade corresponding to the vehicle driving data; And inputting the query level and the criticality level corresponding to the vehicle running data into the two-factor mapping sub-model, and determining a data storage level corresponding to the vehicle running data.
2. The method of claim 1, wherein the two-factor mapping sub-model describes a mapping relationship between a query level, a criticality level, and a data storage level corresponding to vehicle travel data.
3. The method of claim 2, wherein the query levels include a real-time level, a near-term level, and a history level, The time grading sub-model determines a query level corresponding to vehicle driving data by the following method: Calculating the time difference between the data acquisition time corresponding to the vehicle running data and the current time; if the time difference is smaller than or equal to the first time period, determining that the query level of the vehicle driving data is a real-time level; if the time difference is greater than the first time period and the time difference is less than or equal to the second time period, determining that the query level of the vehicle running data is a recent level; and if the time difference is larger than a second time period, determining that the query level of the vehicle running data is a history level, wherein the second time period is larger than the first time period.
4. The method of claim 2, wherein the vehicle travel data comprises a plurality of travel parameter sample data at the same sample time, the travel parameter sample data comprising a data type and a sample data value corresponding to a travel parameter, The criticality grading sub-model determines a criticality level corresponding to vehicle travel data by: Determining a key degree basic score and a key degree index weight corresponding to the driving parameter sampling data according to the data type corresponding to the driving parameter sampling data; Determining a state coefficient corresponding to the driving parameter sampling data based on a dynamic threshold library corresponding to the vehicle and a sampling data value corresponding to the driving parameter; Determining a criticality judgment score corresponding to the vehicle driving data according to the criticality basic score, the criticality index weight and the state coefficient corresponding to each driving parameter sampling data; Searching a preset criticality level judgment table, and determining a criticality level corresponding to the vehicle running data according to the criticality judgment score corresponding to the vehicle running data, wherein the preset criticality level judgment table describes the mapping relation between the criticality level corresponding to the vehicle running data and the criticality judgment score.
5. The method of claim 4, wherein the step of determining the criticality determination score corresponding to the vehicle travel data based on the criticality base score, the criticality index weight, and the state coefficient corresponding to each of the travel parameter sample data comprises: Calculating a first product between a criticality basic score corresponding to the driving parameter sampling data and the state coefficient, and determining the first product as a first key index score corresponding to the driving parameter sampling data; Calculating a second product between the first key index score corresponding to each driving parameter sampling data and the key index weight, and determining the second product as a second key index score corresponding to the driving parameter sampling data; and determining the sum among the second key index scores corresponding to all the driving parameter sampling data as a criticality judgment score of the vehicle driving data.
6. The method of claim 1, wherein the hierarchical storage configuration rule records a mapping relationship between data storage levels, data storage tables, compression algorithms, partition policies, storage locations, and data retention deadlines, Wherein the step of storing the vehicle travel data according to a hierarchical storage configuration rule corresponding to a data storage level of the vehicle travel data includes: writing the vehicle running data into a corresponding data storage table according to a compression algorithm and a partition strategy corresponding to the data storage level of the vehicle running data; and refreshing the storage medium where the data storage table is located.
7. The method of claim 6, wherein the data storage levels include a first storage level, a second storage level, and a third storage level from high to low, Wherein the vehicle travel data is written into the corresponding data storage table by: If the vehicle running data is of a first storage level, adopting a ZSTD compression algorithm and a partitioning strategy of partitioning according to minutes to write the vehicle running data into a corresponding data storage table; If the vehicle running data is of the second storage level, writing the vehicle running data into a corresponding data storage table by adopting an LZ4 compression algorithm and a partition strategy according to the time division of the cells; and if the vehicle running data is of the third storage level, adopting SNAPPY compression algorithm and partition strategy according to the daily partition, and writing the vehicle running data into a corresponding data storage table.
8. The method of claim 6, wherein the method further comprises: Determining the vehicle driving data to be migrated, which meets the migration triggering condition, in the data storage table according to the data retention period corresponding to the data storage table and the data storage level of the vehicle driving data in the data storage table; migrating the driving data of the vehicle to be migrated from the original data storage table to which the driving data of the vehicle to be migrated are located to a target data storage table; Deleting the driving data of the vehicle to be migrated from the corresponding original data storage table after the migration is completed, and updating the storage medium where the original data storage table is located and the storage medium where the target data storage table is located; and generating a data migration log corresponding to migration.
9. The method of claim 4, wherein prior to determining the data storage level corresponding to the vehicle travel data using a two-factor data classification model based on time and criticality, the method further comprises: converting the driving parameter sampling data into a preset acquisition format; And carrying out data verification on the running parameter sampling data subjected to format conversion processing to obtain a plurality of running parameter sampling data passing the data verification.
10. A data multilevel storage system, the system comprising: the data acquisition module is used for reading vehicle running data from the vehicle-mounted sensor and the remote information processor; The double-factor classification module is used for determining a data storage level corresponding to the vehicle driving data by using a double-factor data classification model based on time and criticality, wherein the double-factor data classification model comprises a time classification sub-model, a criticality classification sub-model and a double-factor mapping sub-model; the storage module is used for storing the vehicle running data according to a hierarchical storage configuration rule corresponding to the data storage level of the vehicle running data; wherein, the two factor classification module is further for: Inputting the data acquisition time corresponding to the vehicle running data into the time grading sub-model, and determining the query level corresponding to the vehicle running data; Inputting the data types related to the vehicle driving data into the criticality grading sub-model, and determining the criticality grade corresponding to the vehicle driving data; And inputting the query level and the criticality level corresponding to the vehicle running data into the two-factor mapping sub-model, and determining a data storage level corresponding to the vehicle running data.

Description

Data multi-stage storage method and system Technical Field The present application relates to the field of data processing technologies, and in particular, to a method and a system for storing data in multiple stages. Background The current new energy automobile has huge data volume of running data (including battery SOC, motor rotation speed, vehicle speed, brake frequency, ambient temperature and the like), and the daily average production data of a single automobile can reach 10GB50GB. The existing new energy automobile driving data storage scheme mainly adopts single-factor hierarchical storage, and is mainly divided into two types, namely time-based hierarchical storage, real-time data, recent data and historical data are divided according to data generation time and are respectively stored in different media such as a memory, an SSD (solid state drive), an HDD (hard disk drive) and the like, and key-degree-based hierarchical storage, wherein the data are divided into different key grades according to the importance degree of the data on safety and fault diagnosis of the automobile and are stored in storage media with corresponding performances. The hierarchical storage mode which only depends on time or criticality single dimension has obvious defects that on one hand, when the data is only classified according to time, the historical data with high criticality is easily migrated to a low-speed storage medium, the fault tracing and data analysis efficiency is reduced, and when the data is only classified according to criticality, the real-time data with low criticality occupies high-speed storage resources, so that the storage space is wasted. On the other hand, the existing storage scheme mostly adopts a relational database or a conventional distributed file system, is difficult to adapt to the service characteristics of dense writing of running data and screening and reading according to grades of new energy automobiles, generally has the problems of high data writing delay, slow query response and the like, and cannot meet the requirements of efficient storage and quick access of large-scale vehicle data. Disclosure of Invention Therefore, the application aims to at least provide a data multi-stage storage method and device, which can realize the storage level determination of vehicle driving data through a double-factor data hierarchical model, improve the data storage efficiency and avoid the waste of storage resources. The application mainly comprises the following aspects: In a first aspect, an embodiment of the present application provides a method for storing data in multiple levels, the method including reading vehicle travel data from a vehicle-mounted sensor and a telematics processor, determining a data storage level corresponding to the vehicle travel data using a two-factor data classification model based on time and criticality, and storing the vehicle travel data according to a classification storage configuration rule corresponding to the data storage level of the vehicle travel data. In one possible implementation, the two-factor data classification model comprises a time classification sub-model, a criticality classification sub-model and a two-factor mapping sub-model, wherein the step of determining the data storage level corresponding to the vehicle running data by utilizing the two-factor data classification model based on time and criticality comprises the steps of inputting the data acquisition time corresponding to the vehicle running data into the time classification sub-model, determining the query level corresponding to the vehicle running data, inputting the data type related to the vehicle running data into the criticality classification sub-model, determining the criticality level corresponding to the vehicle running data, inputting the query level corresponding to the vehicle running data and the criticality level into the two-factor mapping sub-model, and determining the data storage level corresponding to the vehicle running data, wherein the two-factor mapping sub-model describes the mapping relation among the query level corresponding to the vehicle running data, the criticality level and the data storage level. In one possible implementation, the query level includes a real-time level, a near-term level and a history level, wherein the time ranking sub-model determines the query level of the vehicle travel data by calculating a time difference between a data collection time corresponding to the vehicle travel data and a current time, determining the query level of the vehicle travel data as the real-time level if the time difference is less than or equal to a first time period, determining the query level of the vehicle travel data as the near-term level if the time difference is greater than the first time period and the time difference is less than or equal to a second time period, and determining the query level of the vehicle travel data as the history level