CN-116303460-B - Data dimension table generation method and device, server and storage medium

CN116303460BCN 116303460 BCN116303460 BCN 116303460BCN-116303460-B

Abstract

The application relates to the technical field of big data, in particular to a method, a device, a server and a storage medium for generating a data dimension table, wherein the method comprises the following steps of creating an automatic configuration item, and introducing remote procedure call dependence, geographic index and component transaction monitor dependence; the method comprises the steps of providing a service LBS interface based on a position, receiving a call request by the service LBS interface, obtaining consumption data of a distributed publishing and subscribing message system Kafka according to the call request, analyzing the consumption data to obtain positioning data and unique identification of a vehicle, inquiring a topology tree by using the position data to obtain position coding data of the vehicle, inquiring a database by using the unique identification to obtain vehicle information of the vehicle, generating a data maintenance table according to the coding data and the vehicle information, and synchronizing the data maintenance table to the database. Therefore, the problems of overlarge external access pressure, longer data delay, poor expandability and the like in the data dimension table generation process in the related technology are solved.

Inventors

SHI XINGLIANG

Assignees

雄狮汽车科技(南京)有限公司
芜湖雄狮汽车科技有限公司
奇瑞汽车股份有限公司

Dates

Publication Date: 20260508
Application Date: 20230321

Claims (8)

1. The method for generating the data dimension table is characterized by being applied to a server, wherein the server comprises a topology tree and a memory-based database which are arranged in a memory, and the method comprises the following steps: creating an automatic configuration item, introducing remote procedure call dependence, a geographic index and a component transaction monitor dependence, wherein the automatic configuration item is utilized to configure data to the topology tree and the database, serialization and network communication are realized by utilizing the remote procedure call dependence, service places in a vehicle target range are searched by utilizing the geographic index, a spatial position relation and a minimum diameter are correspondingly calculated by utilizing the component transaction monitor dependence for an algorithm in topology service, the automatic configuration item is utilized to configure the data to the topology tree and the database, and condition annotation is automatically added to a class under a current class path by utilizing the automatic configuration item, and position coding data is stored in the topology tree and the vehicle information is stored in the database according to the condition annotation; Providing a location-based service LBS interface, wherein a call request is received by utilizing the service LBS interface, consumption data of a distributed publishing and subscribing message system Kafka is acquired according to the call request, the consumption data is analyzed to obtain positioning data and unique identification of a vehicle, the topology tree is queried by utilizing the position data to obtain position coding data of the vehicle, the database is queried by utilizing the unique identification to obtain vehicle information of the vehicle, a data dimension table is generated according to the coding data and the vehicle information, and the data dimension table is synchronized to the database.
2. The method of claim 1, wherein the serialization uses a binary stream and the network communication uses a Socket communication mechanism.
3. The method as recited in claim 1, further comprising: And introducing a ring buffer zone, and writing the analysis result of the vehicle history track into the database by using the ring buffer zone, wherein the ring buffer zone is a data structure in an embedded system, and returns to the starting point of the ring buffer zone when a pointer is used for accessing the position behind the last memory position of the buffer zone.
4. The method for generating the data dimension table is characterized by being applied to a server, wherein the server comprises a topology tree and a memory-based database which are arranged in a memory, and the method comprises the following steps: Creating an automatic configuration item, utilizing the automatic configuration item to configure data to the topology tree and the database, introducing remote procedure call dependence, utilizing the remote procedure call dependence to realize serialization and network communication, introducing a geographic index, utilizing the geographic index to search a service place in a vehicle target range, introducing a component transaction monitor dependence, utilizing the component transaction monitor dependence to correspondingly calculate a spatial position relationship and a minimum diameter by utilizing an algorithm in topology service; Identifying a call request of a location-based service LBS service interface; Acquiring consumption data of the distributed publish-subscribe message system Kafka according to the call request, and analyzing the consumption data to obtain positioning data and unique identification of the vehicle; And inquiring the topology tree by using the position data to obtain position coding data of the vehicle, inquiring the database by using the unique identification to obtain vehicle information of the vehicle, generating a data dimension table according to the coding data and the vehicle information, and synchronizing the data dimension table to the database.
5. A device for generating a data dimension table, wherein the device is applied to a server, the server comprises a topology tree and a memory-based database which are arranged in a memory, and the device comprises: The system comprises a creation module, a control module and a control module, wherein the creation module is used for creating an automatic configuration item, introducing remote procedure call dependence, a geographic index and component transaction monitor dependence, utilizing the automatic configuration item to configure data to the topology tree and the database, utilizing the remote procedure call dependence to realize serialization and network communication, utilizing the geographic index to search a service place in a vehicle target range, utilizing the component transaction monitor dependence to supply an algorithm in topology service, and correspondingly calculating a spatial position relationship and a minimum diameter; The service LBS interface is used for receiving a call request, acquiring consumption data of the distributed publishing and subscribing message system Kafka according to the call request, analyzing the consumption data to obtain positioning data and unique identification of a vehicle, inquiring the topology tree by using the position data to obtain position coding data of the vehicle, inquiring the database by using the unique identification to obtain vehicle information of the vehicle, generating a data dimension table according to the coding data and the vehicle information, and synchronizing the data dimension table to the database.
6. A device for generating a data dimension table, wherein the device is applied to a server, the server comprises a topology tree and a memory-based database which are arranged in a memory, and the device comprises: the identification module is used for identifying the calling request of the location-based service LBS service interface; The acquisition module is used for acquiring consumption data of the distributed publishing and subscribing message system Kafka according to the calling request, and analyzing the consumption data to obtain positioning data and unique identification of the vehicle; The generating module is used for inquiring the topology tree by utilizing the position data to obtain position coding data of the vehicle, inquiring the database by utilizing the unique identification to obtain vehicle information of the vehicle, generating a data dimension table according to the coding data and the vehicle information, and synchronizing the data dimension table to the database.
7. A server comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor executing the program to implement the method of generating a data dimensional table as claimed in any one of claims 1 to 4.
8. A computer-readable storage medium, on which a computer program is stored, characterized in that the program is executed by a processor for implementing a method of generating a data dimension table according to any of claims 1-4.

Description

Data dimension table generation method and device, server and storage medium Technical Field The present application relates to the field of big data technologies, and in particular, to a method, an apparatus, a server, and a storage medium for generating a data dimension table. Background In actual production of large data analysis of the internet of vehicles, an original data stream is often required to be used as a basis, then a large number of external basic tables are associated to supplement some attributes, in the flank streaming calculation, the dimension table data are updated at regular time, association is required according to a service, and the following related technologies exist according to the timeliness requirement of the service on the dimension table data association: (1) Real-time query of the dimension table generally causes thread blocking and Task waiting data return, and influences throughput of the whole Task. (2) The full data is preloaded, and the dimension table data is cached in the memory, so that the memory consumption of the computing node is high, and the method cannot be applied to a large number of dimension tables. (3) LRU cache in this way has some data delay and requires additional dead time for each piece of data, so once the change of the dimension table occurs, the change of the data cannot be perceived because the hot spot data is always in the cache because the hot spot data is frequently used. (4) A large data processing dimension table storage computing system based on cache and a method thereof are provided, which mainly comprise dimension table data writing and dimension table association computing, wherein the dimension table data writing can acquire data from batch or stream data sources, the data is stored into Redis cache service after being cleaned and converted, the dimension table association computing is applied to data information complementation, service data is acquired from batch or stream data sources, corresponding expansion data is read in Redis cache according to index Key to complete the complementation of service data missing field, but memory overflow problem caused by overlarge associated data exists in the cache, and the Redis itself has limitation on the size of single Key. Disclosure of Invention The application provides a method, a device, a server and a storage medium for generating a data dimension table, which are used for solving the problems of overlarge external access pressure, longer data delay, poor expandability and the like in the process of generating the data dimension table in the related technology. An embodiment of the first aspect of the present application provides a method for generating a data dimension table, the method being applied to a server, the server comprising a topology tree and a memory-based database arranged in a memory, wherein the method comprises the steps of creating an automatic configuration item, introducing a remote procedure call dependency, a geographic index and a component transaction monitor dependency, wherein the automatic configuration item is used for configuring data to the topology tree and the database, the remote procedure call dependency is used for realizing serialization and network communication, the geographic index is used for searching a service place in a target range of a vehicle, the component transaction monitor is used for relying on an algorithm in a topology service, a spatial position relation and a minimum diameter are correspondingly calculated, a position-based service LBS (Location Based Services, a position-based service interface is provided, wherein a call request is received by using the service LBS service interface, consumption data of a distributed release subscription message system Kafka is acquired according to the call request, the consumption data is parsed to obtain location data and a unique identifier of the vehicle, the topology tree is searched by using the location data to obtain the location data, the vehicle location data is used for obtaining a position identifier of the vehicle, the vehicle is used for obtaining the unique data, the vehicle code table is used for obtaining the vehicle data, and the vehicle code table is used for obtaining the unique data. Optionally, in one embodiment of the present application, the configuring the data to the topology tree and the database by using the automatic configuration item includes automatically adding a condition annotation to a class under a current class path by using the automatic configuration item, storing the position-coding data in the topology tree according to the condition annotation, and storing the vehicle information in the database. Alternatively, in one embodiment of the present application, the serialization uses a binary stream and the network communication uses a Socket communication mechanism. Optionally, in one embodiment of the present application, the method for generating the d