CN-121997285-A - Big data analysis platform system based on artificial intelligence

CN121997285ACN 121997285 ACN121997285 ACN 121997285ACN-121997285-A

Abstract

The invention relates to the technical field of artificial intelligence and big data processing, and discloses a big data analysis platform system based on artificial intelligence, the system comprises a data access layer, a data preprocessing layer, an AI analysis engine layer, a data storage layer, an application service layer and a monitoring operation and maintenance layer which are connected in sequence in a communication way. The data access layer realizes multi-source heterogeneous data access and high-frequency data cache, the preprocessing layer completes data cleaning conversion and quality detection, the AI analysis engine layer realizes full life cycle management, intelligent analysis and result optimization and interpretation of a model, the data storage layer adopts a hybrid architecture for classified storage and remote backup, the application service layer provides visual display and an open interface, and the monitoring operation and maintenance layer ensures stable operation of a system. The invention improves the data processing efficiency and the analysis precision, and enhances the data security and the user experience.

Inventors

CHEN FEI
XU HAITAO
ZHAO JIANHUA
SHEN DAWEI

Assignees

北京阿尔法风控科技有限公司

Dates

Publication Date: 20260508
Application Date: 20260109

Claims (10)

1. The big data analysis platform system based on artificial intelligence is characterized by comprising a data access layer, a data preprocessing layer, an AI analysis engine layer, a data storage layer and an application service layer which are sequentially in communication connection; the data access layer is used for accessing multi-source heterogeneous data; The data preprocessing layer is used for cleaning, converting, integrating and carrying out protocol processing on the original data accessed by the data access layer to obtain standardized data to be analyzed; The AI analysis engine layer is used for calling an analysis model to carry out intelligent analysis on the standardized data to be analyzed and optimizing an analysis result; the data storage layer is used for classifying and storing the original data, the preprocessed data and the analysis result data; the application service layer is used for providing a visual data display and analysis service interface for the user.
2. The big data analysis platform system based on artificial intelligence according to claim 1, wherein the data access layer is further configured with a data caching module for temporarily caching high frequency data accessed in real time.
3. The big data analysis platform system based on artificial intelligence according to claim 1, wherein the data preprocessing layer comprises a data conversion module and a data quality detection module, wherein the data conversion module is used for realizing standardized conversion of different format data, and the data quality detection module is used for detecting and repairing missing values, abnormal values and repeated values in original data.
4. The big data analysis platform system based on artificial intelligence according to claim 1, wherein the AI analysis engine layer comprises a model management sub-module, an intelligent analysis sub-module and a result optimization sub-module; The model management sub-module is used for realizing registration, deployment, update and cancellation of the analysis model; The intelligent analysis sub-module is used for calling a target analysis model in the model management sub-module and carrying out feature extraction, trend prediction and association analysis on the standardized data to be analyzed; And the result optimization sub-module is used for verifying the accuracy of the analysis result of the intelligent analysis sub-module and carrying out iterative optimization on the parameters of the target analysis model when the accuracy does not reach a preset threshold.
5. The artificial intelligence based big data analysis platform system of claim 4, wherein the model management sub-module further comprises a model training sub-module for training or fine tuning the base model based on the user provided annotation data set to generate the customized analysis model.
6. The big data analysis platform system based on artificial intelligence according to claim 4, wherein the result optimizing sub-module further comprises an analysis result interpretation sub-module, the analysis result interpretation sub-module adopts an interpretable AI algorithm to perform visual interpretation on the generation process of the analysis result and key influence factors.
7. The big data analysis platform system based on artificial intelligence according to claim 1, wherein the data storage layer adopts a hybrid storage architecture, and comprises a relational database, a distributed file system and a time sequence database, which are respectively used for storing structured data, unstructured data and time sequence type analysis data.
8. The big data analysis platform system based on artificial intelligence according to claim 1 or 7, wherein the data storage layer further comprises a data backup sub-module for periodically backing up the stored data by combining incremental backup with full backup, and storing the backup data to a remote backup node.
9. The big data analysis platform system based on artificial intelligence according to claim 1, wherein the visualized data presentation interface provided by the application service layer supports the generation and style customization of various data visualized charts.
10. The big data analysis platform system based on artificial intelligence according to claim 1, further comprising a monitoring operation and maintenance layer, wherein the monitoring operation and maintenance layer is in communication connection with the data access layer, the data preprocessing layer, the AI analysis engine layer, the data storage layer and the application service layer, and is used for monitoring the operation state, the resource occupancy rate and the task execution progress of each layer of module in real time and triggering an alarm when an abnormality is detected.

Description

Big data analysis platform system based on artificial intelligence Technical Field The invention relates to the technical field of artificial intelligence and big data processing, in particular to a big data analysis platform system based on artificial intelligence. Background With the rapid development of information technology, especially the wide application of internet of things, mobile internet and cloud computing, data presents an explosive growth trend, and data types are increasingly diversified, covering structured, semi-structured and unstructured data. The huge amount of heterogeneous data contains abundant business value, and how to efficiently collect, store, process and analyze the data becomes an important problem to be solved in the current enterprise and scientific research fields. The traditional big data processing platform often has the defects of limited data access capability, imperfect preprocessing, single analysis model, lack of dynamic optimization mechanism, low storage efficiency, complex system operation and maintenance and the like, and is difficult to meet the requirements of modern intelligent data analysis. Meanwhile, the rapid development of artificial intelligence technology provides new power for big data analysis. Through machine learning, deep learning and other technologies, deeper rules and trends can be mined from complex data, and the accuracy and the intelligent level of analysis are improved. However, the existing artificial intelligent big data analysis platform still has many challenges in terms of model management, result interpretation and overall system coordination, such as difficulty in flexibly updating and customizing a model, lack of interpretability of analysis results, difficulty in realizing efficient data flow and resource monitoring and the like, and limits popularization and effect exertion in practical application. Therefore, an intelligent big data analysis platform system which can support efficient access and processing of multi-source heterogeneous data, has a perfect data quality guarantee mechanism, integrates an advanced artificial intelligent analysis model and supports dynamic optimization and interpretability is urgently needed. Meanwhile, the system has an efficient data storage scheme and sound monitoring operation and maintenance capability, and ensures that the platform runs stably and reliably so as to meet diversified demands of enterprises in the aspects of data driving decision, business optimization and intelligent service. Based on the above, the development of the big data analysis platform system based on artificial intelligence with reasonable structure, perfect functions and excellent performance becomes an urgent need of the important direction and practical application of the current technical development. Disclosure of Invention In order to overcome the defects of the prior art, the invention aims to overcome the defects of the existing big data analysis system, and provides a big data analysis platform system based on artificial intelligence, which realizes high-efficiency access and caching of multi-source heterogeneous data, accurate data preprocessing, intelligent analysis and result optimization, classified safe storage and convenient visual application service, and ensures the stable operation of the system through comprehensive monitoring operation and maintenance. In order to achieve the purpose, the technical scheme of the invention is realized by the fact that the big data analysis platform system based on artificial intelligence comprises a data access layer, a data preprocessing layer, an AI analysis engine layer, a data storage layer and an application service layer which are connected in sequence in a communication mode, and the big data analysis platform system further comprises a monitoring operation and maintenance layer which is connected with all the layers in a communication mode. The specific structure and functions of each layer are as follows: the data access layer is used for accessing multi-source heterogeneous data, wherein the multi-source heterogeneous data comprises but is not limited to structured data (such as relational database data), semi-structured data (such as XML, JSON data) and unstructured data (such as text, image and audio data). In order to improve the processing efficiency of the real-time high-frequency data, the data access layer is further configured with a data buffer module for temporarily buffering the real-time accessed high-frequency data, so that the data transmission delay is reduced, and the efficient operation of the subsequent processing links is ensured. The data preprocessing layer is used for cleaning, converting, integrating and carrying out protocol processing on the original data accessed by the data access layer to obtain standardized data to be analyzed. The data preprocessing layer comprises a data conversion module and a data quality detection module, whe