Search

CN-121092208-B - Method, device, processor and storage medium for realizing task calculation and adaptive parameter adaptation based on information creation big data platform

CN121092208BCN 121092208 BCN121092208 BCN 121092208BCN-121092208-B

Abstract

The invention relates to a method for realizing task calculation and self-adaptive parameter adaptation based on a message creation big data platform, which comprises the following steps of configuring database addresses and data synchronization frequency, calculating index data, selecting different decision models, generating parameters required by calculating new indexes according to GBT regression models if the new indexes are the new indexes, acquiring parameter values for calculating the stock indexes from a knowledge base model if the new indexes are the stock indexes, generating parameters required by calculating the stock indexes according to the knowledge base model, and receiving parameters and operation packages required by calculation tasks generated by the GBT regression models and the knowledge base model. The method, the device, the processor and the computer readable storage medium thereof for realizing task calculation and self-adaptive parameter adaptation based on the information creation big data platform are adopted, the index calculation service efficiency, the robustness and the safety are improved based on the mixed decision model, the knowledge base model learning is sustainable, the model precision is continuously optimized, and the coverage rate and the accuracy are continuously improved through closed loop feedback.

Inventors

  • YU FENG
  • TAO HUIYONG
  • ZHU HONGSHUAI
  • Mei Kebo
  • XIE LIJUN
  • GAO YU
  • ZHANG YI
  • LI SHUANG
  • DENG YANJUN

Assignees

  • 国泰海通证券股份有限公司

Dates

Publication Date
20260512
Application Date
20250718

Claims (8)

  1. 1. A method for realizing task calculation and self-adaptive parameter adaptation based on a credit creation big data platform is characterized by comprising the following steps: (1) Preparing a credit creation big data platform, an index calculation service, a self-adaptive parameter regulator, a collector and a credit creation database; (2) Installing a credit collector, configuring a database address and a data synchronization frequency, and configuring a cryptographic algorithm sm2 to encrypt transmission data; (3) Acquiring initialization parameters and calculating index data; (4) Receiving submitted tasks, selecting different decision models according to the types of the index tasks, and continuing the step (5) if the tasks are newly added indexes, and continuing the step (6) if the tasks are stock indexes; (5) Generating parameters required for calculating the new index according to the GBT regression model; (6) Obtaining parameter values for calculating the stock index from the knowledge base model, and generating parameters required by calculating the stock index according to the knowledge base model; (7) Parameters and operation packages required by calculation tasks generated by the GBT regression model and the knowledge base model are received and submitted to a credit creation big data platform; (8) Performing resource scheduling and performance monitoring; The step (5) specifically comprises the following steps: (5.1) constructing a characteristic model; (5.2) optimizing model training; (5.3) evaluating the calculation time eva_time of the predicted data based on the regression model, the predicted data set and the calculation task dynamic parameters, reading the task calculation time computer_time of the created database obtained created large data platform, calculating the root mean square error by using a regression task evaluator, and evaluating the performance condition of the model; (5.4) updating the knowledge base model according to the input characteristics and the output characteristics of the model training; (5.5) dynamically increasing and decreasing the characteristic values of the dynamic parameters for tuning according to the server configuration and the model performance condition; The step (5.1) specifically comprises the following steps: (5.1.1) constructing a time window characteristic, and calculating average time consumption of a plurality of indexes in the past; (5.1.2) inputting system resource characteristics, data scale characteristics, calculation task dynamic parameter characteristics and outputting time sequence characteristics; (5.1.3) merging the system resources, the data scale, the dynamic parameter characteristics of the computing task and the time series characteristics into a single vector column; (5.1.4) performing standardization treatment; the step (6) specifically comprises the following steps: (6.1) storing the data into a parameter knowledge base, and calculating task parameter values; (6.2) setting a confidence coefficient threshold value, judging the confidence coefficient, and triggering the self-adaptive parameter adjustment; the step (6.1) specifically comprises the following steps: (6.1.1) filtering the operation results of the performance monitoring of the input characteristic values and the output characteristic values generated by the GBT regression model through a rule engine and storing the operation results into a parameter knowledge base; (6.1.2) providing parameter values required by the inventory index calculation task, inputting system resource characteristics and data scale characteristics, and obtaining the closest calculation task parameter values according to the similarity variance; the step (6.2) specifically comprises the following steps: (6.2.1) storing the performance data of training tuning and performance monitoring into a parameter knowledge base through a rule engine; (6.2.2) setting a confidence threshold based on the system resource characteristics and the data size characteristics; (6.2.3) carrying out confidence judgment according to the index data of the performance monitoring, triggering self-adaptive parameter adjustment if the confidence is judged to be low, triggering model retraining if the confidence is judged to be middle-set, and storing the characteristic value and the performance data into a parameter knowledge base if the confidence is judged to be high; The rule engine is used for storing the performance data of training optimization and performance monitoring into a parameter knowledge base through the rule engine, setting a confidence judging rule and a labeling flow, and improving an active learning mechanism of the knowledge base; The method (6.2.3) specifically comprises the following steps: if cpu and memory exceed 70% or the calculation time is more than 1 hour, triggering manual marking as low confidence, not storing in a knowledge base, triggering model retraining and self-adaptive parameter adjustment; If the cpu and the memory are less than 70% or the calculation time is more than 1 hour, judging that the central confidence is achieved, triggering the model to retrain, temporarily not regenerating parameters required by the calculation task, and storing the characteristic values and the performance data into a knowledge base; If cpu and memory are less than 70% or the calculation time is less than 1 hour, the high confidence is determined, and the feature value and performance data are stored in the knowledge base.
  2. 2. The method for implementing task calculation and adaptive parameter adaptation based on the big data platform of claim 1, wherein the step (2) specifically comprises the following steps: (2.1) installing a credit collector, downloading a driver corresponding to the database, and importing the driver into an installation directory; (2.2) selecting the database types of an external data source and a credit database, configuring the address and the data synchronization frequency of the database, verifying whether the connection is normal, if so, continuing the step (2.3), otherwise, continuing the step (2.2); And (2.3) loading an acquisition script, configuring a cryptographic algorithm sm2 to encrypt transmission data, and importing external historical market data into a credit database.
  3. 3. The method for implementing task calculation and adaptive parameter adaptation based on the big data platform of claim 1, wherein the step (3) specifically comprises the following steps: (3.1) reading the loading history quotation data from the credit database; (3.2) acquiring initialization parameters required to be set for submitting the task from a parameter knowledge base; (3.3) marking the index calculation service as a jar packet, and submitting the jar packet to the task decision service.
  4. 4. The method for implementing task calculation and adaptive parameter adaptation based on the big data platform of claim 1, wherein the step (5.2) specifically comprises the following steps: (5.2.1) dividing the data set into training set data and prediction set data according to a time sequence; (5.2.2) training using the GBT regression model based on the training set data to obtain a regression model.
  5. 5. The method for implementing task calculation and adaptive parameter adaptation based on the big data platform of claim 1, wherein the step (8) specifically comprises the following steps: (8.1) accessing the read data through a standard interface; (8.2) receiving the task and the parameter submitted by the self-adaptive parameter regulator, completing the resource scheduling and the task allocation, and calculating an index task; (8.3) storing the calculated performance data and the result data in a knowledge base; and (8.4) monitoring the operation performance of the server in real time through the performance monitoring service, alarming if the threshold value is reached, and storing the performance data into a knowledge base.
  6. 6. The device for realizing task calculation and self-adaptive parameter adaptation based on the information creation big data platform is characterized by comprising the following components: A processor configured to execute computer-executable instructions; A memory storing one or more computer-executable instructions which, when executed by the processor, perform the steps of the method for performing task computation and adaptive parameter adaptation based on a big-data-over-a-message platform of any of claims 1 to 5.
  7. 7. A processor for performing task computation and adaptive parameter adaptation based on a big data platform, wherein the processor is configured to execute computer executable instructions that, when executed by the processor, perform the steps of the method for performing task computation and adaptive parameter adaptation based on a big data platform as claimed in any of claims 1 to 5.
  8. 8. A computer readable storage medium having stored thereon a computer program executable by a processor to perform the steps of the method of any one of claims 1 to 5 for task calculation and adaptive parameter adaptation based on a big data platform.

Description

Method, device, processor and storage medium for realizing task calculation and adaptive parameter adaptation based on information creation big data platform Technical Field The invention relates to the field of financial securities, in particular to the field of transaction strategy platforms, and specifically relates to a method, a device, a processor and a computer readable storage medium thereof for realizing task calculation and self-adaptive parameter adaptation based on a credit-creation big data platform. Background In order to meet the multi-scene investment decision function of securities clients, the trading strategy platform needs to calculate processing strategy indexes based on massive market data, and a java application index calculation program is submitted to a big data platform through spark-submit by means of a signal creation big data platform to complete complex index task calculation. With the improvement of the security, stability and intelligent requirements of the financial security industry on a customer service system, a certain challenge exists for the development of the existing trading strategy platform. The existing transaction strategy platform calculates strategies, operation parameters are required to be manually set when a calculation task is submitted, and the parameters cannot be dynamically modified according to the operation condition of the system after the operation parameters are set. Meanwhile, middleware and the like adopted by the existing transaction strategy platform are not completely produced, and certain potential safety hazards exist. Disclosure of Invention The invention aims to overcome the defects of the prior art and provide a method, a device, a processor and a computer readable storage medium thereof for realizing task calculation and self-adaptive parameter adaptation based on a created big data platform, which have high safety, high calculation service efficiency and wider application range. In order to achieve the above objective, the method, the device, the processor and the computer readable storage medium thereof for achieving task calculation and adaptive parameter adaptation based on the information creation big data platform of the present invention are as follows: The method for realizing task calculation and self-adaptive parameter adaptation based on the information creation big data platform is mainly characterized by comprising the following steps: (1) Preparing a credit creation big data platform, an index calculation service, a self-adaptive parameter regulator, a collector and a credit creation database; (2) Installing a credit collector, configuring a database address and a data synchronization frequency, and configuring a cryptographic algorithm sm2 to encrypt transmission data; (3) Acquiring initialization parameters and calculating index data; (4) Receiving submitted tasks, selecting different decision models according to the types of the index tasks, and continuing the step (5) if the tasks are newly added indexes, and continuing the step (6) if the tasks are stock indexes; (5) Generating parameters required for calculating the new index according to the GBT regression model; (6) Obtaining parameter values for calculating the stock index from the knowledge base model, and generating parameters required by calculating the stock index according to the knowledge base model; (7) Parameters and operation packages required by calculation tasks generated by the GBT regression model and the knowledge base model are received and submitted to a credit creation big data platform; (8) And carrying out resource scheduling and performance monitoring. Preferably, the step (2) specifically includes the following steps: (2.1) installing a credit collector, downloading a driver corresponding to the database, and importing the driver into an installation directory; (2.2) selecting the database types of an external data source and a credit database, configuring the address and the data synchronization frequency of the database, verifying whether the connection is normal, if so, continuing the step (2.3), otherwise, continuing the step (2.2); And (2.3) loading an acquisition script, configuring a cryptographic algorithm sm2 to encrypt transmission data, and importing external historical market data into a credit database. Preferably, the step (3) specifically includes the following steps: (3.1) reading the loading history quotation data from the credit database; (3.2) acquiring initialization parameters required to be set for submitting the task from a parameter knowledge base; (3.3) marking the index calculation service as a jar packet, and submitting the jar packet to the task decision service. Preferably, the step (5) specifically includes the following steps: (5.1) constructing a characteristic model; (5.2) optimizing model training; (5.3) evaluating the calculation time eva_time of the predicted data based on the regression model, the predicted