CN-115840603-B - General algorithm verification platform and algorithm registration calling method thereof

CN115840603BCN 115840603 BCN115840603 BCN 115840603BCN-115840603-B

Abstract

The invention discloses a general algorithm verification platform and an algorithm registration calling method thereof, wherein the platform provides an algorithm plug-in framework template of an open source code, the algorithm plug-in framework comprises a plurality of languages, a user integrates an algorithm by means of the algorithm plug-in framework and inputs a processing result according to the network interface protocol requirement of interaction of the algorithm and the platform, the user algorithm does not consider related content of network communication, an interface of interaction of the algorithm and the platform adopts a TCP/IP communication mode, the uploaded algorithm is used as a TCP client side and is actively pulled up by the platform and is transmitted with connection information of a TCP server side, and initialization parameter information, the algorithm continuously tries to be connected with the platform, and when the connection is successful, the information of a process ID is automatically transmitted, the platform data input is waited, the platform transmits the data into the algorithm, and the algorithm feeds the processing result back to the platform. The invention is convenient for the user to verify and integrate different algorithms, thereby reducing the development cost of the user to the greatest extent.

Inventors

WANG HUAN
LIAO XIAOMIN
TAN MING
ZHAO ZHIYUAN
LIU LONGWU
SHANG DONGDONG
JIN XIAONING
Kang Guoqin
QIN JIN
LIU LU
SONG WEI
LIU HAN
CHEN GUIFANG
WANG HUAXIA
ZHU HANG

Assignees

中国航天科工集团八五一一研究所
中国人民解放军国防科技大学

Dates

Publication Date: 20260505
Application Date: 20221222

Claims (6)

1. The universal algorithm verification platform is characterized in that the platform provides an algorithm plug-in frame template of open source codes, the algorithm plug-in frame comprises multiple languages, a user integrates an algorithm by means of the algorithm plug-in frame, and according to the network interface protocol requirement of the interaction of the algorithm and the platform, a processing result is filled in, and the user algorithm does not consider related content of network communication; The interface of the algorithm and platform interaction adopts a TCP/IP communication mode, the uploaded algorithm is used as a TCP client, is actively pulled up by the platform, and is transmitted with the connection information of a TCP server and initialization parameter information; The algorithm registration call comprises three processing procedures of algorithm uploading, algorithm registration starting and algorithm call; the algorithm uploading specifically comprises the following steps: the method comprises the steps that (1.1) a user uploads an algorithm plug-in to a data management module of a platform through an algorithm management interface of the platform, and when uploading, an uploading button is clicked in the algorithm management interface, and an algorithm name, an algorithm category, an initialization parameter, an algorithm starting relative path and whether to use the information are filled in a popup window; After receiving the algorithm plug-in and the description information uploaded by the front end, the data management module of the platform generates a corresponding algorithm ID according to the algorithm type, and stores the information into a database according to the absolute starting path set by the platform; The method comprises the steps of (1.3) informing a scheduling module to transmit an algorithm plug-in to a node management module on each computing node through network communication, and storing the algorithm module to a designated path by node management on the nodes, so that the function of automatically synchronizing the algorithm to each computing node by a platform is realized; The algorithm registration starting method specifically comprises the following steps: (2.1) the user creates a data processing analysis task and selects an algorithm to be called in the process; (2.2) the conventional computing software receives the task to be processed, analyzes the task flow parameters, finds an algorithm plug-in of the node according to the algorithm ID, and starts the algorithm plug-in; (2.3) after the algorithm plug-in is started, according to the starting parameters of the conventional calculation software module, the algorithm is used as a TCP client and is connected with the conventional calculation software module; (2.4) each conventional computing software module maintains information such as network connection ID, algorithm name, process ID with the algorithm plug-in a mapping table, and the algorithm is registered in the platform; The algorithm call is specifically as follows: when writing business logic and requiring algorithm calling, a user calls an API (application program interface) encapsulated by a platform, and transmits the called algorithm name, function name and input structure body to realize the algorithm calling, and synchronous message, asynchronous message and delay message are realized through setting; The input structure body is in a format appointed by a platform and comprises structured information overall description, a JSON description length, a data body length, JSONs and data bodies, wherein attribute and parameter information is written into the JSONs, data to be processed are stored in the data bodies and are unstructured or semi-structured data, and the platform performs serialization operation on the input data; (3.3) the platform API informs the conventional computing software to transmit the pending data to the algorithm plug-in TCP/IP manner; (3.4) the conventional computing software monitors the result return information of the algorithm plug-in, after receiving the algorithm plug-in return structure, the algorithm plug-in is subjected to deserialization operation, and the result return information is stored in a processing result queue of the algorithm and is notified to a user by means of a platform API (application program interface) to call a service logic algorithm; and (3.5) the user acquires the data processing result by means of the platform API according to the received notification information.
2. The universal algorithm verification platform according to claim 1, wherein the message form of the algorithm interaction with the platform is interacted in a unified way of JSON plus bin attachment, and the input and output parameter description filled in during uploading registration is analyzed and processed.
3. The general algorithm verification platform according to claim 1, wherein the user uploads the algorithm to the system, fills in the algorithm name, the initialization parameters, and whether to use the information, the system automatically synchronizes the algorithm to each computing node, each computing node selects whether to start the algorithm according to whether to use the flag, and automatically maintains the algorithm call mapping table.
4. The generic algorithm verification platform of claim 1, wherein the language used by the algorithm plug-in framework includes Python and c++.
5. An algorithm registration and call method based on the general algorithm verification platform according to any one of claims 1 to 4, which is characterized in that the method performs algorithm registration and synchronization based on a node management module running on a node and a scheduling module on a scheduling node, wherein the scheduling module is a process mainly running on a program, and the node management module runs on each node; The algorithm registration is carried out by logging in the system, the system automatically synchronizes the algorithm to all computing nodes, and the user realizes the verification of the algorithm by writing and calling processing logic; the method comprises three processing procedures of algorithm uploading, algorithm registration starting and algorithm calling; the algorithm uploading specifically comprises the following steps: the method comprises the steps that (1.1) a user uploads an algorithm plug-in to a data management module of a platform through an algorithm management interface of the platform, and when uploading, an uploading button is clicked in the algorithm management interface, and an algorithm name, an algorithm category, an initialization parameter, an algorithm starting relative path and whether to use the information are filled in a popup window; After receiving the algorithm plug-in and the description information uploaded by the front end, the data management module of the platform generates a corresponding algorithm ID according to the algorithm type, and stores the information into a database according to the absolute starting path set by the platform; The method comprises the steps of (1.3) informing a scheduling module to transmit an algorithm plug-in to a node management module on each computing node through network communication, and storing the algorithm module to a designated path by node management on the nodes, so that the function of automatically synchronizing the algorithm to each computing node by a platform is realized; The algorithm registration starting method specifically comprises the following steps: (2.1) the user creates a data processing analysis task and selects an algorithm to be called in the process; (2.2) the conventional computing software receives the task to be processed, analyzes the task flow parameters, finds an algorithm plug-in of the node according to the algorithm ID, and starts the algorithm plug-in; (2.3) after the algorithm plug-in is started, according to the starting parameters of the conventional calculation software module, the algorithm is used as a TCP client and is connected with the conventional calculation software module; (2.4) each conventional computing software module maintains information such as network connection ID, algorithm name, process ID with the algorithm plug-in a mapping table, and the algorithm is registered in the platform; The algorithm call is specifically as follows: when writing business logic and requiring algorithm calling, a user calls an API (application program interface) encapsulated by a platform, and transmits the called algorithm name, function name and input structure body to realize the algorithm calling, and synchronous message, asynchronous message and delay message are realized through setting; The input structure body is in a format appointed by a platform and comprises structured information overall description, a JSON description length, a data body length, JSONs and data bodies, wherein attribute and parameter information is written into the JSONs, data to be processed are stored in the data bodies and are unstructured or semi-structured data, and the platform performs serialization operation on the input data; (3.3) the platform API informs the conventional computing software to transmit the pending data to the algorithm plug-in TCP/IP manner; (3.4) the conventional computing software monitors the result return information of the algorithm plug-in, after receiving the algorithm plug-in return structure, the algorithm plug-in is subjected to deserialization operation, and the result return information is stored in a processing result queue of the algorithm and is notified to a user by means of a platform API (application program interface) to call a service logic algorithm; and (3.5) the user acquires the data processing result by means of the platform API according to the received notification information.
6. The method of claim 5, wherein the method is used in two scenarios, one is the same interface, adapting to multiple algorithms, the algorithm can choose to perform verification test, and the other is a different interface, writing call processing logic according to different algorithms to implement specific software functions.

Description

General algorithm verification platform and algorithm registration calling method thereof Technical Field The invention belongs to the technical field of computer data processing, and particularly relates to a general algorithm verification platform and an algorithm registration and call method thereof. Background The existing electromagnetic signal general distributed intelligent processing analysis platform (for example, patent CN 113157252A) can realize the function of distributed computation in a distributed computation cluster. The distributed intelligent processing analysis platform consists of a scheduling node, a plurality of conventional computing nodes, a plurality of advanced computing nodes, a plurality of data access nodes and the like. The dispatching node is used for deploying a dispatching module as a central of the system and is responsible for dispatching and resource coordination, and the computing node is used for adapting to parallel processing tasks. The UI front-end module receives user task instructions and sends the user task instructions to the rear-end service module through a communication protocol, the node management module starts tasks after the resource monitoring and resource allocation are successful, the data access module, the preprocessing calculation module and the deep learning calculation module interact with the data management module to achieve data storage and management, the result feedback is completed through the communication protocol after the tasks are finished, the rear-end service module is responsible for task scheduling, the node management module is responsible for resource recovery, and the UI front-end module is responsible for result display. The software framework of the platform comprises an Nginx front-end module, a scheduling module, a proxy assistant software module, a data management module, a plurality of data access modules, a plurality of conventional calculation software modules and a plurality of advanced calculation software modules, and can be operated on one computer or a plurality of computers according to roles, for example, the data access software modules are operated on an acquisition server, the calculation nodes are of two types, the conventional calculation software modules are operated on conventional calculation nodes, and the advanced calculation software modules are operated on advanced calculation nodes. In the distributed computing and multi-task concurrency scene, a plurality of clients send task applications to the scheduling software at the same time, each user can apply for a plurality of tasks, and meanwhile, each task can have parallel computing processes and even be distributed on different computing nodes. The essence of the problem is that the scheduling module can receive a plurality of task applications, and can reasonably allocate tasks to corresponding computing resources according to the current computing resource state (comprising a conventional computing software module and an advanced computing software module), and monitor and manage the running condition of the resources. The algorithm of the platform in the patent CN113157252a calls communication through the conventional computing software module of the conventional computing node, and the user logic is also written in the conventional computing software module, but the user has difficulty in verifying and integrating different algorithms, and the development cost of the user is high. Disclosure of Invention The invention aims to provide a general algorithm verification platform and an algorithm registration and call method thereof, which can facilitate the verification and integration of different algorithms by users and reduce the development cost of the users to the greatest extent. The technical scheme for realizing the purpose of the invention is that a general algorithm verification platform provides an algorithm plug-in frame template of open source codes, the algorithm plug-in frame comprises a plurality of languages, a user integrates an algorithm by means of the algorithm plug-in frame, and fills in a processing result according to the network interface protocol requirement of the interaction of the algorithm and the platform, and the user algorithm does not consider related content of network communication; The interface between algorithm and platform adopts TCP/IP communication mode, the uploaded algorithm is used as TCP client end, and is actively pulled up by platform, and connected information of TCP server end and initialization parameter information are transferred, and the algorithm continuously tries to connect with platform, when the connection is successful, automatically sends process ID information, waits for platform data input, the platform transfers data into algorithm, and the algorithm feeds back the processing result to the platform. The method is based on the node management module running on the node and the scheduling modu