KR-20260064941-A - SYSTEM FOR AUTOMATICICALLY GENERATIING A DATA COLLETCION PROGRAM BASED ON OPEN API AND DYNAMICALLY REFLECTING MODIFICATIONS AND ITS EXECUTION METHOD

KR20260064941AKR 20260064941 AKR20260064941 AKR 20260064941AKR-20260064941-A

Abstract

The present invention discloses an open API-based system for automatically generating a data collection program and dynamically reflecting modifications, as well as a method for executing the same. When a dynamic modification is requested, the system modifies the data collection program by modifying the parts that change among the information regarding environment variables, collection modules, data conversion modules, and data schemas. During the operation of the collection program, the system checks whether there are any changes in at least one of the information regarding environment variables, collection modules, and data schemas; if there are changes, it applies the changes; and collects open API-based data based on the applied (modified) environment variables, collection modules, and data schemas.

Inventors

김현수
이무용
오상규
하상욱

Assignees

(주)진우소프트이노베이션

Dates

Publication Date: 20260508
Application Date: 20241030

Claims (9)

A user terminal including a UI, and inputting environment variable settings, collection module settings, data conversion module settings, and data schema settings for generating a collection program through the UI; A program generation and management server including a generation program, which generates the collection program based on environment variable settings, collection module settings, data conversion module settings, and data schema settings input from the user terminal through the generation program, and distributes the collection program to a collection program server; and It includes a collection program server that installs the collection program distributed from the program creation and management server, collects data by accessing an open API based on environment variables and a collection module through the collection program, and converts the collected data based on a data conversion module and a data schema and transmits it to a data loading server. The above user terminal inputs modifications to environment variables, modifications to the collection module, modifications to the data conversion module, and modifications to the data schema for modifying the above collection program through the above UI, and The above program generation and management server includes a management program, and transmits environment variable modifications, collection module modifications, data conversion module modifications, and data schema modifications input from the user terminal through the management program to the collection program server, and An open API-based system for the automatic generation of a data collection program and dynamic reflection of modifications, wherein the collection program server modifies the data collection program by modifying the changing parts of the information regarding environment variables, collection modules, data transformation modules, and data schemas when a dynamic modification is requested from the program generation and management server.
In paragraph 1, The above collection program server is, An open API-based system for automatic generation of a data collection program and dynamic reflection of modifications, which checks whether there is a change in at least one of the information regarding environment variables, collection modules, and data schemas during the operation of the above collection program, and if there is a change in at least one, applies the change to at least one of the environment variables, collection modules, and data schemas.
In paragraph 2, The above collection program server is, When the scheduler of the above collection program for data collection starts, it checks whether environment variables have changed, and if environment variables have changed, applies the changed environment variables, and Check if the collection module has been changed, and if the collection module has been changed, apply the changed collection module, Checking if the data schema has changed, and applying the changed collection module if the data schema has changed, Open API-based data collection program automatic generation and dynamic reflection of modifications system.
In paragraph 1, A data loading server that parses collected data transmitted from the above-mentioned collection program server, generates a DB query, and loads it into the DB; and An open API-based system for automatically generating data collection programs and dynamically reflecting modifications, further comprising a DB server that stores collected data loaded from the above data loading server in a DB.
In paragraph 1, The above collection program server is, An open API-based system for automatic generation of a data collection program and dynamic reflection of modifications, comprising a dynamic modification application unit that modifies environment variables, data schemas, and dynamic modules including a collection module and a data transformation module when a dynamic modification is requested from the program generation and management server while the data collection program is in operation.
A step in which the user terminal inputs modifications to environment variables, modifications to the collection module, modifications to the data conversion module, and modifications to the data schema for modifying the collection program through the UI; The program generation and management server transmits to the collection program server modifications, collection module modifications, data conversion module modifications, and data schema modifications input from the user terminal through the management program; and The collection program server includes the step of modifying the data collection program by modifying the changing parts of the information regarding environment variables, collection modules, data conversion modules, and data schemas when a dynamic modification is requested from the program creation and management server. Execution method of API-based data collection program automatic generation and modification dynamic reflection system.
In paragraph 6, The step of modifying the above data collection program is, The method comprises the step of checking whether there is a changed part in at least one of the information regarding environment variables, collection modules, and data schemas during the operation of the collection program through the collection program of the collection program server, and if there is a changed part in the at least one, applying the changed part to at least one of the environment variables, collection modules, and data schemas. Execution method of an Open API-based data collection program automatic generation and modification dynamic reflection system.
In Paragraph 7, The step of applying the above-mentioned changes is, when the scheduler of the collection program for data collection starts, A step to check if environment variables have changed, and if they have changed, to apply the changed environment variables; A step of checking whether the collection module has been changed, and if the collection module has been changed, applying the changed collection module; and A step including checking whether the data schema has changed and, if the data schema has changed, applying the changed collection module. Execution method of an Open API-based data collection program automatic generation and modification dynamic reflection system.
In paragraph 6, A step in which a data loading server parses collected data transmitted from the collection program server, generates a DB query, and loads it into the DB; and A method for executing an open API-based data collection program automatic generation and dynamic modification reflection system, further comprising the step of a DB server storing collected data loaded from the data loading server into a DB.

Description

System for Automatically Generating a Data Collection Program Based on Open API and Dynamically Reflecting Modifications and Method for Executing the Same The present invention relates to an open API (Application Programming Interface)-based system for automatically generating a data collection program and dynamically reflecting modifications, and a method for executing the same. Data is becoming increasingly important day by day, to the point where people say, "Data is the food of the future." Consequently, collecting good data is becoming increasingly important, but obtaining high-quality data is not easy unless you are a specific company or institution. Fortunately, however, government agencies such as the Public Data Portal and several companies are providing high-quality data through Open APIs. As the use of these open APIs for data collection increases, the writing of data collection programs is becoming more common. These collection programs are largely identical, differing mainly in configuration aspects such as the open API URL, collected data, and schema, as well as in the program code responsible for calling the open API and storing the received data. FIG. 1 shows software components of a data collection system according to one embodiment of the present invention. Figure 2 shows an example in which software components of a data collection system according to one embodiment of the present invention are loaded into hardware. FIG. 3 shows another example in which software components of a data collection system according to one embodiment of the present invention are loaded into hardware. FIG. 4 shows the process of generating a data collection program according to one embodiment of the present invention. FIG. 5 shows a data collection program component according to one embodiment of the present invention. FIG. 6 shows the operation process of a data collection program according to one embodiment of the present invention. FIG. 7 shows the modification process of a data collection program according to one embodiment of the present invention. FIG. 8 shows the internal process of a data collection program when modifying a data collection program according to an embodiment of the present invention. FIG. 9 shows the process of loading data from an Open API to a DB during the operation of a data collection program according to an embodiment of the present invention. FIG. 10 shows a flowchart of a data collection system according to one embodiment of the present invention. FIG. 11 shows a collection module according to one embodiment of the present invention. FIG. 12 shows a data conversion module according to one embodiment of the present invention. FIG. 13 shows a flowchart of a data loading unit according to one embodiment of the present invention. The aforementioned objectives, means, and effects are described in detail below with reference to the attached drawings, thereby enabling a person skilled in the art to easily implement the technical concept of the present invention. In describing the present invention, detailed descriptions of known technologies related to the present invention are omitted if it is determined that such descriptions would unnecessarily obscure the essence of the invention. Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the attached drawings. In the drawings, the same reference numerals are used to indicate the same or similar components. In the following, a system capable of automatically generating a public data collection program provided through an open API and dynamically reflecting modifications to the program, and a method for executing the same are disclosed. FIG. 1 shows software components of a data collection system according to one embodiment of the present invention. FIG. 2 shows an example in which software components of a data collection system according to one embodiment of the present invention are loaded into hardware. FIG. 3 shows another example in which software components of a data collection system according to one embodiment of the present invention are loaded into hardware. Referring to Fig. 1, the data collection system may include, in terms of software, a generation and management program, a UI, a collection program, a data loading unit, and a DB. Referring to FIG. 2, the data collection system may include, in terms of hardware, a program creation and management server, a user terminal, a collector server (or collection program server), a data loading server, and a DB server. A program creation and management program may be installed on the program creation and management server, a UI may be installed on the user terminal, at least one data collection program may be installed on the collector server, a data loading unit may be installed on the data loading server, and a DB may be installed on the DB server. Referring to FIG. 3, as another example, the data collection system may inclu