CN-121996825-A - Data processing method, data processing apparatus, and computer-readable storage medium

CN121996825ACN 121996825 ACN121996825 ACN 121996825ACN-121996825-A

Abstract

The present disclosure relates to the field of big data technology, and in particular, to a data processing method, a data processing device, and a computer readable storage medium. The method comprises the steps of obtaining a call request for calling a target application programming interface, obtaining association information associated with the target application programming interface when the target application programming interface is an interface with unstructured meta information, obtaining data blood edges of the actual file sets, all actual files contained in the actual file sets in the data blood edges and the actual file meta information corresponding to the actual files when the file set state of the actual file sets corresponding to the file set information is unexpired, obtaining result file meta information of the data blood edges after the actual files participate in processing operation when the actual files exist in processing operation, screening the target meta information based on filtering conditions to obtain screening meta information, and generating call information corresponding to the call request based on the screening meta information.

Inventors

SHI XIUTAO
SUN CHUNXIAO
LIU KAI
GAO YUE
ZHANG LE

Assignees

聚好看科技股份有限公司

Dates

Publication Date: 20260508
Application Date: 20251210

Claims (10)

1. A data processing apparatus, comprising: A communicator configured to communicate with an external application; A controller configured to: Acquiring a call request for calling a target application programming interface; when the target application programming interface is an interface configured with unstructured meta information, acquiring associated information associated with the target application programming interface, wherein the associated information at least comprises file set information and filtering conditions; When the file set state of the actual file set corresponding to the file set information is unexpired, acquiring the data blood margin of the actual file set, each actual file contained in the actual file set in the data blood margin and the actual file meta-information corresponding to each actual file; When the actual file has processing operation, obtaining result file meta-information of the data blood margin after the actual file participates in the processing operation; Screening target meta-information based on the filtering condition to obtain screening meta-information, wherein the target meta-information comprises the result file meta-information and actual file meta-information corresponding to an actual file which does not participate in processing operation; And generating calling information corresponding to the calling request based on the filtering meta information.
2. The data processing device of claim 1, wherein the controller, prior to executing the get call request to call the target application programming interface, is further configured to: Acquiring configuration information for configuring a target application programming interface, a designated file set and screening conditions, wherein the configuration information at least comprises one or more of a route, a protocol and a security hierarchy, and the screening conditions at least comprise file meta information and file sources of at least one file accessible in the designated file set; The target application programming interface for external invocation is generated based on the configuration information, the specified set of file sets, and the filter criteria.
3. The data processing device of claim 1, wherein the controller, prior to executing the get call request to call the target application programming interface, is further configured to: Responding to construction operation of creating a target file set, and acquiring file set information of the target file set; And generating the target file sets based on the file set information, wherein one target file set corresponds to a storage position in one data lake.
4. A data processing apparatus according to claim 3, wherein the controller is further configured to: Collecting target files, and storing the target files into the data lake, wherein one target file corresponds to one target file set; based on one or more target files in the target file set, a data blood-edge of the target file set is generated.
5. A data processing apparatus according to claim 3, wherein the controller is further configured to: Responding to the processing operation of the target file, obtaining at least one result file, and the result file meta-information of each result file, and storing the result file meta-information into the data lake; and generating the data blood edges of the target file set based on the target file and the result file.
6. The data processing apparatus according to claim 1, wherein the controller, when performing filtering of target meta-information based on the filtering condition, is further configured to: screening target meta-information based on one or more specified file meta-information contained in the filtering condition, and determining the target meta-information with the specified file meta-information having a data blood margin; And obtaining screening meta-information based on the target meta-information with the data blood-source of the specified file meta-information.
7. A method of data processing, comprising: Acquiring a call request for calling a target application programming interface; when the target application programming interface is an interface configured with unstructured meta information, acquiring associated information associated with the target application programming interface, wherein the associated information at least comprises file set information and filtering conditions; When the file set state of the actual file set corresponding to the file set information is unexpired, acquiring the data blood margin of the actual file set, each actual file contained in the actual file set in the data blood margin and the actual file meta-information corresponding to each actual file; When the actual file has processing operation, obtaining result file meta-information of the data blood margin after the actual file participates in the processing operation; Screening target meta-information based on the filtering condition to obtain screening meta-information, wherein the target meta-information comprises the result file meta-information and actual file meta-information corresponding to an actual file which does not participate in processing operation; And generating calling information corresponding to the calling request based on the filtering meta information.
8. The data processing method of claim 1, wherein the method further comprises: Acquiring configuration information for configuring a target application programming interface, a designated file set and screening conditions, wherein the configuration information at least comprises one or more of a route, a protocol and a security hierarchy, and the screening conditions at least comprise file meta information and file sources of at least one file accessible in the designated file set; The target application programming interface for external invocation is generated based on the configuration information, the specified set of file sets, and the filter criteria.
9. The data processing method of claim 1, wherein the method further comprises: Responding to construction operation of creating a target file set, and acquiring file set information of the target file set; And generating the target file sets based on the file set information, wherein one target file set corresponds to a storage position in one data lake.
10. A computer readable storage medium having stored therein computer executable instructions which when executed by a processor are adapted to carry out the data processing method according to any of claims 7-9.

Description

Data processing method, data processing apparatus, and computer-readable storage medium Technical Field The present disclosure relates to the field of big data technology, and in particular, to a data processing method, a data processing device, and a computer readable storage medium. Background Currently, big data platforms face a serious technical bottleneck in processing file information retrieval of unstructured data (e.g., documents, images). Unlike structured data, which possess a mature "table structure driven" dynamic application programming interface (Application Programming Interface, API) generation scheme, the metadata model of unstructured data is missing and heterogeneous, resulting in its retrieval that relies heavily on a "custom development" schema. Specifically, the platform can only develop a specific interface for each specific metadata query requirement (e.g., screening by tag, creation time combination) by pre-specifying file directories. The mode has long development period and extremely poor flexibility, and can not quickly respond to the dynamic changing retrieval demands of service users, such as temporarily adding filtering conditions or cross-knowledge-base combined query, thereby becoming a key obstacle for preventing the efficient release of the value of unstructured knowledge assets. Therefore, how to improve the efficiency of unstructured data retrieval is a urgent problem to be solved. Disclosure of Invention In order to solve the technical problems described above, the present disclosure provides a data processing method, a data processing apparatus, and a computer-readable storage medium. The data processing device comprises a communicator, a controller and a filtering unit, wherein the communicator is configured to communicate with an external application, the controller is configured to acquire a calling request for calling a target application programming interface, acquire association information associated with the target application programming interface when the target application programming interface is an interface configured with unstructured meta information, the association information at least comprises file set information and filtering conditions, acquire data blood edges of the actual file sets, all actual files contained in the actual file sets in the data blood edges and actual file meta information corresponding to all the actual files when the file set state of the actual file sets corresponding to the file set information is unexpired, acquire result file meta information of the actual file with the data blood edges after the actual file participates in processing operation when the actual file exists in processing operation, screen the target meta information based on the filtering conditions to acquire screening meta information, wherein the target meta information comprises the result file meta information and the actual file meta information corresponding to the actual file not participating in processing operation, and generate calling information corresponding to the calling request based on the screening meta information. The second aspect provides a data processing method, which comprises the steps of obtaining a call request for calling a target application programming interface, obtaining association information associated with the target application programming interface when the target application programming interface is an interface with unstructured meta information, wherein the association information at least comprises file set information and filtering conditions, obtaining data blood edges of the actual file sets, all actual files contained in the actual file sets in the data blood edges and the actual file meta information corresponding to all the actual files when the file set state of the actual file sets corresponding to the file set information is unexpired, obtaining result file meta information of the data blood edges after the actual files participate in processing operation when the actual files exist in processing operation, screening the target meta information based on the filtering conditions to obtain screening meta information, wherein the target meta information comprises the result file meta information and the actual file meta information corresponding to the actual files which do not participate in processing operation, and generating call information corresponding to the call request based on the screening meta information. In a third aspect, the present disclosure provides a computer readable storage medium having a computer program stored thereon, the computer program being executable by a controller to perform the data processing method of any one of the second aspects. In a fourth aspect, the present disclosure provides a computer program product for, when run on a computer, causing the computer to perform the data processing method as any one of the second aspects provides. It should be noted that the above-