CN-121979611-A - Automatic operation method and system of non-interface industrial software

CN121979611ACN 121979611 ACN121979611 ACN 121979611ACN-121979611-A

Abstract

The invention provides an automatic operation method and system of interface-free industrial software, which comprise the steps of obtaining a screenshot of each interface of the industrial software, storing the screenshot in folders named with corresponding types according to the types of each interface, training a stored screenshot sample based on a neural network model, identifying the types and the functions of the interfaces, identifying the current operation interface according to the neural network model, loading an interface template drawing set corresponding to the current operation interface, selecting a matching result with highest confidence as an interface template by adopting a template matching algorithm, determining the position of an operation target of the current operation interface according to the interface template, automatically operating the current operation interface according to the position of the operation target, identifying the types and the functions of pages by the neural network model, identifying the dynamic interfaces of the software in real time, positioning the operation target according to the corresponding interface template, and rapidly and accurately realizing the automatic operation of the interface-free industrial software.

Inventors

LIAO JIAYUAN
SHAO DONGDONG
ZHANG HAIFANG
ZHAO TIELIANG
DING KUNPENG

Assignees

深圳中科四合科技有限公司

Dates

Publication Date: 20260505
Application Date: 20260409

Claims (10)

1. A method of automatic operation of interfacing-free industrial software, comprising: acquiring screenshot of each interface of the industrial software, and storing the screenshot to folders named by corresponding types according to the types of each interface; Training the stored screenshot samples based on a neural network model to obtain a deep learning model of each interface of the industrial software, and identifying the interface type and the interface function; Identifying the interface type and the interface function of the current operation interface according to the deep learning model; Loading an interface template chart set corresponding to the current operation interface according to the interface type and the interface operation function, and selecting a matching result with highest screenshot confidence coefficient of the current operation interface from the interface template chart set by adopting a template matching algorithm as an interface template of the current operation interface; and determining the position of an operation target of the current operation interface according to the interface template, and automatically operating the current operation interface according to the position of the operation target.
2. The method of claim 1, wherein the non-interface industrial software comprises a plurality of types, wherein the capturing the screenshot of each interface of the industrial software and storing the screenshot to the folder named by the corresponding type according to the type of each interface comprises, Obtaining screenshot of each interface of various types of industrial software, and respectively storing the screenshot to folders named by corresponding interface types under corresponding software type directories according to the software types of each interface; Correspondingly, the neural network model is based on training the stored screenshot samples to obtain a deep learning model of each interface of the industrial software, the identification of the interface type and the interface function comprises, Training the stored screenshot samples based on the neural network model to obtain a deep learning model of each interface of the industrial software of multiple types, and identifying the software, the interface type and the interface function of the interface.
3. The method of claim 1, wherein the interface types include a home page, a parameter settings page, and other pages.
4. The method of claim 1, wherein after identifying the interface type and interface function of the current operation interface according to the deep learning model, further comprising: reversely generating a corresponding digital type according to the interface type, and reading a type mapping relation stored in the deep learning model; If the corresponding relation between the interface type and the digital type of the current operation interface is inconsistent with the type mapping relation stored in the deep learning model, triggering an alarm and starting the type mapping relation stored in the deep learning model to correct the interface type of the current operation interface; and recording and outputting the corresponding relation between the interface type and the digital type of the current operation interface.
5. The method of claim 1, wherein the obtaining the screenshot of each interface of the industrial software and storing the screenshot in a folder named by the corresponding type according to the type of each interface comprises: And acquiring the screenshot of each interface of the industrial software, normalizing the screenshot of each interface to a preset standard size, converting the screenshot into an image tensor in the input format of the neural network model, and storing the image tensor in a folder named by the corresponding interface type according to the type of each interface.
6. The method according to claim 1, wherein after determining the position of the operation target of the current operation interface according to the interface template and automatically operating the current operation interface according to the position of the operation target, further comprises: And naming the current operation interface according to the interface type and the timestamp and storing the current operation interface into the folder named with the corresponding type.
7. The method according to claim 1, wherein after determining the position of the operation target of the current operation interface according to the interface template and automatically operating the current operation interface according to the position of the operation target, further comprises: waiting for a preset time period to ensure that the interface response is completed; Identifying an interface screenshot after response completion according to the deep learning model, and if the interface screenshot is consistent with an expected response interface template, verifying that the response passes, and automatically pushing the interface to the next operation step; if the interface screenshot is not matched with the expected response interface template or the matching result does not reach the threshold value, triggering a retry mechanism; if the retry fails, the exception handling step is automatically executed and recorded to an exception log.
8. An automated operating system for interfacing industrial software, comprising: The interface image acquisition unit is used for acquiring screenshot of each interface of the industrial software and storing the screenshot into a folder named by the corresponding type according to the type of each interface; the model training unit is used for training the stored screenshot samples based on the neural network model to obtain a deep learning model of each interface of the industrial software, and identifying the interface type and the interface function; The interface identification unit is used for identifying the interface type and the interface function of the current operation interface according to the deep learning model; the interface template selection unit is used for loading an interface template drawing set corresponding to the current operation interface according to the interface type and the interface operation function, and selecting a matching result with highest screenshot confidence coefficient of the current operation interface from the interface template drawing set by adopting a template matching algorithm as an interface template of the current operation interface; and the operation target determining unit is used for determining the position of the operation target of the current operation interface according to the interface template and automatically operating the current operation interface according to the position of the operation target.
9. A computer device comprising a processor and a memory, the memory having stored therein a computer program which, when loaded and executed by the processor, implements the method of any of claims 1 to 7.
10. A computer storage medium, characterized in that the computer storage medium has stored therein a computer program which, when executed, implements the method according to any of claims 1 to 7.

Description

Automatic operation method and system of non-interface industrial software Technical Field The invention relates to the technical field of industrial automation operation, in particular to an automatic operation method and system of non-interface industrial software. Background At present, the paths of the industrial automation operation technology are mainly divided into two types, one is a fixed coordinate simulation interaction technology, operation is realized by presetting absolute coordinates of a mouse and a keyboard, but the scheme completely depends on the fixed position and the display resolution of a software window, and once the position of the window is changed or terminal equipment is replaced, coordinate positioning is immediately invalid, so that the adaptability of multiple terminals is extremely poor. And secondly, an interaction technology relying on an open interface realizes automatic interaction in modes of API call, control attribute reading and the like, but a large amount of special control software on an industrial site is used for guaranteeing system stability and data safety, the open interface is not externally opened, and bottom control information is not exposed, so that the technology cannot be applied in the software scene at all, and a technical blind area for automatic control of the industrial software without the interface is formed. Disclosure of Invention The invention provides an automatic operation method, a system, computer equipment and a storage medium of non-interface industrial software, which are used for identifying page types and page functions through a neural network model, identifying a dynamic interface of the software in real time, positioning an operation target according to a corresponding interface template, and realizing the automatic operation of the non-interface industrial software rapidly and accurately. In a first aspect, the present invention provides a method for automatically operating industrial software without an interface, comprising: acquiring screenshot of each interface of the industrial software, and storing the screenshot to folders named by corresponding types according to the types of each interface; Training the stored screenshot samples based on a neural network model to obtain a deep learning model of each interface of the industrial software, and identifying the interface type and the interface function; Identifying the interface type and the interface function of the current operation interface according to the deep learning model; Loading an interface template chart set corresponding to the current operation interface according to the interface type and the interface operation function, and selecting a matching result with highest screenshot confidence coefficient of the current operation interface from the interface template chart set by adopting a template matching algorithm as an interface template of the current operation interface; and determining the position of an operation target of the current operation interface according to the interface template, and automatically operating the current operation interface according to the position of the operation target. In one embodiment, the non-interface industrial software comprises a plurality of types, the screenshot of each interface of the industrial software is obtained and stored to a folder named by the corresponding type according to the type of each interface, Obtaining screenshot of each interface of various types of industrial software, and respectively storing the screenshot to folders named by corresponding interface types under corresponding software type directories according to the software types of each interface; Correspondingly, the neural network model is based on training the stored screenshot samples to obtain a deep learning model of each interface of the industrial software, the identification of the interface type and the interface function comprises, Training the stored screenshot samples based on the neural network model to obtain a deep learning model of each interface of the industrial software of multiple types, and identifying the software, the interface type and the interface function of the interface. In one embodiment, the interface types include a home page, a parameter settings page, and other pages. In one embodiment, after the identifying the interface type and the interface function of the current operation interface according to the deep learning model, the method further includes: reversely generating a corresponding digital type according to the interface type, and reading a type mapping relation stored in the deep learning model; If the corresponding relation between the interface type and the digital type of the current operation interface is inconsistent with the type mapping relation stored in the deep learning model, triggering an alarm and starting the type mapping relation stored in the deep learning model to correct the interf