CN-116311326-B - Hand-drawn flow chart identification method and device, storage medium and electronic equipment

CN116311326BCN 116311326 BCN116311326 BCN 116311326BCN-116311326-B

Abstract

The application discloses a method, a device, a storage medium and equipment for identifying a hand-drawn flow chart, which are used for detecting key points of each element node and line segment in the hand-drawn flow chart through a target detection network, extracting the visual characteristics of each element node and the key point characteristics of the key points of the line segment, wherein the accuracy of line segment representation is improved by utilizing the key point characteristics, an initial chart network is constructed according to the visual characteristics of each element node and the key point characteristics of the key points of the line segment, the initial characteristics of vertexes and edges in the chart network are determined, and the corresponding chart network characteristic extraction processing and classification processing are carried out on the chart network so as to obtain the connection relation between each element node in the hand-drawn flow chart.

Inventors

YAN QIANDONG
LIU CHENYU
WU JIAJIA
HU JINSHUI
YIN BING

Assignees

科大讯飞股份有限公司

Dates

Publication Date: 20260512
Application Date: 20230309

Claims (10)

1. A method for identifying a hand-drawn flow chart, comprising: Acquiring a hand-drawn flow chart; Detecting each element node and line segment key points in the hand-drawn flow chart by using a target detection network, and extracting visual characteristics of each element node and key point characteristics of the line segment key points; According to the visual characteristics of each element node and the key point characteristics of the key points of the line segment, taking each key point and each element node in the key points of the line segment as the vertexes of an initial graph network, taking the connecting line between any two vertexes as the edges of the initial graph network to construct an initial graph network, and determining the initial characteristics of the vertexes and the initial characteristics of the edges of the initial graph network; According to the initial characteristics of the vertexes and the initial characteristics of the edges, carrying out graph network characteristic extraction and classification processing on the initial graph network to identify the connection relationship between the vertexes and the edges, and mapping the connection relationship into the connection relationship between each element node in the hand-drawing flow chart; determining the recognition result of the hand-drawn flow chart based on the connection relation; The step of extracting and classifying the initial graph network according to the initial characteristics of the vertexes and the initial characteristics of the edges to identify the connection relationship between the vertexes and the edges comprises the following steps: according to the initial characteristics of the vertexes and the initial characteristics of the edges, carrying out graph network characteristic extraction processing and de-densification processing on the initial graph network by utilizing a graph convolution network based on a self-attention module so as to obtain a sparse graph network; Carrying out graph network feature extraction processing on the features of the vertexes and the edges in the sparse graph network by utilizing the graph convolution network based on the self-attention module so as to obtain enhanced features; and classifying the connection relation between the vertexes and the edges in the sparse graph network according to the enhancement features to obtain the connection relation between the vertexes and the edges.
2. The method of claim 1, wherein the line segment keypoints are keypoints in a line segment or an arrowed line segment in the hand-drawn flowchart, and wherein the step of determining initial features of vertices and initial features of edges of the initial graph network comprises: taking the key point characteristics of each key point in the line segment key points and the visual characteristics of each element node as initial characteristics of corresponding vertexes of the initial graph network; And determining the initial characteristics of the edges between any two vertexes according to the initial characteristics of the any two vertexes.
3. The method of claim 2, wherein the step of determining the initial characteristics of the edge between any two vertices from the initial characteristics of the any two vertices comprises: Performing nonlinear mapping processing on the initial characteristics of each vertex of the initial graph network to obtain the mapping characteristics of each vertex; and carrying out fusion processing on the mapping characteristics of any two vertexes to obtain the initial characteristics of the edge between any two vertexes.
4. The method of claim 1, wherein the step of detecting individual element nodes and line segment keypoints in the hand-drawn flowchart using a target detection network comprises: Extracting a plurality of candidate areas and area visual feature graphs corresponding to the candidate areas from the hand-drawn flow chart by utilizing a feature extraction module of the target detection network; Classifying the region visual feature map by using a classification regression module of the target detection network to determine that a plurality of candidate regions correspond to each basic element in the hand-drawn flow chart, wherein the basic elements comprise element nodes and line segments or line segments with arrows; When the basic element is an element node, identifying the position information of the element node in the hand-drawn flow chart by utilizing the classification regression module; And when the basic element is a line segment or an arrow-headed line segment, identifying a line segment key point of the line segment or the arrow-headed line segment and the position information of the line segment key point in the hand-drawn flow chart by using the classification regression module.
5. The method of claim 4, wherein the feature extraction module comprises a feature extraction network and a domain of interest pooling network, and wherein the step of extracting visual features of individual element nodes and key point features of line segment key points comprises: Taking the regional visual feature map of the candidate region corresponding to each element node as the visual feature of each element node; and extracting key point characteristics of the line segment or the line segment key points with arrows from a feature map according to the position information of the line segment key points by using an interest domain pooling network of the target detection network.
6. The method of claim 4, wherein the step of identifying the line segment keypoints of the line segment or the arrowed line segment and the location information of the line segment keypoints in the hand-drawn flowchart using the classification regression module comprises: Predicting a preset number of key points in the line segment or the line segment with the arrow and position coordinates of the key points in a candidate area corresponding to the line segment or the line segment with the arrow by using a prediction network in the classification regression module; identifying the coordinate offset of the candidate region corresponding to the line segment or the line segment with the arrow by utilizing the classification regression module; And determining the position information of the key points in the hand-drawn flow chart according to the coordinate offset, the original coordinates of the candidate areas corresponding to the line segments or the line segments with arrows and the position coordinates, taking the preset number of key points as line segment key points, and taking the position information of the preset number of key points in the hand-drawn flow chart as the position information of the line segment key points in the hand-drawn flow chart.
7. The method as recited in claim 1, further comprising: and carrying out structural processing on each element node and the connection relation between each element node to obtain a structural identification result corresponding to the hand-drawn flow chart.
8. A hand-drawn flowchart identifying device, comprising: the acquisition module is used for acquiring the hand-painted flow chart; The detection module is used for detecting each element node and line segment key points in the hand-drawn flow chart by utilizing a target detection network, and extracting visual characteristics of each element node and key point characteristics of the line segment key points; The image construction module is used for constructing an initial image network by taking each key point in the key points of the line segments and each element node as the vertex of the initial image network and taking the connecting line between any two vertexes as the edge of the initial image network according to the visual characteristics of each element node and the key point characteristics of the key points of the line segments, and determining the initial characteristics of the vertex and the initial characteristics of the edge of the initial image network; the graph feature classification module is used for extracting and classifying graph network features of the initial graph network according to the initial features of the vertexes and the initial features of the edges so as to identify the connection relationship between the vertexes and the edges, and mapping the connection relationship into the connection relationship among the element nodes in the hand-drawing flow chart; The graph characteristic classification module is further used for carrying out graph network characteristic extraction processing and de-densification processing on the initial graph network by utilizing a graph rolling network based on the self-attention module according to the initial characteristics of the vertexes and the initial characteristics of the edges so as to obtain a sparse graph network; carrying out graph network feature extraction processing on the features of the vertexes and the edges in the sparse graph network by utilizing the graph convolution network based on the self-attention module to obtain enhanced features; and the determining module is used for determining the recognition result of the hand-drawn flow chart based on the connection relation.
9. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program adapted to be loaded by a processor for performing the steps in the method according to any of claims 1-7.
10. An electronic device comprising a memory in which a computer program is stored and a processor that performs the steps in the method of any of claims 1-7 by invoking the computer program stored in the memory.

Description

Hand-drawn flow chart identification method and device, storage medium and electronic equipment Technical Field The application relates to the technical field of artificial intelligence, in particular to a hand-drawn flow chart identification method, a hand-drawn flow chart identification device, a computer readable storage medium and electronic equipment. Background With the advent of various intelligent devices, particularly tablet computers, electronic whiteboards, smart phones and the like, they are widely put into use, so that information recording becomes more convenient. Handwriting (hand drawing) is one of the most natural and efficient ways for humans to record information. Existing handwriting data can be broadly divided into two categories, text and graphics. While text remains the primary carrier of information transfer in life, with the advancement of society, the way in which information is transferred graphically is becoming more and more important, particularly in office and educational settings, the way in which some of the more important information is presented by way of a flow chart is also becoming extremely convenient and important. However, the flow chart constructed by handwriting cannot be displayed and edited in a standardized manner. Meanwhile, the writing freedom degree is large, and the display in a handwriting way is quite unattractive. Therefore, how to structure the handwriting flow chart can be more attractive to show to the audience, and the handwriting flow chart can be edited later, so that the handwriting flow chart has a certain practical value and commercial prospect. The currently mainstream flowchart identification method still mainly aims at solving the problem of printed body flowchart identification. However, compared with the printed flow chart, the handwriting flow chart has the characteristics of changeable form, disordered typesetting and the like. For example, different writers have different writing habits, even with the same flow chart, which can vary widely from writer to writer. In addition, even for the same writer, writing the same flowchart at different times may cause differences. These problems all present significant challenges to handwriting flow diagram recognition. Disclosure of Invention The embodiment of the application provides a method, a device, a computer readable storage medium and electronic equipment for identifying a hand-drawn flow chart, which are used for identifying the hand-drawn flow chart in a mode of cascading a target detection network and a graph network, so that the accuracy of identifying the hand-drawn flow chart is improved. The embodiment of the application provides a hand-drawn flow chart identification method, which comprises the following steps: Acquiring a hand-drawn flow chart; Detecting each element node and line segment key points in the hand-drawn flow chart by using a target detection network, and extracting visual characteristics of each element node and key point characteristics of the line segment key points; constructing an initial graph network according to the visual characteristics of each element node and the key point characteristics of the key points of the line segments, and determining the initial characteristics of the vertexes and the initial characteristics of the edges of the initial graph network; According to the initial characteristics of the vertexes and the initial characteristics of the edges, carrying out graph network characteristic extraction and classification processing on the initial graph network to identify the connection relationship between the vertexes and the edges, and mapping the connection relationship into the connection relationship between each element node in the hand-drawing flow chart; And determining the recognition result of the hand-drawn flow chart based on the connection relation. The embodiment of the application also provides a device for identifying the hand-drawn flow chart, which comprises the following steps: the acquisition module is used for acquiring the hand-painted flow chart; The detection module is used for detecting each element node and line segment key points in the hand-drawn flow chart by utilizing a target detection network, and extracting visual characteristics of each element node and key point characteristics of the line segment key points; the diagram construction module is used for constructing an initial diagram network according to the visual characteristics of each element node and the key point characteristics of the key points of the line segments, and determining the initial characteristics of the vertexes and the initial characteristics of the edges of the initial diagram network; the graph feature classification module is used for extracting and classifying graph network features of the initial graph network according to the initial features of the vertexes and the initial features of the edges so as to identify the connection relatio