CN-117252256-B - Knowledge graph, text graph coding model and graph-based pre-training method

CN117252256BCN 117252256 BCN117252256 BCN 117252256BCN-117252256-B

Abstract

The invention aims to provide a dialectical knowledge graph and an automatic construction method thereof, a model for simultaneously coding texts and graphs and a pre-training method thereof, and a method for utilizing graph information to secondarily pre-train the model. The method comprises the steps of organizing the discussion points and the discussion sentences into semantic graphs and inter-sentence logic graphs in the sentences, automatically constructing the intra-sentence semantic graphs and inter-sentence logic graphs from the original discussion corpus, combining a pre-training language model and a graphic neural network to encode text and graph information, realizing interaction of the two types of information, pre-training the corresponding model on the discussion corpus through various self-supervision tasks, expanding pre-training expected content through the graph information, adding corresponding pre-training tasks, and applying the pre-training task to secondary pre-training of the model or other pre-training language models. The invention can effectively integrate the information of various dialect corpus, provide clear and accurate dialect semantic and logic information for the language model, and is beneficial to the model to complete the task of dialect scene requiring complex reasoning and argumentation.

Inventors

WEI ZHONGYU
Liang Jingcong

Assignees

复旦大学

Dates

Publication Date: 20260508
Application Date: 20230912

Claims (8)

1. An automated method for constructing a discourse knowledge graph, wherein the discourse knowledge graph comprises a lower-layer intra-sentence semantic graph and an upper-layer inter-sentence logic graph, wherein the intra-sentence semantic graph comprises a plurality of sub-graphs which are respectively in one-to-one correspondence with each discourse point or discourse data, each sub-graph has a top node representation, and the inter-sentence logic graph consists of top nodes which respectively represent each discourse point or discourse data; The semantic graph in the sentence is composed of semantic unit nodes and semantic relation edges, semantic information is represented by attribute values by the semantic unit nodes and the semantic relation edges, and the directed acyclic is satisfied, wherein any one of the arguments or sub-graphs corresponding to the arguments forms a communicated directed acyclic graph, and the directed acyclic graph comprises a top node which can represent the arguments or the arguments, so that no other edges in the sub-graph corresponding to the arguments or the arguments point to the top node; The inter-sentence logic atlas comprises a plurality of top nodes, at least two top nodes form a combined node through combined edges, the two top nodes, the top node and the combined node are correspondingly connected through logic relation edges, the attribute values of the logic relation edges record logic information among nodes forming the logic relation edges, and node attributes are left blank; Extracting arguments, arguments and relations thereof from original arguments corpus through an argumentation mining model, then analyzing each argumentation and arguments sentence into AMR (automatic measure) atlas by utilizing an abstract semantic representation analysis model, finally merging all AMR atlas into semantic atlas in the sentence according to a mode of merging isomorphic nodes, and completing logical atlas among sentences according to the arguments and relations of arguments obtained by the argumentation mining algorithm.
2. The method of claim 1, wherein the step of merging all AMR atlases into the intra-sentence semantic atlas in a manner that merges isomorphic nodes comprises: the following steps are carried out iteratively until the codes of all nodes are different from each other, wherein all nodes are firstly coded according to node attributes, edge-out attributes and pointing nodes thereof, then each group of nodes with the same codes are replaced by a single new node, the edge-out of the nodes is the same as any node in the group, and the edge-in is the union of the edge-in of all nodes in the group.
3. The coding method for simultaneously coding the text and the corresponding arguments knowledge graph is characterized in that the arguments knowledge graph comprises a lower-layer intra-sentence semantic graph and an upper-layer inter-sentence logic graph, wherein the intra-sentence semantic graph comprises a plurality of sub-graphs which are respectively in one-to-one correspondence with each arguments or arguments, each sub-graph has a top node representation, and the inter-sentence logic graph consists of top nodes which respectively represent each arguments or arguments; The semantic graph in the sentence is composed of semantic unit nodes and semantic relation edges, semantic information is represented by attribute values by the semantic unit nodes and the semantic relation edges, and the directed acyclic is satisfied, wherein any one of the arguments or sub-graphs corresponding to the arguments forms a communicated directed acyclic graph, and the directed acyclic graph comprises a top node which can represent the arguments or the arguments, so that no other edges in the sub-graph corresponding to the arguments or the arguments point to the top node; The inter-sentence logic atlas comprises a plurality of top nodes, at least two top nodes form a combined node through combined edges, the two top nodes, the top node and the combined node are correspondingly connected through logic relation edges, the attribute values of the logic relation edges record logic information among nodes forming the logic relation edges, and node attributes are left blank; Acquiring a argumentation text and a sub-graph corresponding to the argumentation text in the argumentation knowledge graph, dividing the text into words and performing preliminary coding, performing preliminary coding on node attributes and edge attributes of the argumentation knowledge graph, performing multi-layer interactive joint processing on the preliminarily coded text coding and graph node coding, and finally outputting final text coding representation and graph node coding representation by using a cross-modal attention layer, wherein the sub-graph corresponding to the argumentation knowledge graph is at least one of intra-sentence semantic graph and/or inter-sentence logic graph.
4. A coding method according to claim 3, characterized in that: The text word segmentation and preliminary coding comprises the steps of segmenting a text by a word segmentation device and coding by an embedding layer and a plurality of previous layers of a pre-training language model; the step of carrying out preliminary coding on node attributes and side attributes of the arguments knowledge graph comprises the steps of expanding an embedded layer of the pre-training language model and calculating embedded coding of the node attributes and the side attributes of the graph by utilizing the embedded layer; The method comprises the steps of processing text codes and atlas codes through multi-layer interaction joint processing, wherein the text codes are processed by adopting a plurality of later layers of the pre-training language model, processing atlas node codes by introducing multi-layer atlas neural networks with the same layer number, and processing interaction parts in the text and atlas node codes by using a full-connection layer before each layer is output to the next layer; The step of outputting the final text coding representation and the atlas node coding representation by using the cross-modal attention layer comprises the steps of inputting the text coding output by the pre-training language model and the atlas node coding output by the multi-layer atlas neural network into a unified cross-modal attention layer, and outputting the final text coding representation and the atlas node coding representation.
5. The encoding method of claim 3, wherein the step of processing the nodes of the atlas by using the multi-layer neural network includes transforming unidirectional edges in the atlas into bidirectional edges, wherein the attribute of the reverse edges is the attribute of the original edges, adding transition nodes corresponding to each sentence in the atlas when processing a plurality of sentences containing the context, and adding a root node to connect all the transition nodes, and simultaneously, only preserving one sub-atlas for a plurality of sentences with the same sub atlas.
6. A method of pre-training a language model formed by the encoding method of claim 3, wherein the pre-training is performed using a dialect corpus using a plurality of self-supervising tasks, including mask class tasks and graph topology class tasks.
7. The method as in claim 6 wherein the masking class task comprises: Text mask prediction, namely randomly masking or replacing words in a text, and predicting the masked or replaced words through the model; Node mask prediction, namely applying the same text mask prediction method to the map node attribute; edge mask prediction, namely applying the same text mask prediction method to the map edge attribute; The graph topology class tasks include: Graph contrast learning, namely randomly disturbing the node attributes of the graph, and predicting whether the node attributes of the graph are disturbed or not through the model; Predicting the relative sequence of the text corresponding to the added transition node; And predicting the edge direction, namely predicting the original edge direction after transforming the unidirectional edge in the map into a bidirectional edge.
8. The method for utilizing the graph information secondary pre-training model of the discussion knowledge graph is characterized in that the discussion knowledge graph comprises a lower-layer intra-sentence semantic graph and an upper-layer inter-sentence logic graph, wherein the intra-sentence semantic graph comprises a plurality of sub-graphs which are respectively in one-to-one correspondence with each discussion point or discussion data, each sub-graph is provided with a top node representation, and the inter-sentence logic graph consists of top nodes which respectively represent each discussion point or discussion data; The semantic graph in the sentence is composed of semantic unit nodes and semantic relation edges, semantic information is represented by attribute values by the semantic unit nodes and the semantic relation edges, and the directed acyclic is satisfied, wherein any one of the arguments or sub-graphs corresponding to the arguments forms a communicated directed acyclic graph, and the directed acyclic graph comprises a top node which can represent the arguments or the arguments, so that no other edges in the sub-graph corresponding to the arguments or the arguments point to the top node; The inter-sentence logic atlas comprises a plurality of top nodes, at least two top nodes form a combined node through combined edges, the two top nodes, the top node and the combined node are correspondingly connected through logic relation edges, the attribute values of the logic relation edges record logic information among nodes forming the logic relation edges, and node attributes are left blank; Sampling a plurality of associated dialect sentences from each document participating in pre-training, placing the documents after the documents, and using an updated document secondary pre-training language model or a language model formed after coding, wherein when the plurality of associated dialect sentences are sampled from the atlas, each sentence in the documents is searched for the sentences associated with each sentence in the atlas according to the connection tightness degree of sub-images, a batch of associated dialect sentences are obtained by sampling according to the tightness degree as weight, and the task of judging the angle relation between the associated dialect sentences and the corresponding documents is added on the basis of the original pre-training task of the model when the updated document secondary pre-training language model or the language model formed after coding is used.

Description

Knowledge graph, text graph coding model and graph-based pre-training method Technical Field The invention relates to the field of computers, in particular to a dialect knowledge graph, a text graph coding model and a pre-training method based on the graph. Background Autonomous arguments are one of the important capabilities of strong artificial intelligence, and require the artificial intelligence to comment on a topic according to world knowledge and its reasoning conclusions, and respond to comments or replies of others on the topic or comments thereof, and relate to tasks such as understanding, reasoning, generating, etc. of arguments. The autonomous arguments may provide multi-angle analysis in decision making processes (including industrial decisions, public policies, business decisions, etc.), such as providing countermeasures or ideas using autonomous arguments techniques to interactively refine decision content when making unilateral decisions, or using multiple autonomous arguments to perform the above-described interactive processes to help the decision maker view different angles to make comprehensive optimal selections. For example, it may be applied to the advertising industry, particularly in developing related products such as document generation systems, simulated delivery systems, and the like. In addition, the autonomous debate can be used for simulating community discussion of focus subjects, helps explore possible development of community public opinion when different angle views (related utterances are generated by a controlled autonomous debate model) are introduced, and thus helps to develop various customs works and evaluation works, and facilitates evaluation and customs of subsequent scenes including but not limited to industrial construction. In addition, the autonomous dialect can also use manual dialogue to produce an artificial intelligent dialogue robot, and can assist the human operation in an industrial scene. Current research methods focus on modeling knowledge and inference methods with the aid of large-scale parameters of pre-trained language models, but models obtained by such methods do not exhibit sufficiently good inference capabilities in complex inference scenarios, such as arguments. Therefore, it is necessary to organize the knowledge, particularly the views with subjective components and their relationships, by explicit methods as auxiliary information sources for the current language model. Disclosure of Invention Aiming at the defects, the invention provides a dialectical knowledge graph and an automatic construction method thereof, a model for simultaneously coding texts and graphs and a pre-training method thereof, and a method for utilizing graph information to pre-train the model secondarily. In order to achieve the above object, the present invention proposes a knowledge graph for organizing arguments and arguments, which consists of a lower-layer intra-sentence semantic graph and an upper-layer inter-sentence logical graph, wherein each argument or arguments corresponds to a sub-graph of the intra-sentence semantic graph and is represented by one of the top nodes, and the top nodes representing each argument or arguments constitute the inter-sentence logical graph. The sentence internal semantic graph is composed of semantic unit nodes and semantic relation edges, the directed acyclic performance is met, any one of the arguments or sub-graphs corresponding to the arguments form a communicated directed acyclic graph, the directed acyclic graph comprises a top node capable of representing the arguments or the arguments, no other edges in the sub-graph are pointed to the top node, the sentence-to-sentence logic graph is composed of the top node, a combination node, the combination edge and a logic relation edge, the combination node is connected with corresponding top nodes of a plurality of parallel arguments or arguments related in a logic relation through the combination edge, and the logic relation edge is connected with the corresponding top node or the combination node according to different logic relations among the arguments or arguments. The invention also provides an automatic method for constructing the discussion knowledge graph, which is characterized in that the discussion points, the discussion data and the relation thereof are extracted from the original discussion corpus through a discussion mining model, each discussion point and the discussion data sentence are analyzed into AMR graphs by utilizing an abstract semantic representation (AMR) analysis model, all AMR graphs are finally combined into the intra-sentence semantic graph according to a mode of combining isomorphic nodes, and the inter-sentence logic graph is completed according to the relation between the discussion points and the discussion data obtained by the discussion mining algorithm. The method comprises the steps of firstly encoding all nodes according to node attributes,