CN-115238889-B - Training method, device, equipment and storage medium of self-supervision learning model

Abstract

The application discloses a training method, device and equipment of a self-supervision learning model and a storage medium, belonging to the technical field of computers and Internet. The method comprises the steps of obtaining a sample set, carrying out splicing processing on a target text sample in the sample set and other text samples except the target text sample in the sample set to generate a first negative sample corresponding to the target text sample, and carrying out self-supervision training on a text feature extraction model by adopting the first negative sample corresponding to the target text sample, wherein the text feature extraction model is used for obtaining feature information of an input text based on the input text so as to match a search text similar to the input text in terms of semantics. According to the text feature extraction method and device, the distinguishing capability of the text feature extraction model for text information with smaller semantic difference is improved, and the searching capability of the text feature extraction model is further improved.

Inventors

• GAO LIZHAO

Assignees

• 腾讯科技（深圳）有限公司

Dates

Publication Date

20260505

Application Date

20220728

Claims (9)

1. 1. A method of training a self-supervised learning model, the method comprising: Obtaining a sample set, wherein the sample set comprises at least two text samples; For a target text sample in the sample set, performing splicing processing on the target text sample and other text samples except the target text sample in the sample set to generate a first negative sample corresponding to the target text sample; The method comprises the steps of generating a first mask model, a second mask model and a third mask model based on a text feature extraction model, inputting a target text sample into the first mask model to obtain sample feature information, inputting a positive sample corresponding to the target text sample into the second mask model to obtain positive sample feature information, inputting a first negative sample corresponding to the target text sample into the third mask model to obtain first negative sample feature information, determining a first semantic distance corresponding to the target text sample according to the sample feature information and the positive sample feature information, determining a second semantic distance corresponding to the target text sample according to the sample feature information and the first negative sample feature information, performing self-supervision training on the text feature extraction model based on the first semantic distance and the second semantic distance corresponding to each text sample, wherein the text feature extraction model is used for obtaining feature information of the input text based on the input text so as to match text similar to the input text, and different corresponding different semantic parameters, and generating a keyword-based text sample for the target text sample or the positive text sample.
2. 2. The method according to claim 1, wherein the performing a stitching process on the target text sample and other text samples in the sample set except for the target text sample to generate a first negative sample corresponding to the target text sample includes: Determining an interference text sample corresponding to the target text sample from other text samples except the target text sample in the sample set; And splicing and inserting the interference text sample into the target text sample to generate a first negative sample corresponding to the target text sample.
3. 3. The method according to claim 2, wherein the determining, from the other text samples in the sample set except for the target text sample, the disturbing text sample corresponding to the target text sample includes: respectively acquiring semantic distances between the other text samples and the target text samples; and determining other text samples with the semantic distance smaller than a first threshold value as interference text samples corresponding to the target text samples.
4. 4. The method according to claim 2, wherein the splicing and inserting the disturbing text sample in the target text sample generates a first negative sample corresponding to the target text sample, including: dividing the target text sample to obtain at least one text segment; Respectively acquiring semantic distances between each text segment and the interference text sample; and according to the position of the text segment with the semantic distance smaller than the second threshold value in the target text sample, splicing and inserting the interference text sample in the target text sample to generate a first negative sample corresponding to the target text sample.
5. 5. The method according to claim 2, wherein the splicing and inserting the disturbing text sample in the target text sample generates a first negative sample corresponding to the target text sample, including: Acquiring key text information of the interference text sample from the interference text sample; and splicing and inserting the key text information in the target text sample, and generating a first negative sample corresponding to the target text sample.
6. 6. The method of any of claims 1 to 5, wherein generating the first mask model, the second mask model, and the third mask model based on the text feature extraction model comprises: Randomly masking model parameters of each layer of network in the text feature extraction model to obtain the first mask model, the second mask model and the third mask model; Wherein model parameters masked by different mask models are different.
7. 7. A training device for a self-supervised learning model, the device comprising: a set acquisition module for acquiring a sample set, the sample set includes at least two text samples; The sample generation module is used for performing splicing processing on the target text sample in the sample set and other text samples except the target text sample in the sample set to generate a first negative sample corresponding to the target text sample; The system comprises a model training module, a text feature extraction module and a text feature extraction module, wherein the model training module is used for generating a first mask model, a second mask model and a third mask model based on a text feature extraction model, inputting the target text sample into the first mask model to obtain sample feature information, inputting a positive sample corresponding to the target text sample into the second mask model to obtain positive sample feature information, inputting a first negative sample corresponding to the target text sample into the third mask model to obtain first negative sample feature information, determining a first semantic distance corresponding to the target text sample according to the sample feature information and the positive sample feature information, determining a second semantic distance corresponding to the target text sample according to the sample feature information and the first negative sample feature information, and performing self-supervision training on the text feature extraction model based on the first semantic distance and the second semantic distance corresponding to each text sample respectively, wherein the text feature extraction model is used for obtaining the feature information of the input text based on the input text to match search text similar to the input text, different mask parameters are different mask models, and the positive text sample is generated as the target text sample or the target text sample.
8. 8. A computer device comprising a processor and a memory, wherein the memory has stored therein at least one program that is loaded and executed by the processor to implement the method of training the self-supervised learning model as recited in any of claims 1 to 6.
9. 9. A computer-readable storage medium, wherein at least one program is stored in the storage medium, the at least one program being loaded and executed by a processor to implement the method of training the self-supervised learning model as recited in any one of claims 1 to 6.

Description

Training method, device, equipment and storage medium of self-supervision learning model Technical Field The present application relates to the field of the internet and computers, and in particular, to a training method, apparatus, device and storage medium for a self-supervised learning model. Background At present, when searching, the corresponding search information can be obtained through searching the target information input by the user through the search model. In the related art, in the process of training a search model, a plurality of training samples are obtained from an open source database and are divided into a plurality of training sample sets, so that no similar training samples exist in one training sample set, further, one training sample set is used as a batch, the training samples in the batch are input into the search model, the training samples are used as positive samples, and different training samples in the batch are negative samples, so that the search model is trained. However, in the related art, different training samples in the batch are negative samples, and no similar training sample exists in one batch, that is, no similar information exists in the negative samples of the training samples, after the search model is trained, information with larger phase difference can be distinguished, but information with smaller phase difference cannot be distinguished, for example, the search model cannot well distinguish two information, namely "a user cannot log in a social platform" and "a user logs in a social platform publishing view", so that the search effect of the search model is poor. Disclosure of Invention The embodiment of the application provides a training method, device, equipment and storage medium of a self-supervision learning model, which improve the distinguishing capability of a text feature extraction model for text information with smaller semantic difference, and further improve the searching capability of the text feature extraction model. The technical scheme is as follows. According to an aspect of an embodiment of the present application, there is provided a training method of a self-supervised learning model, the method including the steps of: Obtaining a sample set, wherein the sample set comprises at least two text samples; For a target text sample in the sample set, performing splicing processing on the target text sample and other text samples except the target text sample in the sample set to generate a first negative sample corresponding to the target text sample; and performing self-supervision training on a text feature extraction model by adopting a first negative sample corresponding to the target text sample, wherein the text feature extraction model is used for obtaining feature information of an input text based on the input text so as to match a search text with similar semantics to the input text. According to an aspect of an embodiment of the present application, there is provided a training apparatus of a self-supervised learning model, the apparatus including: a set acquisition module for acquiring a sample set, the sample set includes at least two text samples; The sample generation module is used for performing splicing processing on the target text sample in the sample set and other text samples except the target text sample in the sample set to generate a first negative sample corresponding to the target text sample; The text feature extraction module is used for obtaining feature information of the input text based on the input text so as to match a search text with similar semantics to the input text. According to an aspect of the embodiment of the present application, the embodiment of the present application provides a computer device, where the computer device includes a processor and a memory, where at least one section of program is stored in the memory, and the at least one section of program is loaded and executed by the processor to implement the training method of the self-supervised learning model. According to an aspect of the embodiment of the present application, there is provided a computer readable storage medium having stored therein at least one program loaded and executed by a processor to implement the training method of the self-supervised learning model described above. According to an aspect of embodiments of the present application, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the training method of the self-supervised learning model. The technical scheme provided by the embodiment of the application can bring the following beneficial effects: The text samples and other text samples except the text samples in the s