CN-121997992-A - Context compression method, device, electronic equipment, storage medium and program product

CN121997992ACN 121997992 ACN121997992 ACN 121997992ACN-121997992-A

Abstract

The disclosure relates to the technical field of computers, and discloses a context compression method, a device, electronic equipment, a storage medium and a program product. The method comprises the steps of obtaining first context information and second context information corresponding to the first context information, respectively compressing the first context information and the second context information by using a compression model to obtain a first compression feature corresponding to the first context information and a second compression feature corresponding to the second context information, determining difference information between the first context information and the second context information based on the first compression feature and the second compression feature, and optimizing the compression model based on the difference information to enable the compression model to conduct context compression. By implementing the technical scheme disclosed by the invention, the compression model is ensured to learn the lost detail information, and the accurate compression of the context information is realized.

Inventors

LIU CHENGYUAN
Zhao Fubang
LIU QINGSONG
KANG YANGYANG
RAN JIAO
KUANG KUN

Assignees

北京字跳网络技术有限公司
浙江大学

Dates

Publication Date: 20260508
Application Date: 20260104

Claims (13)

1. A method of context compression, the method comprising: acquiring first context information and second context information corresponding to the first context information; Respectively compressing the first context information and the second context information by using a compression model to obtain a first compression characteristic corresponding to the first context information and a second compression characteristic corresponding to the second context information; determining difference information between the first context information and the second context information based on the first compression feature and the second compression feature; optimizing the compression model based on the difference information to enable the compression model to conduct context compression.
2. The method of claim 1, wherein obtaining second context information corresponding to the first context information comprises: Compressing the first context information by using the compression model to obtain text features corresponding to the first context information; And recovering the text characteristics by using a decompression model corresponding to the compression model to obtain the second context information.
3. The method of claim 1 or 2, wherein the determining difference information between the first context information and the second context information based on the first compression feature and the second compression feature comprises: Obtaining content difference information between the first context information and the second context information based on text content characterized by the first context information and the second context information; Obtaining feature difference information between the first compression feature and the second compression feature based on the text feature characterized by the first compression feature and the second compression feature; wherein the difference information includes the content difference information and the feature difference information.
4. The method of claim 1, wherein the obtaining second context information corresponding to the first context information comprises: compressing the first context information by using the compression model to obtain a space feature token set corresponding to the first context; Obtaining tokens to be modified in the spatial feature token set; Based on the position of the token to be modified in the spatial feature token set, the token to be modified is adjusted to be a target token; And generating the second context information based on the fusion result of the target token and the non-modified token.
5. The method of claim 4, wherein the obtaining the tokens to be modified in the set of spatial feature tokens comprises: selecting a plurality of first tokens with the smallest prediction generation probability from the spatial feature token set based on a prediction strategy corresponding to the compression model; And determining the first token as the token to be modified.
6. The method of claim 4, wherein the adjusting the token to be modified to a target token based on the location of the token to be modified in the set of spatial feature tokens comprises: based on the position relation between the token to be modified and the non-modified token, sampling the position of the token to be modified to obtain the target token; and replacing the token to be modified with the target token.
7. The method of any of claims 4-6, wherein the determining difference information between the first context information and the second context information based on the first compression feature and the second compression feature comprises: Performing differential processing on the first compression characteristic and the second compression characteristic to obtain a first characteristic difference; performing differential processing on the second compression characteristic and the first compression characteristic to obtain a second characteristic difference; the difference information includes the first feature difference and the second feature difference.
8. The method of claim 7, wherein the optimizing the compression model based on the difference information comprises: Obtaining modeling loss information corresponding to the compression model; Acquiring first difference loss information based on the first characteristic difference information; Acquiring second difference loss information based on the second characteristic difference information; Acquiring target loss information based on the modeling loss information, the first differential loss information, and the second differential loss information; Optimizing the compression model using the target loss information.
9. The method of claim 8, wherein the obtaining target loss information based on the modeling loss information, the first differential loss information, and the second differential loss information comprises: Determining target difference loss information based on a fusion result of the first difference loss information and the second difference loss information; and superposing the target difference loss information and the modeling loss information to obtain the target loss information.
10. A context compression apparatus, the apparatus comprising: the device comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring first context information and second context information corresponding to the first context information; The first compression module is used for respectively compressing the first context information and the second context information by utilizing a compression model to obtain a first compression characteristic corresponding to the first context information and a second compression characteristic corresponding to the second context information; a difference information determining module configured to determine difference information between the first context information and the second context information based on the first compression feature and the second compression feature; And the second compression module is used for optimizing the compression model based on the difference information so as to enable the compression model to conduct context compression.
11. An electronic device, comprising: A memory and a processor in communication with each other, the memory having stored therein computer instructions which, upon execution, perform the context compression method of any of claims 1 to 9.
12. A computer-readable storage medium having stored thereon computer instructions for causing a computer to perform the context compression method of any of claims 1 to 9.
13. A computer program product comprising computer instructions for causing a computer to perform the context compression method of any one of claims 1 to 9.

Description

Context compression method, device, electronic equipment, storage medium and program product Technical Field The present disclosure relates to the field of computer technology, and in particular, to a method, an apparatus, an electronic device, a storage medium, and a program product for compressing a context. Background In a multi-modal scenario, compressing a long context into a few tokens (token) can reduce the sequence length in the question-answering (QA) process, saving the coding overhead of the model and the reasoning overhead of the model. However, the current compression scheme adopts a specific training mode, and only changes simply on task instruction diversity and loss functions, so that larger detail loss exists, and the compression accuracy of the context information is affected. Disclosure of Invention The disclosure provides a context compression method, a device, an electronic device, a storage medium and a program product, so as to solve the problem of poor context compression accuracy. In a first aspect, the disclosure provides a context compression method, which includes obtaining first context information and second context information corresponding to the first context information, compressing the first context information and the second context information by using a compression model to obtain a first compression feature corresponding to the first context information and a second compression feature corresponding to the second context information, determining difference information between the first context information and the second context information based on the first compression feature and the second compression feature, and optimizing the compression model based on the difference information to enable the compression model to perform context compression. In a second aspect, the disclosure provides a context compression device, which comprises an acquisition module, a first compression module, a difference information determination module and a second compression module, wherein the acquisition module is used for acquiring first context information and second context information corresponding to the first context information, the first compression module is used for respectively compressing the first context information and the second context information by using a compression model to obtain a first compression feature corresponding to the first context information and a second compression feature corresponding to the second context information, the difference information determination module is used for determining difference information between the first context information and the second context information based on the first compression feature and the second compression feature, and the second compression module is used for optimizing the compression model based on the difference information so that the compression model performs context compression. In a third aspect, the present disclosure provides an electronic device, including a memory and a processor, where the memory and the processor are communicatively connected to each other, and the memory stores computer instructions, and the processor executes the computer instructions, thereby performing the context compression method of the first aspect or any implementation manner corresponding to the first aspect. In a fourth aspect, the present disclosure provides a computer readable storage medium having stored thereon computer instructions for causing a computer to perform the context compression method of the first aspect or any of its corresponding embodiments. In a fifth aspect, the present disclosure provides a computer program product comprising computer instructions for causing a computer to perform the context compression method of the first aspect or any of its corresponding embodiments. The context compression method, the device, the electronic equipment, the storage medium and the program product provided by the disclosure determine difference information between the first context information and the second context information by acquiring the first context information and second context information which is different from the first context information and combining a first compression characteristic corresponding to the first context information and a second compression characteristic corresponding to the second context information, wherein the difference information can represent the detail information lost by the second context information compared with the first context information. And optimizing the compression model by utilizing the difference information so that the compression model can learn the lost detail information to fit the lost detail information in the subsequent compression process, and ensuring the accurate compression of the context information. Drawings In order to more clearly illustrate the embodiments of the present disclosure or the prior art, the drawings that are required in the d