US-12627813-B2 - Residual coding method and device, video coding method and device, and storage medium

US12627813B2US 12627813 B2US12627813 B2US 12627813B2US-12627813-B2

Abstract

A residual coding method is provided. The method comprises: when the current frame is an inter-frame prediction frame, determining whether a residual of a background region in the current frame needs to be compensated for; and when the residual of the background region in the current frame needs to be compensated for, coding first residual data, and when the residual of the background region in the current frame does not need to be compensated for, coding second residual data, wherein the first residual data is residual data of an entire image of the current frame, and the second residual data is residual data of a target region in the current frame. The present disclosure further provides a video coding method using the residual coding method, and corresponding devices and a storage medium.

Inventors

Zhan Ma
Qi Xia
Haojie Liu

Assignees

GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP., LTD.

Dates

Publication Date: 20260512
Application Date: 20231016

Claims (20)

1 . A residual encoding method, comprising: in response to that a current picture is an inter predicted picture, determining whether a residual of a background area in the current picture needs to be compensated; encoding first residual data in response to that the residual of the background area in the current picture needs to be compensated, and encoding second residual data in response to that the residual of the background area in the current picture does not need to be compensated; wherein the first residual data is residual data of a whole picture of the current picture, and the second residual data is residual data of an object area in the current picture, and wherein determining whether the residual of the background area in the current picture needs to be compensated comprises: determining whether the residual of the background area in the current picture needs to be compensated according to at least one of an impact factor calculated for the current picture or an impact factor calculated for an encoded inter predicted picture in a same group of pictures; wherein the impact factors are determined based on a difference between a reconstructed picture with compensating the residual of the background area and a reconstructed picture without compensating the residual of the background area.
2 . The residual encoding method of claim 1 , wherein determining whether the residual of the background area in the current picture needs to be compensated comprises: adding a predicted picture of the current picture and the second residual data to obtain a reference picture of the current picture; calculating a similarity between an original picture and the reference picture of the current picture, and determining whether the similarity is less than a set first threshold value: in response to that the similarity is less than the set first threshold value, determining that the residual of the background area in the current picture needs to be compensated; in response to that the similarity is greater than or equal to the set first threshold value, determining that the residual of the background area in the current picture does not need to be compensated.
3 . The residual encoding method of claim 1 , wherein the method further comprises: in response to that the current picture is an earliest inter predicted picture after intra-coded (I) frame in a group of pictures, adding a predicted picture of the current picture and the second residual data to obtain a reference picture of the current picture; and performing a similarity calculation to obtain a similarity between an original picture and the reference picture of the current picture, and determining, according to the similarity, a difference Ns between picture serial numbers of two pictures for compensating the residual of the background area; and determining whether the residual of the background area in the current picture needs to be compensated comprises: determining that the residual of the background area in the current picture needs to be compensated in response to that a difference N between picture serial numbers of the current picture and a most recent inter predicted picture for compensating the residual of the background area or determining that the residual of the background area in the current picture needs to be compensated in response to that a difference N between picture serial numbers of the current picture and a most recent I frame is equal to Ns, where both N and Ns are positive integers.
4 . The residual encoding method of claim 3 , wherein: determining whether the residual of the background area in the current picture needs to be compensated further comprises: when N<Ns, determining that the residual of the background area in the current picture does not need to be compensated directly; or when N<Ns, determining whether a set residual compensation condition is satisfied, in response to that the set residual compensation condition is satisfied, determining that the residual of the background area in the current picture needs to be compensated, and in response to that the set residual compensation condition is not satisfied, determining that the residual of the background area in the current picture does not need to be compensated; when N=Ns, re-performing a similarity calculation, and re-determining a difference Ns between picture serial numbers of two pictures for compensating the residual of the background region according to a calculated similarity.
5 . The residual encoding method of claim 4 , wherein when residual encoding is performed on each inter predicted picture, similarity calculation is performed to obtain a similarity; and the set residual compensation condition comprises at least one of the following: a difference value obtained by subtracting a calculated similarity for the current picture from a calculated similarity for a previous picture is greater than a set second threshold value; the calculated similarity for the current picture is less than a set third threshold value.
6 . The residual encoding method of claim 3 , wherein determining the difference Ns between picture serial numbers of two pictures for compensating the residual of the background area according to the similarity comprises: determining a similarity range to which the similarity belongs; and searching corresponding relationships between preset similarity ranges and numbers of interval pictures according to the similarity range determined, and taking a searched number of interval pictures as Ns, wherein in the correspondences, a number of interval pictures which corresponds to a similarity range with a greater similarity in the preset similarity ranges is also larger; or obtaining Ns according to the similarity and a set equation, the set equation being used for calculating the number of interval pictures according to the similarity.
7 . The residual encoding method of claim 1 , wherein the method further comprises: setting residual data of other areas except the object area in the current picture to be equal to 0, and encoding the residual data of other areas except the object area in the current picture.
8 . The residual encoding method of claim 1 , wherein the residual data of the object area in the current picture is obtained through multiplying the first residual data by a first object mask, the first object mask being obtained by inputting an original picture of the current picture into an object segmentation network and performing binarization processing on a segmentation result; or the residual data of the object area in the current picture is obtained through multiplying the first residual data by a second object mask, the second object mask being obtained by performing dilation processing on the first object mask.
9 . The residual encoding method of claim 8 , wherein performing the dilation processing on the first object mask comprises: determining a dilation kernel used for the dilation processing, and performing the dilation processing on the first object mask by using the dilation kernel, wherein a size of the dilation kernel is positively correlated with a statistical shift value for pixels in the object area.
10 . The residual encoding method of claim 9 , wherein the dilation kernel is a square, and a side length k d of the square is calculated according to the following equation: k d = c ⁢ e ⁢ i ⁢ l ⁡ ( max ⁡ ( D * M O ) ) + k 0 wherein D is a matrix composed of shift values of pixels in the current picture, M O is the first object mask before the dilation processing, k 0 is a set constant, ceil( ) is a rounding up function, and max( ) is a function for taking a maximum value of elements in the matrix.
11 . A non-transitory computer-readable medium, having stored thereon instructions and a bitstream, wherein the instructions, when executed by a processor, cause the processor to perform the method of claim 1 to generate the bitstream.
12 . A video encoding method, wherein in response to that a current picture is an inter predicted picture, a predicted picture of the current picture is obtained through inter prediction calculation; first residual data is calculated and obtained according to an original picture and the predicted picture of the current picture; and residual encoding is performed according to the following method: in response to that a current picture is an inter predicted picture, determining whether a residual of a background area in the current picture needs to be compensated; encoding first residual data in response to that the residual of the background area in the current picture needs to be compensated, and encoding second residual data in response to that the residual of the background area in the current picture does not need to be compensated; wherein the first residual data is residual data of a whole picture of the current picture, and the second residual data is residual data of an object area in the current picture, and wherein determining whether the residual of the background area in the current picture needs to be compensated comprises: determining whether the residual of the background area in the current picture needs to be compensated according to at least one of an impact factor calculated for the current picture or an impact factor calculated for an encoded inter predicted picture in a same group of pictures; wherein the impact factors are determined based on a difference between a reconstructed picture with compensating the residual of the background area and a reconstructed picture without compensating the residual of the background area.
13 . The video encoding method of claim 12 , wherein the method further comprises: in response to that the current picture is an intra-coded (I) frame, encoding the original picture of the current picture by adopting a first code rate and a second code rate respectively to obtain an picture feature map of the first code rate and an picture feature map of the second code rate, wherein the first code rate is greater than the second code rate; multiplying the picture feature map of the first code rate by a third object mask to obtain an object feature map; multiplying the picture feature map of the second code rate by a background mask to obtain a background feature map; and encoding the object feature map and the background feature map respectively; wherein the third object mask and the background mask are obtained by inputting the original picture of the current picture into an object segmentation network and performing binarization processing on a segmentation result.
14 . A video encoding device, comprising a processor and a memory storing a computer program executable by the processor, wherein the processor is configured to cause the video encoding device to implement, when running the computer program, the video encoding method of claim 12 .
15 . A residual encoding device, comprising a processor and a memory storing a computer program executable by the processor, wherein the processor is configured to cause the residual encoding device to implement, when running the computer program, operations of: in response to that a current picture is an inter predicted picture, determining whether a residual of a background area in the current picture needs to be compensated; encoding first residual data in response to that the residual of the background area in the current picture needs to be compensated, and encoding second residual data in response to that the residual of the background area in the current picture does not need to be compensated; wherein the first residual data is residual data of a whole picture of the current picture, and the second residual data is residual data of an object area in the current picture, and wherein determining whether the residual of the background area in the current picture needs to be compensated comprises: determining whether the residual of the background area in the current picture needs to be compensated according to at least one of an impact factor calculated for the current picture or an impact factor calculated for an encoded inter predicted picture in a same group of pictures; wherein the impact factors are determined based on a difference between a reconstructed picture with compensating the residual of the background area and a reconstructed picture without compensating the residual of the background area.
16 . The residual encoding device of claim 15 , wherein determining whether the residual of the background area in the current picture needs to be compensated comprises: adding a predicted picture of the current picture and the second residual data to obtain a reference picture of the current picture; calculating a similarity between an original picture and the reference picture of the current picture, and determining whether the similarity is less than a set first threshold value: in response to that the similarity is less than the set first threshold value, determining that the residual of the background area in the current picture needs to be compensated; in response to that the similarity is greater than or equal to the set first threshold value, determining that the residual of the background area in the current picture does not need to be compensated.
17 . The residual encoding device of claim 15 , wherein the processor is further configured to cause the residual encoding device to implement operations of: in response to that the current picture is an earliest inter predicted picture after intra-coded (I) frame in a group of pictures, adding a predicted picture of the current picture and the second residual data to obtain a reference picture of the current picture; and performing a similarity calculation to obtain a similarity between an original picture and the reference picture of the current picture, and determining, according to the similarity, a difference Ns between picture serial numbers of two pictures for compensating the residual of the background area; and determining whether the residual of the background area in the current picture needs to be compensated comprises: determining that the residual of the background area in the current picture needs to be compensated in response to that a difference N between picture serial numbers of the current picture and a most recent inter predicted picture for compensating the residual of the background area or determining that the residual of the background area in the current picture needs to be compensated in response to that a difference N between picture serial numbers of the current picture and a most recent I frame is equal to Ns, where both N and Ns are positive integers.
18 . The residual encoding device of claim 15 , wherein the residual data of the object area in the current picture is obtained through multiplying the first residual data by a first object mask, the first object mask being obtained by inputting an original picture of the current picture into an object segmentation network and performing binarization processing on a segmentation result; or the residual data of the object area in the current picture is obtained through multiplying the first residual data by a second object mask, the second object mask being obtained by performing dilation processing on the first object mask.
19 . The residual encoding device of claim 18 , wherein performing the dilation processing on the first object mask comprises: determining a dilation kernel used for the dilation processing, and performing the dilation processing on the first object mask by using the dilation kernel, wherein a size of the dilation kernel is positively correlated with a statistical shift value for pixels in the object area.
20 . The residual encoding device of claim 19 , wherein the dilation kernel is a square, and a side length k d of the square is calculated according to the following equation: k d = c ⁢ e ⁢ i ⁢ l ⁡ ( max ⁡ ( D * M O ) ) + k 0 wherein D is a matrix composed of shift values of pixels in the current picture, M O is the first object mask before the dilation processing, k 0 is a set constant, ceil( ) is a rounding up function, and max( ) is a function for taking a maximum value of elements in the matrix.

Description

CROSS-REFERENCE TO RELATED APPLICATION This is a continuation application of International Patent Application No. PCT/CN2021/087916, filed on Apr. 16, 2021, entitled “RESIDUAL CODING METHOD AND DEVICE, VIDEO CODING METHOD AND DEVICE, AND STORAGE MEDIUM”, the disclosure of which is hereby incorporated by reference in its entirety. BACKGROUND Digital video compression technology is mainly to compress huge digital imaging video data, so as to facilitate transmission and storage. With the proliferation of videos on the Internet and the increasing demand for video definition, although existing digital video compression standards can save a lot of video data, it is still necessary to pursue better digital video compression technology to reduce a bandwidth and traffic pressure for transmitting a digital video. SUMMARY Embodiments of the present disclosure relate to but is not limited to video encoding and decoding technology, in particular to a coefficient encoding and decoding method, an encoding and decoding device and a storage medium. The following is a summary of subjects described in detail herein. The summary is not intended to limit the scope of protection of claims. There is provided a residual encoding method in an embodiment of the present disclosure, which includes the following operations. In response to that a current picture is an inter predicted picture, it is determined whether a residual of a background area in the current picture needs to be compensated; first residual data is encoded in response to that the residual of the background area in the current picture needs to be compensated, and second residual data is encoded in response to that the residual of the background area in the current picture does not need to be compensated. Herein the first residual data is residual data of a whole picture of the current picture, and the second residual data is residual data of an object area in the current picture. There is also provided a video encoding method in an embodiment of the present disclosure. In response to that a current picture is an inter predicted picture, a predicted picture of the current picture is obtained through inter prediction calculation; first residual data is calculated and obtained according to an original picture and the predicted picture of the current picture; andresidual encoding is performed according to the method described in any embodiment of the present disclosure. There is also provided a residual encoding device in an embodiment of the present disclosure, including a processor and a memory storing a computer program executable by the processor. The processor is configured to implement, when running the computer program, the residual encoding method as described in any embodiment of the present disclosure. There is also provided a video encoding device in an embodiment of the present disclosure, including a processor and a memory storing a computer program executable by the processor. The processor is configured to implement, when running the computer program, the video encoding method as described in any embodiment of the present disclosure. After reading and understanding drawings and detailed description, other aspects can be understood. BRIEF DESCRIPTION OF THE DRAWINGS The drawings are used to provide an understanding of the embodiments of the present disclosure, and constitute a part of the description. The drawings, together with the embodiments of the present disclosure, serve to explain a technical solution of the present disclosure and do not constitute a limitation to the technical solution of the present disclosure. FIG. 1 is a schematic diagram of a framework of video compression encoding. FIG. 2 is a flowchart of a residual encoding method according to an embodiment of the present disclosure. FIG. 3A is a schematic diagram of a residual encoding method according to an embodiment of the present disclosure. FIG. 3B is another schematic diagram of a residual encoding method according to an embodiment of the present disclosure. FIG. 4 is a schematic diagram of a dilation kernel used for dilation processing. FIG. 5 is a flowchart of a video encoding method according to an embodiment of the present disclosure. FIG. 6 is a schematic diagram of a residual encoding device according to an embodiment of the present disclosure. FIG. 7 is a schematic diagram of a video encoding and decoding method according to an embodiment of the present disclosure. FIG. 8A is a schematic diagram of an object mask before dilation processing according to an embodiment of the present disclosure. FIG. 8B is a schematic diagram of an object mask after performing dilation processing on the object mask of FIG. 8A. FIG. 8C is a picture obtained after processing using the object mask of FIG. 8A. FIG. 8D is a picture obtained after processing using the object mask of FIG. 8B. FIG. 9A is a flowchart of encoding I frame by a video encoding method according to an embodiment of the present dis