US-12627818-B2 - Constraints on locations of reference blocks for intra block copy prediction

Abstract

When encoding/decoding a current block of a current picture using intra block copy (“BC”) prediction, the location of a reference block is constrained so that it can be entirely within an inner search area of the current picture or entirely within an outer search area of the current picture, but cannot overlap both the inner search area and the outer search area. In some hardware-based implementations, on-chip memory buffers sample values of the inner search area, and off-chip memory buffers sample values of the outer search area. By enforcing this constraint on the location of the reference block, an encoder/decoder can avoid memory access operations that are split between on-chip memory and off-chip memory when retrieving the sample values of the reference block. At the same time, a reference block close to the current block may be used for intra BC prediction, helping compression efficiency.

Inventors

• You Zhou
• Chih-Lung Lin
• Ming-Chieh Lee

Assignees

• MICROSOFT TECHNOLOGY LICENSING, LLC

Dates

Publication Date

20260512

Application Date

20240802

Claims (20)

1. 1 . In a computer system that implements a video encoder or image encoder, a method comprising: encoding a current picture to produce encoded data, including, for a given unit of a current coding tree unit (“CTU”) of the current picture: determining a block vector (“BV”) value associated with the given unit, the BV value indicating a displacement from a given block of the given unit to a reference block in the current picture, wherein the determining the BV value includes enforcing a constraint that allows the reference block to be at a first location at which the reference block is entirely within the current CTU and allows the reference block to be at a second location at which the reference block is entirely within a previous CTU of the current picture but prohibits the reference block from being at a third location at which the reference block overlaps both the current CTU and the previous CTU, the previous CTU being left of the current CTU and not overlapping the current CTU, and wherein the determining the BV value associated with the given unit further includes: enforcing a constraint that the BV value associated with the given unit be within a range of permissible BV values; and enforcing a constraint that the reference block includes sample values that have been reconstructed; and performing intra block copy (“BC”) prediction for the given block of the given unit using the BV value associated with the given unit; and outputting the encoded data as part of a bitstream.
2. 2 . The method of claim 1 , wherein the determining the BV value associated with the given unit further includes: enforcing a constraint that the reference block does not violate a rule on dependencies between different sections of the current picture for parallel encoding or decoding.
3. 3 . The method of claim 1 , wherein the determining the BV value associated with the given unit further includes: enforcing a constraint that the given block and the reference block be in the same tile, if applicable, and in the same slice, if applicable.
4. 4 . The method of claim 1 , wherein the given unit is a superblock, a macroblock, or a prediction unit, wherein the given block of the given unit is a primary component block, wherein the given unit also includes secondary component blocks, and wherein the encoding the current picture further includes, for the given unit: deriving a BV value for the secondary component blocks of the given unit based at least in part on the BV value associated with the given unit; and performing intra BC prediction for the secondary component blocks using the derived BV value.
5. 5 . The method of claim 4 , wherein the primary component block includes luma sample values and the secondary component blocks include chroma sample values that have been sub-sampled relative to the luma sample values.
6. 6 . The method of claim 1 , wherein the determining the BV value associated with the given unit uses hash-based block matching and/or block matching according to a search pattern.
7. 7 . The method of claim 1 , wherein the current picture includes multiple CTUs, and wherein each of the multiple CTUs has a maximum CTU size and can be partitioned into smaller units.
8. 8 . The method of claim 1 , wherein the current CTU has a size that is: pre-defined; or indicated by a syntax element in the bitstream.
9. 9 . The method of claim 1 , wherein a first buffer stores reconstructed sample values of the current CTU, wherein a second buffer stores reconstructed sample values of the previous CTU, and wherein the encoding the current picture further includes: after reconstruction of the current CTU, transferring the sample values of the current CTU from the first buffer to the second buffer.
10. 10 . The method of claim 9 , wherein memory of the computer system includes on-chip memory and off-chip memory, wherein the first buffer is part of the on-chip memory, and wherein the second buffer is part of the off-chip memory.
11. 11 . One or more computer-readable media having stored thereon computer-executable instructions for causing a computer system, when programmed thereby, to perform operations comprising: receiving encoded data as part of a bitstream, wherein the encoded data satisfies bitstream conformance requirements; and decoding a current picture using the encoded data, including, for a given unit of a current coding tree unit (“CTU”) of the current picture: reconstructing a block vector (“BV”) value associated with the given unit, the BV value indicating a displacement from a given block of the given unit to a reference block in the current picture, wherein, according to one of the bitstream conformance requirements, the BV value is constrained so that the reference block is allowed to be at a first location at which the reference block is entirely within the current CTU and the reference block is allowed to be at a second location at which the reference block is entirely within a previous CTU of the current picture but the reference block is prohibited from being at a third location at which the reference block overlaps both the current CTU and the previous CTU, the previous CTU being left of the current CTU and not overlapping the current CTU, and wherein, according to the bitstream conformance requirements, the BV value associated with the given unit is further constrained so that: the BV value associated with the given unit is within a range of permissible BV values; and the reference block includes sample values that have been reconstructed; and performing intra block copy (“BC”) prediction for the given block of the given unit using the BV value associated with the given unit.
12. 12 . The one or more computer-readable media of claim 11 , wherein, according to the bitstream conformance requirements, the BV value associated with the given unit is further constrained so that: the reference block does not violate a rule on dependencies between different sections of the current picture for parallel decoding; and/or the given block and the reference block are in the same tile, if applicable, and in the same slice, if applicable.
13. 13 . The one or more computer-readable media of claim 11 , wherein the given unit is a superblock, a macroblock, or a prediction unit, wherein the given block of the given unit is a primary component block, wherein the given unit also includes secondary component blocks, and wherein the decoding the current picture further includes, for the given unit: deriving a BV value for the secondary component blocks of the given unit based at least in part on the BV value associated with the given unit; and performing intra BC prediction for the secondary component blocks using the derived BV value.
14. 14 . The one or more computer-readable media of claim 11 , wherein the current picture includes multiple CTUs, and wherein each of the multiple CTUs has a maximum CTU size and can be partitioned into smaller units.
15. 15 . The one or more computer-readable media of claim 11 , wherein the current CTU has a size that is: pre-defined; or indicated by a syntax element in the bitstream.
16. 16 . The one or more computer-readable media of claim 11 , wherein a first buffer stores reconstructed sample values of the current CTU, wherein a second buffer stores reconstructed sample values of the previous CTU, and wherein the decoding the current picture further includes: after reconstruction of the current CTU, transferring the sample values of the current CTU from the first buffer to the second buffer.
17. 17 . The one or more computer-readable media of claim 11 , wherein the BV value associated with the given unit has integer-sample value resolution, and wherein the given unit has a size of 256×256, 256×128, 128×256, 128×128, 128×64, 64×128, 64×64, 64×32, 32×64, 32×32, 32×16, 16×32, 16×16, 16×8, 8×16, 8×8, 8×4, 4×8, or 4×4 sample values.
18. 18 . One or more computer-readable media having programmed thereon encoded data as part of a bitstream, wherein the encoded data satisfies bitstream conformance requirements, the encoded data being usable to cause a video decoder, when processing the encoded data in a computer system having one or more processing units and memory, to perform operations that include, for a given unit of a current coding tree unit (“CTU”) of the current picture: based on one or more syntax elements in the encoded data, reconstructing a block vector (“BV”) value associated with the given unit, the BV value indicating a displacement from a given block of the given unit to a reference block in the current picture, wherein, according to one of the bitstream conformance requirements, the reference block is allowed to be at a first location at which the reference block is entirely within the current CTU and the reference block is allowed to be at a second location at which the reference block is entirely within a previous CTU of the current picture but the reference block is prohibited from being at a third location at which the reference block overlaps both the current CTU and the previous CTU, the previous CTU being left of the current CTU and not overlapping the current CTU, and wherein, according to the bitstream conformance requirements, the BV value associated with the given unit is further constrained so that: the BV value associated with the given unit is within a range of permissible BV values; and the reference block includes sample values that have been reconstructed; and performing intra block copy (“BC”) prediction for the given block of the given unit using the BV value associated with the given unit.
19. 19 . The one or more computer-readable media of claim 18 , wherein, according to the bitstream conformance requirements, the BV value associated with the given unit is further constrained so that: the reference block does not violate a rule on dependencies between different sections of the current picture for parallel encoding or decoding; and/or the given block and the reference block are in the same tile, if applicable, and in the same slice, if applicable.
20. 20 . The one or more computer-readable media of claim 18 , wherein the given unit is a superblock, a macroblock, or a prediction unit, wherein the given block of the given unit is a primary component block, wherein the given unit also includes secondary component blocks, and wherein the operations further include, for the given unit: deriving a BV value for the secondary component blocks of the given unit based at least in part on the BV value associated with the given unit; and performing intra BC prediction for the secondary component blocks using the derived BV value.

Description

CROSS REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. patent application Ser. No. 18/100,639, filed Jan. 24, 2023, which is a continuation of U.S. patent application Ser. No. 17/198,421, filed Mar. 11, 2021, now U.S. Pat. No. 11,595,667, which is a divisional of U.S. patent application Ser. No. 15/866,291, filed Jan. 9, 2018, now U.S. Pat. No. 10,986,349, the disclosure of which is hereby incorporated by reference. U.S. patent application Ser. No. 15/866,291 claims the benefit of U.S. Provisional Patent Application No. 62/612,293, filed Dec. 29, 2017, the disclosure of which is hereby incorporated by reference. BACKGROUND Engineers use compression (also called source coding or source encoding) to reduce the bit rate of digital video. Compression decreases the cost of storing and transmitting video information by converting the information into a lower bit rate form. Decompression (also called decoding) reconstructs a version of the original information from the compressed form. A “codec” is an encoder/decoder system. Over the last 30 years, various video codec standards have been adopted, including the ITU-T H.261, H.262 (MPEG-2 or ISO/IEC 13818-2), H.263, H.264 (MPEG-4 AVC or ISO/IEC 14496-10), and H.265 (HEVC or ISO/IEC 23008-2) standards, the MPEG-1 (ISO/IEC 11172-2) and MPEG-4 Visual (ISO/IEC 14496-2) standards, and the SMPTE 421M (VC-1) standard. More recently, various extensions to the H.265 standard (e.g., for scalable video coding/decoding, for coding/decoding of video with higher fidelity in terms of sample bit depth or chroma sampling rate, for screen content) have also been adopted. A video codec standard typically defines options for the syntax of an encoded video bitstream, detailing parameters in the bitstream when particular features are used in encoding and decoding. In many cases, a video codec standard also provides details about the decoding operations a decoder should perform to achieve conforming results in decoding. Aside from codec standards, various codec formats such as VP8, VP9, and AV1 (currently under development) define other options for the syntax of an encoded video bitstream and corresponding decoding operations. Intra block copy (“BC”) is a prediction mode in the H.265 standard. An intra BC prediction mode is also under consideration for the AV1 format. For intra BC prediction, the sample values of a current block of a picture are predicted using previously reconstructed sample values of a reference block in the same picture. A block vector (“BV”) value indicates a displacement from the current block to the reference block in the same picture. The BV value can be signaled in the bitstream. In particular, intra BC prediction can be effective for encoding screen content. In intra BC prediction mode for the H.265 standard, a reference block is allowed to be anywhere in previously reconstructed content in the same picture, even right next to the current block. This can make intra BC prediction mode very effective in terms of lowering bit rate and improving quality, since a nearby reference block may closely match the current block. In hardware implementations of intra BC prediction mode for the H.265 standard, however, memory access operations for a reference block can be slow and inefficient. A hardware-based encoder or hardware-based decoder typically stores sample values in fast “on-chip” memory (such as a fast cache) during reconstruction of those sample values, when the sample values are accessed frequently. Because size of the on-chip memory is limited, the hardware-based encoder/decoder eventually transfers a batch of reconstructed sample values to slower “off-chip” memory (such as dynamic random access memory). In the off-chip memory, the reconstructed sample values may still be accessed for intra BC prediction mode or other prediction, but memory access operations are slower. In particular, in hardware implementations of intra BC prediction mode for the H.265 standard, memory access operations can be slow and inefficient when one part of a reference block (close to the current block) is stored in on-chip memory but the rest of the reference block (further from the current block) is separately stored in off-chip memory. In contrast, in some current implementations of intra BC prediction mode for the AV1 format, a reference block is constrained to be outside a “superblock” that includes the current unit being encoded/decoded with intra BC prediction, or the reference block is constrained to be outside the superblock that includes the current unit and also outside the previous superblock. With such a constraint on locations of reference blocks, intra BC prediction is more “hardware friendly”—slow, inefficient memory access operations can be avoided in hardware implementations—but intra BC prediction may be less effective in terms of lowering bit rate and improving quality. SUMMARY In summary, the detailed description presents innovations in int