US-12627829-B2 - Systems and methods for optical flow-based motion vector refinement

Abstract

An example method of video coding includes receiving a video bitstream comprising a plurality of blocks. The method also includes deriving a set of subblock motion vectors for a current subblock in a first block of the plurality of blocks, and deriving a set of refined subblock motion vectors for the current subblock by applying an optical flow refinement that uses one or more values that are outside of the current subblock, where a block size for the optical flow refinement is a block size of the current subblock. The method further includes reconstructing the current subblock using the set of refined subblock motion vectors.

Inventors

• Han Gao
• Xin Zhao
• Liang Zhao
• Jing Ye
• Shan Liu

Assignees

• Tencent America LLC

Dates

Publication Date

20260512

Application Date

20240329

Claims (20)

1. 1 . A method of video decoding performed at a computing system having memory and one or more processors, the method comprising: receiving a video bitstream comprising a plurality of blocks; deriving a set of subblock motion vectors for a current subblock in a first block of the plurality of blocks; deriving a set of refined subblock motion vectors for the current subblock by applying an optical flow refinement that uses one or more values that are outside of the current subblock, wherein a block size for the optical flow refinement is a block size of the current subblock; in accordance with applying the optical flow refinement that uses the one or more values that are outside of the current subblock: performing an initial interpolation at a subblock level; and selecting an interpolation filter based on a position of the current subblock; and reconstructing the current subblock using the set of refined subblock motion vectors.
2. 2 . The method of claim 1 , wherein the optical flow refinement uses a set of lines outside of the current subblock, and wherein the one or more values are from the set of lines.
3. 3 . The method of claim 2 , wherein a number of the set of lines is selected based on a location for the current subblock.
4. 4 . The method of claim 2 , wherein the set of lines outside of the current subblock comprises one or more lines above the current subblock.
5. 5 . The method of claim 1 further comprising, before applying the optical flow refinement for the current subblock, padding one or more interpolated predictors for the current subblock.
6. 6 . The method of claim 5 , wherein the one or more values that are outside of the current subblock correspond to N lines outside of the current subblock, and wherein the one or more interpolated predictors are padded with N−1 values.
7. 7 . The method of claim 5 , wherein the padding is performed at a block level or a subblock level.
8. 8 . The method of claim 1 , further comprising: in accordance with applying the optical flow refinement that uses the one or more values that are outside of the current subblock, applying an N-tap interpolation filter to generate one or more interpolated predictors, wherein Nis less than eight; and in accordance with applying an optical flow refinement that does not use values that are outside of the current subblock, applying an 8-tap interpolation filter to generate the one or more interpolated predictors.
9. 9 . The method of claim 8 , wherein the N-tap interpolation filter comprises a bilinear interpolation filter.
10. 10 . The method of claim 1 , further comprising, in accordance with applying the optical flow refinement that uses the one or more values that are outside of the current subblock, using a rounded motion vector to generate one or more interpolated predictors.
11. 11 . A computing system, comprising: control circuitry; memory; and one or more sets of instructions stored in the memory and configured for execution by the control circuitry, the one or more sets of instructions comprising instructions for: receiving video data comprising a plurality of blocks including a current block; identifying a set of subblock motion vectors for a current subblock in a first block of the plurality of blocks; identifying a set of refined subblock motion vectors for the current block by applying an optical flow refinement that uses one or more values that are outside of the current subblock, wherein a block size for the optical flow refinement is a block size of the current subblock; in accordance with applying the optical flow refinement that uses the one or more values that are outside of the current subblock: performing an initial interpolation at a subblock level; and selecting an interpolation filter based on a position of the current subblock; and encoding the current subblock using the set of refined subblock motion vectors.
12. 12 . The computing system of claim 11 , wherein the optical flow refinement uses a set of lines outside of the current subblock, and wherein the one or more values are from the set of lines.
13. 13 . The computing system of claim 12 , wherein a number of the set of lines is selected based on a location for the current subblock.
14. 14 . The computing system of claim 12 , wherein the set of lines outside of the current subblock comprises one or more lines above the current subblock.
15. 15 . The computing system of claim 12 , wherein the interpolation filter comprises an N-tap interpolation filter.
16. 16 . The computing system of claim 12 , wherein the interpolation filter comprises a bilinear interpolation filter.
17. 17 . The computing system of claim 12 , wherein the one or more sets of instructions further comprise instructions for, before applying the optical flow refinement for the current subblock, padding one or more interpolated predictors for the current subblock.
18. 18 . A method of obtaining a video bitstream, the video bitstream comprising an encoded subblock of a current block, the encoded subblock encoded using a set of refined subblock motion vectors, the method comprising: identifying a set of subblock motion vectors for a current subblock in the current block, the current subblock corresponding to the encoded subblock; identifying the set of refined subblock motion vectors for the current block by applying an optical flow refinement that uses one or more values that are outside of the current subblock, wherein a block size for the optical flow refinement is a block size of the current subblock; and in accordance with applying the optical flow refinement that uses the one or more values that are outside of the current subblock: performing an initial interpolation at a subblock level; and selecting an interpolation filter based on a position of the current subblock.
19. 19 . The method of claim 18 , wherein the optical flow refinement uses a set of lines outside of the current subblock, and wherein the one or more values are from the set of lines.
20. 20 . The method of claim 18 , wherein a number of the set of lines is selected based on a location for the current subblock.

Description

RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63/611,687, entitled “Overlapped Optical Flow-Based Motion Vector Refinement” filed Dec. 18, 2023, which is hereby incorporated by reference in its entirety. TECHNICAL FIELD The disclosed embodiments relate generally to video coding, including but not limited to systems and methods for refining motion vectors using values outside of a current subblock of video data. BACKGROUND Digital video is supported by a variety of electronic devices, such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video gaming consoles, smart phones, video teleconferencing devices, video streaming devices, etc. The electronic devices transmit and receive or otherwise communicate digital video data across a communication network, and/or store the digital video data on a storage device. Due to a limited bandwidth capacity of the communication network and limited memory resources of the storage device, video coding may be used to compress the video data according to one or more video coding standards before it is communicated or stored. The video coding can be performed by hardware and/or software on an electronic/client device or a server providing a cloud service. Video coding generally utilizes prediction methods (e.g., inter-prediction, intra-prediction, or the like) that take advantage of redundancy inherent in the video data. Video coding aims to compress video data into a form that uses a lower bit rate, while avoiding or minimizing degradations to video quality. Multiple video codec standards have been developed. For example, High-Efficiency Video Coding (HEVC/H.265) is a video compression standard designed as part of the MPEG-H project. ITU-T and ISO/IEC published the HEVC/H.265 standard in 2013 (version 1), 2014 (version 2), 2015 (version 3), and 2016 (version 4). Versatile Video Coding (VVC/H.266) is a video compression standard intended as a successor to HEVC. ITU-T and ISO/IEC published the VVC/H.266 standard in 2020 (version 1) and 2022 (version 2). AOMedia Video 1 (AV1) is an open video coding format designed as an alternative to HEVC. On Jan. 8, 2019, a validated version 1.0.0 with Errata 1 of the specification was released. SUMMARY The present disclosure describes, amongst other things, a set of techniques for video (image) compression related to optical flow-based motion vector refinement. The set of techniques include techniques for overlapped optical flow-based motion vector refinement. For example, when computing the subblock refined motion vectors of the optical flow refinement, extra samples, and gradient values beyond the current subblock (e.g., lines and/or columns) may be used to derive the subblock refined motion vectors, whereas the block size for final motion compensation of the optical flow-based modes is kept unchanged. Enabling more samples for the optical refinement improves the accuracy of the motion vectors (e.g., without requiring additional memory costs) improves the accuracy of video decoding (e.g., frame reconstruction). In accordance with some embodiments, a method of video decoding includes: (i) receiving a video bitstream comprising a plurality of blocks; (ii) deriving a set of subblock motion vectors for a current subblock in a first block of the plurality of blocks; (iii) deriving a set of refined subblock motion vectors for the current subblock by applying an optical flow refinement that uses one or more values that are outside of the current subblock, where a block size for the optical flow refinement is a block size of the current subblock; and (iv) reconstructing the current subblock using the set of refined subblock motion vectors. In accordance with some embodiments, a method of video encoding includes (i) receiving video data comprising a plurality of blocks including a current block; (ii) identifying a set of subblock motion vectors for a current subblock in a first block of the plurality of blocks; (iii) identifying a set of refined subblock motion vectors for the current block by applying an optical flow refinement that uses one or more values that are outside of the current subblock, where a block size for the optical flow refinement is a block size of the current subblock; and (iv) encoding the current subblock using the set of refined subblock motion vectors. In accordance with some embodiments, a computing system is provided, such as a streaming system, a server system, a personal computer system, or other electronic device. The computing system includes control circuitry and memory storing one or more sets of instructions. The one or more sets of instructions including instructions for performing any of the methods described herein. In some embodiments, the computing system includes an encoder component and a decoder component (e.g., a transcoder). In accordance with some embodiments, a non-t