CN-121985088-A - Video frame inserting method and device and electronic equipment

CN121985088ACN 121985088 ACN121985088 ACN 121985088ACN-121985088-A

Abstract

The embodiment of the application provides a video frame inserting method and device and electronic equipment. The method comprises the steps of obtaining second optical flow information through optimizing first optical flow information, distinguishing a consistent motion area from a boundary motion area by means of a boundary motion area image obtained based on the second optical flow information, generating first characteristic information by combining a rearranged image and fusing multiple image characteristics, guiding pixel warping by matching with accurate first motion information, and capable of solving the problems of boundary dislocation and detail blurring caused by poor resolution of an old image and optical flow deletion, wherein the distortion degree of a finally generated frame inserting image is obviously reduced, and the quality of the frame inserting image is improved.

Inventors

ZHANG XIANLIN
Qi Mengshi
LI XUEMING
HAN JIATONG
YUAN ZENAN

Assignees

北京邮电大学

Dates

Publication Date: 20260505
Application Date: 20260127

Claims (10)

1. A method for video framing, comprising: acquiring first optical flow information between a first image and a second image, wherein the first image and the second image are used for generating a frame image to be inserted; determining second optical flow information between the frame image to be interpolated and the first and second images based on the first optical flow information; Determining a boundary motion region image based on the second optical flow information, the boundary motion region map being used to distinguish a consistent motion region from a boundary motion region in the second optical flow information; Determining a rearranged image based on the first optical flow information and the second image; determining first feature information and first motion information of information fusing the images based on the first image, the second image, the rearranged image and the boundary motion area map, wherein the first motion information indicates motion estimation from the first image to the image to be interpolated; And determining an interpolated image between the first image and the second image based on the first image, the first feature information and the first motion information.
2. The method of claim 1, wherein determining second optical flow information between the frame image to be interpolated and the first image and the second image based on the first optical flow information comprises: Based on the first optical flow information, determining third optical flow information between the frame image to be inserted and the first image and the second image by adopting a bilateral symmetry motion mode; And performing iterative optimization based on feature matching and residual error learning on the third optical flow information to obtain the second optical flow information.
3. The method of claim 2, wherein determining a rearranged image based on the second optical flow information and the second image comprises: Determining a pixel motion trajectory between the first image and a second image based on the second optical flow information; And carrying out pixel rearrangement on the second image based on the pixel motion track, and determining the rearranged image.
4. A method according to claim 3, wherein determining first feature information and first motion information of information fusing the images based on the first image, the second image, the rearranged image, and the boundary motion region map, comprises: Determining a first feature of the first image, a second feature of the second image, and a third feature of the rearranged image; generating a query vector based on the first feature; Generating a first key vector and a first value vector based on the second feature; generating a second key vector and a second value vector based on the third feature; Determining a first attention map for extracting a correlation between the first image and the second image from the consistent motion region based on the query vector and the first key vector; Determining a second attention map for extracting a correlation between the first image and the rearranged image from the boundary motion region based on the query vector and the second key vector; Determining the first feature information based on the first attention map, the second attention map, the first value vector, the second value vector, the first feature, and the boundary motion region map; the first motion information is determined based on the first image, the first attention profile, the second attention profile, the pixel motion trajectory, and the boundary motion zone map.
5. The method of claim 4, wherein determining the first feature information based on the first attention profile, the second attention profile, the first value vector, the second value vector, the first feature, and the boundary motion region map comprises: obtaining a first weighted feature corresponding to the consistent motion region based on the first attention map and the first value vector; Obtaining a second weighted feature corresponding to a boundary motion region based on the second attention map and the second value vector; And fusing the first weighted feature and the second weighted feature by using the boundary motion area diagram as a weight, and adding the first weighted feature and the second weighted feature to the first feature to obtain the first feature information.
6. The method of claim 4, wherein determining the first motion information based on the first image, the first attention map, the second attention map, the pixel motion trajectory, and the boundary motion region map comprises: determining a coordinate graph based on the first image, wherein the size of the coordinate graph is the same as that of the first image; determining a fourth characteristic of the pixel motion profile; estimating a first motion vector for the consistent motion region based on the first attention map and the coordinate map; estimating a second motion vector for the boundary motion region based on the second attention map, the fourth feature, and the coordinate map; Fusing the first motion vector and the second motion vector by using the boundary motion area diagram as weight to obtain initial motion information from the first image to the second image; the first motion information is determined based on the initial motion information.
7. The method of claim 6, wherein determining the first motion information based on the initial motion information comprises: obtaining a distance graph, wherein the distance graph is used for representing the proportion of the motion from the first image to the frame image to be inserted to the total motion from the first image to the second image; the product of the initial motion information and the distance map is determined as the first motion information.
8. The method of claim 4, wherein determining an inter-frame image between the first image and the second image based on the first image, the first feature information, and the first motion information comprises: based on the first motion information, performing warping processing on the first image and the first characteristic information to obtain an intermediate frame image and an intermediate frame characteristic; And carrying out detail enhancement on the intermediate frame image and the intermediate frame characteristic to obtain the frame inserting image.
9. A video framing apparatus, comprising: The device comprises an acquisition module, a first image generation module and a second image generation module, wherein the acquisition module is used for acquiring first optical flow information between a first image and a second image, and the first image and the second image are used for generating a frame image to be inserted; The first determining module is further used for determining second optical flow information between the frame image to be inserted and the first image and the second image based on the first optical flow information; A second determining module, configured to determine a boundary motion area image based on the second optical flow information, where the boundary motion area image is used to distinguish a consistent motion area from a boundary motion area in the second optical flow information; a third determining module further configured to determine a rearranged image based on the first optical flow information and the second image; A fourth determining module, configured to determine, based on the first image, the second image, the rearranged image, and the boundary motion region map, first feature information and first motion information that fuse information of the images, where the first motion information indicates motion estimation from the first image to the frame image to be inserted; And the fifth determining module is further used for determining an interpolation frame image between the first image and the second image based on the first image, the first characteristic information and the first motion information.
10. An electronic device is characterized by comprising a memory and a processor; the memory stores computer-executable instructions; the processor executing computer-executable instructions stored in the memory, causing the processor to perform the method of any one of claims 1-8.

Description

Video frame inserting method and device and electronic equipment Technical Field The present application relates to the field of computer technologies, and in particular, to a video frame inserting method, a video frame inserting device, and an electronic device. Background At present, a video frame interpolation algorithm mainly relies on optical flow information estimation between a front frame and a rear frame to infer an intermediate motion of the front frame and the rear frame, so as to obtain an interpolation frame image between the front frame and the rear frame. However, the old image has typical degradation problems such as low frame rate, poor resolution, motion blur and the like, so that the optical flow information between the front frame and the rear frame is seriously insufficient, and the interpolated image is distorted. Disclosure of Invention The embodiment of the application provides a video frame inserting method and device and electronic equipment, which are used for improving the quality of frame inserting images. In a first aspect, a video frame inserting method according to an embodiment of the present application includes: Acquiring first optical flow information between a first image and a second image, wherein the first image and the second image are used for generating a frame image to be inserted; determining second optical flow information between the image to be interpolated and the first image and the second image based on the first optical flow information; Determining a boundary motion area image based on the second optical flow information, the boundary motion area image being used to distinguish a consistent motion area from a boundary motion area in the second optical flow information; determining a rearranged image based on the first optical flow information and the second image; Determining first feature information and first motion information of information fusing the images based on the first image, the second image, the rearranged image and the boundary motion area map, the first motion information indicating motion estimation from the first image to the image to be interpolated; An interpolated image between the first image and the second image is determined based on the first image, the first feature information, and the first motion information. In some embodiments, determining second optical flow information between the image to be interpolated and the first and second images based on the first optical flow information comprises: based on the first optical flow information, determining third optical flow information between the image to be inserted and the first image and the second image by adopting a bilateral symmetry motion mode; and performing iterative optimization based on feature matching and residual error learning on the third optical flow information to obtain second optical flow information. In some embodiments, determining a rearranged image based on the second optical flow information and the second image includes: determining a pixel motion trajectory between the first image and the second image based on the second optical flow information; And (3) carrying out pixel rearrangement on the second image based on the pixel motion trail, and determining a rearranged image. In some embodiments, determining first feature information and first motion information of information fusing the images based on the first image, the second image, the rearranged image, and the boundary motion region map includes: determining a first feature of the first image, a second feature of the second image, and a third feature of the rearranged image; generating a query vector based on the first feature; Generating a first key vector and a first value vector based on the second feature; Generating a second key vector and a second value vector based on the third feature; Determining a first attention map based on the query vector and the first key vector, the first attention map being used to extract a correlation between the first image and the second image from the consistent motion region; determining a second attention map based on the query vector and the second key vector, the second attention map being used to extract a correlation between the first image and the rearranged image from the boundary motion region; Determining first feature information based on the first attention map, the second attention map, the first value vector, the second value vector, the first feature, and the boundary motion region map; first motion information is determined based on the first image, the first attention map, the second attention map, the pixel motion trajectory, and the boundary motion region map. In some embodiments, determining the first feature information based on the first attention map, the second attention map, the first value vector, the second value vector, the first feature, and the boundary motion region map comprises: obtaining a first weighted feature corresponding