US-12620067-B2 - Motion blur for multilayer motion

Abstract

Systems, apparatus, and methods for adding post-processing motion blur to video. Conventional post-processing techniques relied on the filmmaker to select and stage their shots. Different motion blur techniques were designed to fix certain types of footage. Vector blur is one technique that “smears” pixel information in the direction of movement. Frame interpolation and stacking attempts to create motion blur by stacking interpolated frames together. Each technique has its own set of limitations. Various embodiments use a combination of motion blur techniques in post-processing for better, more realistic outcomes with faster/more efficient rendering times. In some cases, this may enable adaptive quality post-processing that may be performed in mobile/embedded ecosystems.

Inventors

• Robert McIntosh
• David Newman

Assignees

• GOPRO, INC.

Dates

Publication Date

20260505

Application Date

20230323

Claims (20)

1. 1 . A post-processing device, comprising: a processor; and a non-transitory computer-readable medium comprising a set of instructions that, when executed by the processor, causes the processor to: determine a first amount of motion in a first frame of video data; determine whether the first amount of motion is over a threshold value; select a first motion blur technique between at least a second motion blur technique or a third motion blur technique based on the determining whether the first amount of motion is over the threshold value; perform the first motion blur technique on the first frame of video data; determine a second amount of motion in second frame of the video data; determine whether the second amount of motion is over the threshold value; select a fourth motion blur technique between at least the second motion blur technique or the third motion blur technique based on determining whether the second amount of motion is over the threshold value; and perform the fourth motion blur technique on the second frame of video data.
2. 2 . The post-processing device of claim 1 , where the fourth motion blur technique is different from the first motion blur technique.
3. 3 . The post-processing device of claim 1 , where the second motion blur technique is a faster rendering technique compared with the third motion blur technique.
4. 4 . The post-processing device of claim 1 , where: the second motion blur technique comprises a vector blur technique, and the third motion blur technique comprises frame interpolation.
5. 5 . The post-processing device of claim 1 , where determining the first amount of motion in the first frame of video data comprises determining one or more motion vectors associated with motion.
6. 6 . The post-processing device of claim 1 , where the set of instructions further causes the processor to: determine contrasting motion in the video data, where selecting the first motion blur technique is further based on the determination of contrasting motion.
7. 7 . The post-processing device of claim 1 , where the set of instructions further causes the processor to: receive the video data from a camera device.
8. 8 . The post-processing device of claim 1 , where performing the first motion blur technique and performing the fourth motion blur technique are based on a selection of a shutter angle.
9. 9 . The post-processing device of claim 1 , where the set of instructions further causes the processor to: determine a plurality of sections of the first frame of video data, where performing the first motion blur technique on the first frame of video data comprises performing the first motion blur technique on a first section of the plurality of sections; select a fifth motion blur technique between at least the second motion blur technique or the third motion blur technique based on a third amount of motion on a second section of the plurality of sections of the first frame of video data; perform the fifth motion blur technique on the second section of the plurality of sections of the first frame; and composite each section of the plurality of sections into a composited frame.
10. 10 . The post-processing device of claim 9 , where the set of instructions further causes the processor to: feather regions between the plurality of sections of the composited frame.
11. 11 . A method of adding motion blur to video data, comprising: determining a first amount of motion between a first set of consecutive frames of video data; determining whether the first amount of motion is over a threshold value; selecting a first motion blur technique between at least a second motion blur technique or a third motion blur technique based on the determining whether the first amount of motion is over the threshold value; performing the first motion blur technique on at least one frame of the first set of consecutive frames of video data; determining a second amount of motion between a second set of consecutive frames of video data; determining whether the second amount of motion is over the threshold value; selecting a fourth motion blur technique between at least the second motion blur technique or the third motion blur technique based on determining whether the second amount of motion is over the threshold value; and performing the fourth motion blur technique on at least one frame of the second set of consecutive frames of video data.
12. 12 . The method of claim 11 , further comprising calculating a plurality of motion vectors of pixel data between the first set of consecutive frames of video data.
13. 13 . The method of claim 12 , further comprising performing optical flow on the first set of consecutive frames of video data, where calculating the plurality of motion vectors of pixel data is based on performing optical flow.
14. 14 . The method of claim 12 , where: performing the first motion blur technique comprises performing a vector blur on the at least one frame of the first set of consecutive frames of video data based on the plurality of motion vectors, and performing the vector blur comprises: sampling pixel values of image data across a first motion vector of the plurality of motion vectors of pixel data, the first motion vector associated with at least one pixel of the pixel data; and combining a portion of the at least one pixel with the pixel values of image data.
15. 15 . The method of claim 12 , further comprising performing edge detection on the plurality of motion vectors of pixel data to determine contrasting motion between the first set of consecutive frames of video data.
16. 16 . The method of claim 11 , where performing the first motion blur technique comprises: generating one or more intermediate frames between the first set of consecutive frames of video data; performing a vector blur on the one or more intermediate frames and the at least one frame of the first set of consecutive frames of video data; and stacking the one or more intermediate frames and the at least one frame of the first set of consecutive frames of video data.
17. 17 . The method of claim 16 , where determining a first amount of motion is based on estimating non-linear motion between the first set of consecutive frames of video data using an artificial intelligence model.
18. 18 . A video capture device, comprising: a sensor configured to capture video data; a processor; and a non-transitory computer-readable medium comprising a set of instructions that, when executed by the processor, causes the processor to: determine a first amount of motion in a first frame of video data; determine whether the first amount of motion is over a threshold value; select a first motion blur technique between at least a second motion blur technique or a third motion blur technique based on the determining whether the first amount of motion is over the threshold value; perform the first motion blur technique on the first frame of video data; determine a second amount of motion in second frame of the video data; determine whether the second amount of motion is over the threshold value; select a fourth motion blur technique between at least the second motion blur technique or the third motion blur technique based on determining whether the second amount of motion is over the threshold value; and perform the fourth motion blur technique on the second frame of video data.
19. 19 . The video capture device of claim 18 , where the set of instructions further causes the processor to add a slow-motion effect on at least a portion of the video data.
20. 20 . The video capture device of claim 18 , further comprising an inertial measurement unit (IMU), the IMU comprising at least one of an accelerometer, a gyroscope, and a magnetometer, where determining the first amount of motion in the first frame of video data is based on sensor data received from the IMU.

Description

COPYRIGHT A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights whatsoever. TECHNICAL FIELD This disclosure relates generally to the field of digital image capture and post-processing. More particularly, the present disclosure relates to systems, computer programs, devices, and methods for post-processing motion blur. DESCRIPTION OF RELATED TECHNOLOGY When a camera captures an image, its sensor is exposed to light for a period of time. A captured image represents the collected light information, from the camera's viewpoint, over the exposure time. Typically, the exposure time is selected to maximize image clarity, but blurs can also be intentionally added to create the impression of motion. Motion blur artifacts smear the image along the direction of relative motion. In other words, the smearing is created by both the object's movement (actual motion) and/or the camera's movement (relative motion). Certain camera movements (jitter or camera shake) may introduce camera blur for subjects-even for stationary subjects—that is undesirable. Undesirable motion blur can be reduced or removed using stabilization techniques such as mechanical/optical image stabilization and electronic image stabilization (EIS). In some cases, motion blur may be intentionally manipulated by changing the relative motion between the camera and a target of interest. As but one example, the camera may be panned to track a moving object; in this case, even with long exposure times, the moving objects will appear sharper while the background will become more blurred. This can be used to convey a sense of the viewer's movement and speed. Professional and amateur filmmakers often use motion blur to impart a natural look and a sense of motion to film. Motion blur can also visually indicate the relative speeds of objects. Unfortunately, certain capture modes may not work correctly with motion blur, and vice versa. For example, capture settings for slow-motion and/or stabilized video often conflict with capture settings for motion blur. Specifically, slow-motion effects and electronic image stabilization (EIS) are typically created with high frame rate video with a relatively fast shutter speed, while motion blur naturally occurs in image/video capture with a relatively slow shutter speed. Post-processing techniques can artificially add capture effects with digital image manipulation. For example, a filmmaker may capture video at a high frame rate (e.g., 240 frames per second (fps)) and then convert the video to slow motion video at a normal frame rate (e.g., 30 fps). In another example, a filmmaker may capture video at a high frame rate and then stabilize the video in post-processing. In both these examples, a high frame rate is physically needed for the capture, but the relatively short exposure time will reduce the natural motion blur. During post-processing, motion blur can be artificially added to give the perception of motion for the intended playback speed/stabilized field-of-view. More generally, existing post-processing techniques allow a filmmaker to physically capture video at a first frame rate, but digitally add effects (e.g., slow motion, image stabilization, etc.) consistent with a different capture frame rate. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a graphical representation of a vector blur technique for adding post-processing motion blur. FIG. 2 is a graphical representation of a frame interpolation and stacking technique for adding post-processing motion blur. FIG. 3 is a comparison of motion blur effects introduced with vector blur and multiple levels of frame interpolation (with stacking) on the same video data. FIG. 4 illustrates an exemplary logical flow diagram of a combined vector blur and frame interpolation with stacking technique for adding motion blur according to aspects of the present disclosure. FIGS. 5A-5B are graphical representations of one exemplary technique for adding post-processing motion blur according to aspects of the present disclosure. FIG. 6 illustrates exemplary frames generated with the exemplary technique for adding post-processing motion blur according to aspects of the present disclosure. FIG. 7 illustrates an exemplary logical flow diagram of an adaptive rendering technique for adding motion blur according to aspects of the present disclosure. FIG. 8 illustrates a sample frame of an exemplary video according to aspects of the present disclosure. FIG. 9 illustrates a sample frame of an exemplary video according to aspects of the present disclosure. FIG. 10 illustrates a sample frame of an exemplary video according to aspects of the present disclosure. FIG. 11 illustrates an exemplary logical flo