CN-121985087-A - RIFE improved desert environment video frame inserting method and system

CN121985087ACN 121985087 ACN121985087 ACN 121985087ACN-121985087-A

Abstract

The invention provides a RIFE-based improved desert environment video frame inserting method and system, which are used for acquiring low-frame-rate video frames and environment sensing data, generating an optical flow field through multi-scale motion estimation, carrying out desert environment self-adaptive correction on the optical flow field by combining the environment sensing data, and carrying out 2-time or 4-time self-adaptive frame inserting processing on the basis of the corrected optical flow field to generate high-frame-rate video output. The difficult problem of video frame rate improvement in the desert environment is solved through multi-scale motion estimation, environment self-adaptive frame insertion and embedded optimization.

Inventors

LIU ANJIE
XU TAO
WANG YANG
ZHANG LING
XU MINGXI
WANG CHENYUE
ZOU YIBIN
JIANG JUN
Ge Peijuan

Assignees

上海勘测设计研究院有限公司

Dates

Publication Date: 20260505
Application Date: 20260114

Claims (11)

1. A RIFE-improvement-based desert environment video frame inserting method is characterized by comprising the following steps: s1, acquiring a low-frame-rate video frame and environmental sensing data; s2, generating an optical flow field through multi-scale motion estimation, and carrying out desert environment self-adaptive correction on the optical flow field by combining environment sensing data; s3, based on the corrected optical flow field, performing 2-time or 4-time self-adaptive frame inserting processing to generate high-frame-rate video output.
2. The method for inserting frames in desert environment video based on RIFE improvement according to claim 1, wherein in step S2, multi-scale motion estimation adopts a four-level pyramid structure, optical flow estimation is carried out from 1/8 resolution level by level up to original resolution, and optical flow fields under each level of resolution are subjected to weighted fusion, so that an initial optical flow field is obtained.
3. The method for video frame interpolation based on RIFE improved desert environment according to claim 2, wherein the weighted fusion process of multi-scale motion estimation satisfies the following formula: ; Wherein, the An initial optical flow field generated for multi-scale motion estimation, representing a motion vector for each pixel from one frame to another in a low frame rate video; For the scale level of the pyramid structure, the value is 1 to 4, the corresponding resolution is 1/8 to the original resolution, At the resolution of 1/8 of that of the image, At a resolution of 1/4 of that of the image, At the resolution of 1/2 of that of the image, Is the original resolution; Is the first An optical flow field estimated at each scale level; For upsampling operations for setting Up-sampling the low-resolution optical flow field under the scale level to the original resolution; Is the first Weight coefficient of each scale level optical flow field, and So as to realize the balance of low weight of the coarse-scale optical flow field and high weight of the fine-scale optical flow field.
4. The method for video frame insertion in desert environment based on RIFE improvement according to claim 3, wherein the adaptive correction of desert environment includes two steps: a1, calculating to obtain a rough motion field based on IMU data in environment sensing data, so as to apply motion consistency constraint to an initial optical flow field, and correcting abnormal values which do not accord with a physical motion rule in the initial optical flow field; a2, according to the sand and dust concentration measured by the PM2.5 sensor in the environmental sensing data, carrying out exponential decay correction on the optical flow field corrected in the step A1, and reducing the influence of sand and dust on the optical flow estimation precision.
5. The method for video frame insertion in desert environment based on RIFE improvement according to claim 4, wherein the implementation of motion consistency constraint in step A1 satisfies the following formula: Wherein, the The method is a motion consistency loss function and is used for restraining the initial optical flow field to conform to the physical motion law reflected by IMU data; An optical flow field predicted for the model; for the rough motion field obtained through IMU data calculation, integrating accelerometer data in the IMU to obtain displacement, integrating gyroscope data to obtain an angle, and further converting the displacement and the gyroscope data into motion fields corresponding to camera motion; for L2 norm, used to measure the initial optical flow field And rough playground Differences between; For smooth item of optical flow field, TV-L1 regularization is adopted to the initial optical flow field Is constrained by the gradient of (2) to avoid the occurrence of an unsmooth region in the optical flow field; is a super parameter and is used for balancing the weight of the L2 norm item and the smooth item; The implementation of the digital decay correction in step A2 satisfies the following formula: ; Wherein, the The method is a final optical flow field after self-adaptive correction in desert environments; The optical flow field is corrected by the dynamic consistency constraint in the step A1; The attenuation coefficient is calibrated through a large number of experiments in a desert environment and used for representing the attenuation degree of sand and dust on an optical flow; a concentration of dust measured for the PM2.5 sensor; as an exponential decay factor, concentration of sand and dust The higher the factor value is, the smaller the attenuation degree of the optical flow field is, so as to match the scene with reduced image contrast and reduced optical flow estimation confidence caused by reduced motion visibility under high sand concentration.
6. The method of inserting frames in desert environment video based on RIFE improvement according to claim 1, wherein 4 times of inserting frames in step S3 are processed in two-stage recursion inserting frames, a first stage of inserting frames generates a first stage of intermediate frames between interval frames of original low frame rate video, a second stage of inserting frames generates a second stage of intermediate frames between original frames and the first stage of intermediate frames, and the first stage of intermediate frames and original frames, meanwhile, an error compensation mechanism is introduced in the second stage of inserting frames, optical flow prediction errors are calculated and fed back to the inserting frames, and time sequence error accumulation is restrained.
7. The method for inserting frames into desert environment video based on RIFE improvement according to claim 6, wherein the 4 times of inserting frames are processed by the steps of: b1, obtaining continuous 5 frames of the original low-frame-rate video, and marking as An input frame sequence that is a 4-fold interpolated frame; B2, executing first-stage frame interpolation, and calling an interpolation function based on the corrected optical flow field In the following And (3) with Between generating first level intermediate frames In the following And (3) with Between generating first level intermediate frames Interpolation function The parameter "0.5" of (1) indicates that the frame is And frame Intermediate frames are generated at the time mid-point of (a), i.e , ; B3, executing the second-stage interpolation, and also calling the interpolation function And turning on the error compensation mechanism, in And (3) with Between generating two-level intermediate frames In the following And (3) with Between generating two-level intermediate frames In the following And (3) with Between generating two-level intermediate frames In the following And (3) with Between generating two-level intermediate frames The method comprises the following steps: ; ; ; ; B4, arranging the generated secondary intermediate frame with the original frame and the primary intermediate frame in time sequence to obtain Realizing 4 times frame interpolation from 5 frames of original frames to 9 frames of high frame rate frames; Wherein the error compensation mechanism predicts the error by calculating the optical flow when inserting frames at the second stage Implementation, error The compensation formula is that the difference value between the optical flow field predicted by the interpolation function and the actual motion optical flow field , For the final interpolated frame after error compensation, For the interpolation frames that the interpolation function initially predicts, Is an error compensation coefficient and is used for controlling the intensity of error compensation.
8. The method for inserting frames in desert environment video based on RIFE improvement according to claim 1, wherein the method is deployed on an embedded platform, and the embedded platform is NVIDIA Jetson Orin NX or AGX Xavier; dynamically allocating computing resources according to the selected frame inserting multiple, allocating 50% computing resources by 2 times of frame inserting, and allocating 80% computing resources by 4 times of frame inserting; the memory multiplexing strategy is adopted to carry out cache multiplexing on the feature images in the process of multi-scale motion estimation and frame insertion, so that the memory occupation is reduced; Meanwhile, a temperature self-adaptive strategy is adopted, calculation accuracy is adjusted according to the real-time temperature of the embedded platform, and power consumption and operation stability are controlled.
9. The method for video frame insertion in desert environment based on RIFE improvement according to claim 8, wherein the implementation of temperature self-adaptive strategy satisfies the following formula: ; Wherein, the Representing the setting of the computational accuracy of the embedded platform to a half-accuracy floating point number (FP 16), The method is characterized in that the calculation precision is set to be a single-precision floating point number (FP 32), the temperature is the real-time temperature of an embedded platform and is acquired by a temperature sensor built in the platform; When the temperature is higher than 70.0 ℃, the method is switched to FP16 precision to reduce calculation power consumption and heat generation, and when the temperature is lower than or equal to 70.0 ℃, the FP32 precision is kept to ensure the frame inserting quality.
10. A RIFE-improvement-based desert environment video frame insertion system for implementing the method of any one of claims 1 to 9, comprising: the environment sensing module is used for acquiring IMU, temperature, humidity and PM2.5 data; The image processing module integrates a multi-scale stream estimation unit, a desert environment adaptation unit and a self-adaptive frame inserting unit; And the resource scheduling module is used for dynamically adjusting the calculation load and the precision according to the frame inserting mode.
11. The system for inserting frames into desert environment video based on RIFE improvement according to claim 10, wherein when the adaptive frame inserting unit executes 2 times frame inserting, the two-way optical flow fusion formula is adopted to generate an intermediate frame, and the formula is as follows: ; Wherein, the Intermediate frames generated for 2-fold interpolated frames, located at Frame and the first Between frames; The first to be a low frame rate video The frame of the frame is a frame of a frame, Is the first A frame; To from the first Frame flow to the first The corrected optical flow field of the frame, To from the first Frame flow to the first Corrected optical flow field of the frame; for pixel resampling functions, based on optical flow fields For frames Resampling of the pixels of (1) to obtain a frame In the optical flow field Pixel distribution in the direction is added with the resampling result of the other path to obtain an intermediate frame 。

Description

RIFE improved desert environment video frame inserting method and system Technical Field The invention relates to the technical field of computer vision and embedded artificial intelligence, in particular to a desert environment video frame inserting method and system based on RIFE improvement. Background Booster stations and photovoltaic fields built in extreme environments such as deserts, gobi, barren beaches and the like generally adopt inspection robots for equipment state monitoring. Such environments present special challenges of 1) strong wind and sand causing camera shake and motion blur, 2) high temperature causing image sensor noise increase and thermal drift, 3) high contrast and loss of detail due to intense illumination, 4) optical scattering and color distortion due to sand particles. Together, these factors lead to problems of insufficient frame rate, motion jamming, blurred details, etc. in the acquired video. The RIFE model is used as an advanced real-time frame inserting algorithm, and efficient frame generation is realized through intermediate flow estimation, but the original design has obvious limitations in desert environment aiming at a general scene, namely 1) sensitivity to special degradation such as sand scattering and thermal turbulence, 2) lack of adaptation to the motion characteristics of a robot, 3) serious error accumulation during multiple frame inserting, and 4) insufficient embedded deployment efficiency. In the prior art, optical flow or nuclear prediction methods are mostly adopted for video frame insertion, but the problems of complex calculation or inconsistent time sequence exist, and other solutions do not consider the desert environment characteristics and solve the problems of serious error accumulation and poor real-time performance. Disclosure of Invention The invention mainly aims to provide a RIFE-improved desert environment video frame inserting method and system, which solve the problems of insufficient video frame rate, motion clamping and detail blurring acquired by a patrol robot in a desert environment, poor adaptability, serious error accumulation and low instantaneity of the existing frame inserting scheme. In order to solve the technical problems, the technical scheme adopted by the invention is that the desert environment video frame inserting method based on RIFE improvement comprises the following steps: s1, acquiring a low-frame-rate video frame and environmental sensing data; s2, generating an optical flow field through multi-scale motion estimation, and carrying out desert environment self-adaptive correction on the optical flow field by combining environment sensing data; s3, based on the corrected optical flow field, performing 2-time or 4-time self-adaptive frame inserting processing to generate high-frame-rate video output. In the preferred scheme, in the step S2, the multi-scale motion estimation adopts a four-level pyramid structure, and the optical flow estimation is carried out from 1/8 resolution to the original resolution step by step, and the optical flow fields under the resolutions of all levels are weighted and fused to obtain the initial optical flow field. In a preferred embodiment, the weighted fusion process of the multi-scale motion estimation satisfies the following formula: ; Wherein, the An initial optical flow field generated for multi-scale motion estimation, representing a motion vector for each pixel from one frame to another in a low frame rate video; For the scale level of the pyramid structure, the value is 1 to 4, the corresponding resolution is 1/8 to the original resolution, At the resolution of 1/8 of that of the image,At a resolution of 1/4 of that of the image,At the resolution of 1/2 of that of the image,Is the original resolution; Is the first An optical flow field estimated at each scale level; For upsampling operations for setting Up-sampling the low-resolution optical flow field under the scale level to the original resolution; Is the first Weight coefficient of each scale level optical flow field, andSo as to realize the balance of low weight of the coarse-scale optical flow field and high weight of the fine-scale optical flow field. In a preferred scheme, the adaptive correction of the desert environment comprises two steps of processing: a1, calculating to obtain a rough motion field based on IMU data in environment sensing data, so as to apply motion consistency constraint to an initial optical flow field, and correcting abnormal values which do not accord with a physical motion rule in the initial optical flow field; a2, according to the sand and dust concentration measured by the PM2.5 sensor in the environmental sensing data, carrying out exponential decay correction on the optical flow field corrected in the step A1, and reducing the influence of sand and dust on the optical flow estimation precision. In a preferred embodiment, the implementation of the motion consistency constraint in step A1 sa