CN-121985209-A - Image motion estimation method and electronic device

CN121985209ACN 121985209 ACN121985209 ACN 121985209ACN-121985209-A

Abstract

An image motion estimation method and an electronic device are disclosed, the image motion estimation method including obtaining a first RGB image captured at a current time point and a second RGB image captured at a previous time point by an RGB image sensor of the electronic device, obtaining a first angular velocity of the electronic device at the current time point, obtaining a first acceleration of the electronic device at the current time point, and determining a first motion velocity of a pixel in the first RGB image and a first translation velocity of the electronic device at the current time point based on the first RGB image, the second translation velocity, the first angular velocity, and the first acceleration of the electronic device at the previous time point.

Inventors

FANG JINGYU
LI JIE
XIE YONGGANG
TIAN RUN

Assignees

三星半导体(中国)研究开发有限公司
三星电子株式会社

Dates

Publication Date: 20260505
Application Date: 20260202

Claims (10)

1. An image motion estimation method, comprising: Obtaining a first RGB image photographed by an RGB image sensor of an electronic device at a current time point and a second RGB image photographed at a previous time point; Obtaining a first angular velocity of the electronic device at a current point in time; Obtaining a first acceleration of the electronic device at a current point in time; The method includes determining a first movement speed of a plurality of first pixels in a first RGB image and a first translation speed of the electronic device at a current point in time based on the first RGB image, a second translation speed of the electronic device at a previous point in time, a first angular speed of the electronic device at the current point in time, and a first acceleration of the electronic device at the current point in time to estimate image movement.
2. The image motion estimation method of claim 1 further comprising deblurring a third RGB image captured by the RGB image sensor at a later point in time based on the first motion speed of the plurality of first pixels in the first RGB image.
3. The image motion estimation method of claim 1, wherein determining the first motion speed and the first translation speed comprises: Determining a first motion speed, a first translation speed and a derivative of a first rotation matrix at a current point in time based on the first RGB image, the second translation speed, the first angular speed and the first acceleration in combination with the first function, the second function, the third function and the fourth function, Wherein the first function embodies a relationship between a derivative of camera coordinates corresponding to a first pixel representing a stationary object among the plurality of first pixels and the camera coordinates, a derivative of a first rotation matrix, and a first translation speed, wherein the stationary object is an object stationary during a period of time between a previous point of time and a current point of time among corresponding objects among the first RGB image and the second RGB image, Wherein the second function reflects a relationship between coordinates of the plurality of first pixels in the first RGB image and coordinates of a plurality of second pixels in the second RGB image corresponding to the plurality of first pixels in the first RGB image and a first movement speed of the plurality of first pixels in the first RGB image, Wherein the third function represents a relationship between the first angular velocity and a derivative of the first rotation matrix, Wherein the fourth function embodies the relationship between the second translational velocity and the first acceleration and the first translational velocity.
4. The image motion estimation method according to claim 3, Wherein in the first function, the derivative of the first rotation matrix and the product of the first translation speed and the camera coordinates is equal to the derivative of the camera coordinates, Wherein the camera coordinates are obtained by substituting the coordinates of a first pixel representing the stationary object in a first RGB image and the Z-coordinate of the stationary object at the current point in time into an internal reference matrix, Wherein the Z coordinate of the stationary object at the current point in time is obtained based on a second translation speed, the Z coordinate of the stationary object at a previous point in time, and the time period between the previous point in time and the current point in time, Wherein the derivative of the camera coordinates comprises a first motion speed representing a first pixel of the stationary object.
5. The image motion estimation method according to claim 3, further comprising obtaining a first depth image at a current point in time, Wherein in the first function, the derivative of the first rotation matrix and the product of the first translation speed and the camera coordinates is equal to the derivative of the camera coordinates, Wherein the camera coordinates are obtained by substituting the coordinates of a first pixel representing the stationary object in a first RGB image and the Z-coordinate of the stationary object at the current point in time into an internal reference matrix, Wherein the Z coordinate of the stationary object at the current point in time is obtained based on the first depth image, Wherein the derivative of the camera coordinates comprises a first motion speed representing a first pixel of the stationary object.
6. The image motion estimation method according to claim 5, wherein the step of obtaining the first depth image at the current point in time comprises: Performing conversion on a depth image captured by a depth sensor of the electronic device such that the converted depth image is time-synchronized with the first RGB image and the second RGB image, wherein the converted depth image corresponds to an image coordinate system, and wherein a resolution of the converted depth image is the same as a resolution of each of the first RGB image and the second RGB image; a first depth image at a current point in time is obtained from the converted depth image, Wherein the depth sensor and the RGB image sensor capture the same scene.
7. The image motion estimation method according to claim 3, further comprising: obtaining a first event image at a current point in time, and Further in conjunction with the fifth function, Wherein the fifth function embodies a relationship between a spatial gradient of the first RGB image in the x-direction and the y-direction, a first motion velocity of the pixel in the first RGB image, and the first event image.
8. The image motion estimation method according to claim 7, wherein the step of obtaining the first event image at the current point in time includes: performing conversion on an event image captured by an event sensor of the electronic device such that the converted event image is time-synchronized with the first RGB image and the second RGB image, wherein the converted event image corresponds to an image coordinate system, and wherein a resolution of the converted event image is the same as a resolution of each of the first RGB image and the second RGB image; a first event image at a current point in time is obtained from the converted event image, Wherein the event sensor and the RGB image sensor capture the same scene.
9. An electronic device includes An RGB image sensor configured to capture a first RGB image at a current point in time and a second RGB image at a previous point in time; The image processing apparatus includes a processor configured to obtain a first RGB image and a second RGB image, obtain a first angular velocity of the electronic device at a current point in time, obtain a first acceleration of the electronic device at the current point in time, and determine a first movement velocity of a plurality of first pixels in the first RGB image and a first translation velocity of the electronic device at the current point in time based on the first RGB image, the second translation velocity of the electronic device at a previous point in time, the first angular velocity of the electronic device at the current point in time, and the first acceleration of the electronic device at the current point in time to estimate an image movement.
10. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1.

Description

Image motion estimation method and electronic device Technical Field The present disclosure relates to the field of image processing, and more particularly, to an image motion estimation method and an electronic device. Background In image processing, a main objective of motion estimation is to analyze a displacement of a corresponding pixel (or a motion velocity of the corresponding pixel) between consecutive image frames (e.g., caused by at least one of a motion of an electronic device including an image sensor and a motion of an object photographed by the electronic device). However, when the motion speed of the electronic device and/or the motion speed of the object is fast, the image frame is blurred, and an error in the motion speed of the pixel obtained by the motion estimation is large. Disclosure of Invention This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. The application provides an image motion estimation method and an electronic device, which can at least improve the accuracy of estimating the motion speed of pixels. According to one or more example embodiments, an image motion estimation method includes obtaining a first RGB image captured at a current point in time and a second RGB image captured at a previous point in time by an RGB image sensor of an electronic device, obtaining a first angular velocity of the electronic device at the current point in time, obtaining a first acceleration of the electronic device at the current point in time, determining a first motion velocity of a plurality of first pixels in the first RGB image and a first translation velocity of the electronic device at the current point in time based on the first RGB image, the second translation velocity of the electronic device at the previous point in time, the first angular velocity of the electronic device at the current point in time, and the first acceleration of the electronic device at the current point in time to estimate an image motion. The image motion estimation method may further include deblurring a third RGB image photographed by the RGB image sensor at a later point in time based on a first motion speed of the plurality of first pixels in the first RGB image. The determining of the first movement speed and the first translation speed may include determining the first movement speed, the first translation speed, and a derivative of a first rotation matrix at a current point in time based on a first RGB image, a second translation speed, a first angular speed, and a first acceleration in parallel with a first function, a second function, a third function, and a fourth function, wherein the first function represents a relationship between a derivative of a camera coordinate corresponding to a first pixel representing a stationary object among the plurality of first pixels and the camera coordinate, the derivative of the first rotation matrix, and the first translation speed, wherein the stationary object is an object stationary during a period of time between a previous point in time and a current point in the first RGB image, wherein the second function represents a coordinate of the plurality of first pixels in the first RGB image and a derivative of a first rotation matrix among the plurality of first pixels in the first RGB image, wherein the stationary object represents a relationship between a coordinate of the plurality of second pixels in the first RGB image and the first rotation matrix, and wherein the first function represents a first angular speed. In the present application, in performing motion estimation, accuracy and reliability of estimating a motion speed of a pixel can be improved by determining a translational speed of an electronic device as a global motion vector using an acceleration of the electronic device. In the present application, improved motion estimation is obtained by fusing the outputs of various different types of sensors (e.g., RGB image sensors, sensors that directly or indirectly measure angular velocity and acceleration), and joint optimization based on these outputs as constraints, thereby further improving the accuracy of estimating the motion speed of the pixel. In a first function, a derivative of a first rotation matrix and a product of a first translation speed and the camera coordinates are equal to a derivative of the camera coordinates, wherein the camera coordinates are obtained by substituting a coordinate representing a first pixel of the stationary object and a Z coordinate of the stationary object at a current point in time in a first RGB image into an internal reference matrix, wherein the Z coordinate of the stationary object at the current point in time is