EP-4740178-A1 - METHOD FOR PROCESSING AN IMAGE OF A SCENE IN ORDER TO RENDER A CHANGE IN THE VIEWPOINT OF THE SCENE

Abstract

The invention relates to a method for modifying an image of a scene in order to render a change in the viewpoint of the image of the scene, which method comprises the steps of displaying the image of the scene, detecting a change in the viewpoint compared to the displayed image, modifying the displayed image by moving a portion of the pixels or group(s) of pixels of the displayed image and/or by adding and/or deleting pixels or groups of pixels to/from the displayed image, depending on a disparity map of the displayed image, on the detected change in viewpoint, and on an area of the displayed image that remains unchanged, which is referred to as the attention area, when the displayed image is being modified.

Inventors

• LEGROS, ERIC
• PETITGRAND, Sylvain
• LUONG, BRUNO

Assignees

• Fogale Optique

Dates

Publication Date

20260513

Application Date

20230708

Claims (16)

1. 1. A method of modifying an image of a scene to restore a change in the point of view of the image of the scene, said method comprising the steps of: - display the scene image, - obtain a change of point of view in relation to the displayed image, - modify the displayed image, by moving or spreading part of the pixels or group(s) of pixels in the displayed image and/or by adding and/or deleting pixels or groups of pixels in the displayed image, depending on: - a disparity map of the displayed image, - of the change of point of view obtained, - an area of the displayed image remaining unchanged, called the attention zone, when the displayed image is modified.
2. 2. Method according to claim 1, comprising a step of generating the disparity map of the imaged scene from depth information of the scene of the displayed image and/or from sharpness information of the displayed image.
3. 3. Method according to claim 2, in which the disparity map is generated, from at least two images of the scene coming from the same stationary imaging system and being acquired, each, with a different focal length: - by selection and/or identification of pixels or groups of pixels of the at least two images of the scene or by identification and/or selection of an image or image(s) among the at least two images, - by combination, assembly or association of pixels or groups of pixels of at least two images of the scene.
4. 4. The method of claim 3, wherein the disparity map of the displayed scene is generated by: - according to an alternative, called alternative A: - identification of a local or local maximum or maxima of sharpness within each of the at least two images, and - for each point of the disparity map to be generated, association of the focal length corresponding to the image of the scene presenting the maximum local sharpness, or - according to an alternative, called alternative B: - for each point of the disparity map to be generated, identification of the image of the scene, among the at least two images of the scene, for which the sharpness is maximum, and - for each point of the disparity map to be generated, association of the focal distance corresponding to the image of the scene for which the sharpness is maximum.
5. 5. The method of claim 3, wherein the disparity map of the displayed image is generated by: - for each image of the scene among the at least two images, calculation of a local variance, - for each point of the disparity map to be generated, selection of the image of the scene whose local variance is the highest, - for each point of the disparity map to be generated, association of the focal length corresponding to the selected image.
6. 6. Method according to claim 1 or 2, in which the disparity map of the displayed image is generated, from the image of the displayed scene, by means of a neural network.
7. 7. Method according to one of claims 2 to 6, in which the depth information and/or the sharpness information contained in the image of the displayed scene or, respectively, the depth information contained in the disparity map is refined by processing the image of the displayed scene or, respectively, the disparity map, at the end of the processing phase an image of the displayed scene of which the depth information and/or the sharpness information or, respectively, a disparity map of which the refined depth information is obtained; the processing phase comprises an iterative modification of the displayed scene image or, respectively, of the disparity map so as to minimize a function E comprising: a term D, called a difference term, determined by comparing the scene image or, respectively, of the disparity map convolved by a point spread function (PSF) with the scene image before convolution or, respectively, with the disparity map before convolution, the PSF describing the response of an imaging system from which the displayed scene image is obtained, and a term A, called anomaly term, representative of defects or anomalies within the scene image before convolution or, respectively, of the disparity map. disparity before convolution, determined from the scene image before convolution or, respectively, from the disparity map before convolution.
8. 8. A method according to any preceding claim, wherein: - the given change of viewpoint includes a displacement (du, dv) parallel to a plane of the displayed image, and/or - the given change of point of view includes a displacement (dz) perpendicular to the plane of the image considered.
9. 9. Method according to any one of the preceding claims, comprising, for each pixel or group(s) of pixels of the displayed image, a calculation of a difference between a depth associated with the attention zone and a depth associated with each pixel or group(s) of pixels of the displayed image located outside the attention zone; the modification of the displayed image is carried out as a function of the calculated difference.
10. 10. Method according to any one of the preceding claims, comprising a calculation of an angle corresponding to the change of point of view; the modification of the displayed image is carried out according to the calculated angle.
11. 11. A method according to any preceding claim, wherein: - the change of point of view obtained corresponds to a change of position, in space, of the eyes of an observer, and - the attention zone corresponds to the area of the displayed image on which the observer's gaze is focused.
12. 12. A method according to any preceding claim, wherein the observer's point of view and/or the area of attention is determined from at least one image of the observer's eyes.
13. 13. Method according to any one of the preceding claims, comprising a step of acquiring the displayed image.
14. 14. Data processing device comprising means arranged and/or programmed and/or configured to implement the method according to any one of claims 1 to 13.
15. 15. Computer program comprising instructions which, when the program is executed by a computer, cause the latter to implement the method according to any one of claims 1 to 13.
16. 16. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to implement the method according to any one of claims 1 to 13.

Description

DESCRIPTION TITLE: Method of processing an image of a scene to restore a change in point of view of the scene Technical field The present invention relates to the observation of displayed 2D images. The invention relates to the dynamic observation of displayed 2D images. The invention also aims at the immersive observation of displayed 2D images. The invention does not relate to 3D images or stereoscopic images. State of the prior art In the state of the art, 3D images are known that make it possible to introduce into displayed images a perspective effect such as it would be perceived if the imaged scene were observed directly by the observer. The 3D images described here are not stereoscopic images but rather images allowing a perspective representation of objects. The synthesis of 3D images can be carried out directly from a modeling of a real scene. The production of 3D images is costly in terms of time and computing resources required for processing and storing these large images. The load is increased tenfold when it is a stream of 3D images. Furthermore, the time required for processing is incompatible with dynamic visualization of 2D image(s) acquired in real time. Stereoscopic vision is also known in the state of the art. Binocular human vision, or more broadly animal vision, from two images that are compared by the brain, constitutes the most widespread example. The biological processing of the images obtained is extremely efficient, since it provides in real time a notion of depth of the observed scene, allowing for example to move knowing at what relative distance the observed objects are. The state of the art is known for observing stereoscopic images that is inspired by or mimics the principle of stereoscopic vision. The principle of stereoscopic observation consists in increasing the immersive character of the observation of images by adding a depth effect by projecting a different image onto each eye of an observer. In practice, a stereoscopic image comprises two distinct images of the same scene acquired in two distinct positions in space, in particular by two cameras spaced apart by the pupillary distance. Human stereoscopic vision will allow, from a stereoscopic image, to reconstruct a scene including depth information. As with 3D images, the time required to generate 3D images is incompatible with dynamic visualization of 2D image(s) acquired in real time. The production of stereoscopic images is costly in terms of time and the computing resources required to process and store these images. In addition, to be able to observe stereoscopic images, it is required to wear special glasses or headsets. One aim of the invention is to propose an image processing method enabling: - to address problems with state-of-the-art processes, and/or - to restore a perspective effect to an observer of a 2D image, and/or - to render a parallax effect to an observer of a 2D image, and/or - to restore a change in the point of view of the scene to an observer of a 2D image, and/or - to render a perspective and/or parallax and/or change of point of view effect to an observer solely from 2D images, and/or - to render a perspective and/or parallax and/or change of point of view effect to an observer solely by displaying 2D images. Presentation of the invention For this purpose, a method is proposed for modifying an image of a scene to restore a change in the point of view of the image of the scene. The method comprises the steps of: - display the scene image, - obtain or detect a change of point of view in relation to the displayed image, - modify the displayed image to reflect a change in the viewpoint of the scene image. Preferably, the step of modifying the displayed image comprises: - a displacement or spreading of a part of the pixels or group(s) of pixels of the displayed image, and/or - an addition and/or deletion of pixels or groups of pixels in the displayed image based on: a disparity map of the displayed image, and/or the change of point of view obtained, and/or - an area of the displayed image remaining unchanged, called the attention zone, when the displayed image is modified. Preferably, the image of the scene is displayed by a display means, for example a screen or a projector. Preferably, the detection of the change in viewpoint of the displayed image or the displayed scene is carried out by a detection means, for example an imaging system, for example a camera. It can be understood by point of view, a position or coordinates of space. Preferably, the step of modifying the displayed image is performed by means of a processing unit. Preferably, the change of viewpoint corresponds to the passage from a viewpoint, called the previous viewpoint, to a different viewpoint, called the new viewpoint. Preferably, the image obtained by implementing the method or the image modified according to the method corresponds to the image observed from the new viewpoint. Preferably, each data processing step of the