Search

EP-4407975-B1 - METHOD AND SYSTEM FOR CREATING DIGITALLY AUGMENTED CAMERA IMAGES CONTAINING IMAGE DATA OF A LED WALL

EP4407975B1EP 4407975 B1EP4407975 B1EP 4407975B1EP-4407975-B1

Inventors

  • Baric, Martina

Dates

Publication Date
20260513
Application Date
20230124

Claims (5)

  1. Method, preferably computer-implemented method, for creating digitally augmented camera images based on camera images captured by a camera (2), which contain image data of a LED wall (11) displaying LED wall display data, which is contained in computer-rendered scenery image data, the method comprising: a) rendering scenery image data for a first pose (P 1 ) of the camera (2); b) sending LED wall display data comprised in the scenery image data to the LED wall (11) for display; c) capturing, with the camera (2), a camera image containing image data of the LED wall (11) displaying the LED wall display data at a second pose (P 2 ) of the camera (2); and d) augmenting the camera image with image augmentation data to obtain an augmented camera image, wherein the image augmentation data is derived from the scenery image data, taking into consideration the second pose (P 2 ) of the camera (2), wherein the image augmentation data is derived from the scenery image data by • providing a 3D model of the LED wall (11); • generating an extended 3D model (11a) of the LED wall (11) based on the 3D model of the LED wall (11); • projecting the scenery image data rendered for the first pose (P 1 ) of the camera (2) onto the extended 3D model (11a) of the LED wall to obtain a textured extended 3D model (11b) of the LED wall; • rendering the textured extended 3D model (11b) of the LED wall (11) for the second pose (P 2 ) of the camera (2) to obtain the image augmentation data.
  2. Method according to claim 1, wherein the movement of the camera (2) is predetermined in advance, and wherein the step of rendering scenery image data at the first pose (P 1 ) of the camera (2) takes into account the second pose of the camera (2) for the rendering process.
  3. Digital image processing system (3) for creating digitally augmented camera images with rendered scenery image data that comprises LED wall display data for display on a LED wall (11) and image augmentation data for digitally augmenting camera images containing image data of the LED wall (11), the digital image processing system being communicatively connectable to a camera (2) and comprising: • a processing unit (31) that is configured to • receive information on a first pose (P 1 ) of the camera (2); • communicate information on the first pose (P 1 ) of the camera (2) to a rendering engine for rendering scenery image data for the first pose (P 1 ) of the camera (2); • effect communication of LED wall display data comprised in the scenery image data to a LED wall (11) for display thereon; • receive information on a second pose (P 2 ) of the camera (2); • receive a camera image captured by the camera (2) at the second pose (P 2 ) of the camera (2) that contains image data of the LED wall (11) displaying the LED wall display data; • augment the camera image with image augmentation data to obtain an augmented camera image, wherein the image augmentation data is derived from the scenery image data, taking into consideration the second pose (P 2 ) of the camera (2), • a storage unit (32) for storing data, for storing a 3D model of the LED wall (11) to be provided to the processing unit (31), for storing an extended 3D model (11a) of the LED wall (11) to be provided to the processing unit (31), for storing a textured extended 3D model (11b) of the LED wall (11) to be provided to the processing unit (31), and for storing information on the first pose (P 1 ) of the camera (2) to be provided to the processing unit (31) for creating the image augmentation data; wherein the processing unit (31) is configured to derive image augmentation data from the scenery image data by • generating an extended 3D model (11a) of the LED wall (11) based on the 3D model of the LED wall (11); • projecting the scenery image data rendered for the first pose (P 1 ) of the camera (2) onto the extended 3D model (11a) of the LED wall to obtain a textured extended 3D model (11b) of the LED wall; • rendering the textured extended 3D model (11b) of the LED wall (11) for the second pose (P 2 ) of the camera (2) to obtain the image augmentation data.
  4. Digital image processing system according to claim 3, comprising a communication unit (33) for communicatively coupling the digital image processing system to a LED wall controller (15) and/or the camera (2).
  5. Computer-readable medium containing instructions that, when executed by at least one processor, cause the at least one processor to perform a computer-implemented method according to any of claims 1 to 2.

Description

The present invention relates to a method for creating digitally augmented camera images based on camera images captured by a digital camera which contain image data of a LED wall according to the subject-matter of claim 1, and to a corresponding (processing) system according to the subject-matter of claim 3. Extended Reality (XR) is an umbrella term that covers various technologies that combine real-world camera footage captured by digital cameras with virtual content image data, such as computer-generated virtual graphics. It includes Virtual Reality, Augmented Reality, and Mixed Reality technologies. For the production of said effects, the actual camera footage needs to be combined with rendered virtual graphics or virtual content image data. The virtual content image data needs to be rendered in the correct perspective so that the combination of actual camera footage and the virtual content image data appear consistent. To ensure the correct rendering of virtual content image data, the camera position and/or orientation (referred to as camera pose in the context of this disclosure) need to be determined with great spatial and temporal precision. In XR, the goal is often to combine the real camera footage with the virtual content image data in such a way that the virtual elements blend in seamlessly. This way, the audience watching the content enjoys the added virtual elements, yet feels like everything they are seeing is real. To achieve this goal, the virtual content image data must be rendered to match the real footage as closely as possible. For example, if the real-world camera uses an optical lens that distorts the captured images, then the virtual content image data should also be rendered distorted, imitating the real lens's effect in the captured images. Virtual Reality effects have been achieved using so called green screens. These are areas covered uniformly with a specific shade of green that are placed in the background of a scene to be captured with a camera. Other colors can and have been used, but green is the most common. In the captured images, the specific shade of background color or its hue can be recognized and substituted by other video content. This allows the insertion of virtual content image data, e.g. of scenery image data that represents a background to the actors and objects captured in the scene. The process involves the selection of areas of the captured images based on the prevalence of a specific color or hue present on the background screen (such as a specific tone of green in the case of green screens), which are set transparent, allowing scenery image data to be inserted into the scene. This process is referred to as chroma keying. In the context of the present disclosure, virtual content image data is used synonymously with the expression "scenery image data", referring to computer-generated image data that can be sent to a LED wall for display, or used to (digitally) augment captured camera images. A problem arising with the use of green screens and chroma keying is associated with the color based selection of the image areas that are to be substituted with scenery image data. In a real life scenario, the green background in the camera images does not have a perfectly uniform color tone or hue, due to uneven lighting and color distortions caused by the camera itself. In light of this, it is expedient to apply a certain tolerance in the color selection for chroma keying. The degree of tolerance, however, has to be chosen carefully. If the color or hue selection is overly narrow, not all areas which are to be augmented by (virtual) scenery image data will be selected, so that green artefacts will be visible in the augmented images. An excessive tolerance in the color or hue selection may lead to an inadvertent removal of image content. A further challenge in using green screens is to adjust the lighting so that the actual footage matches the virtual graphics. This is especially difficult when the graphics are changing over time. For example, if a light source in the virtual graphics moves while the actual real-world light source remains static, the resulting footage that combines real and virtual elements looks unnatural. For these reasons, among others, LED walls are preferred over green screens when it comes to producing a high quality virtual studio. LED walls are digital displays consisting of large numbers of individual light emitting diodes (LEDs) arranged in a grid. Extended reality technology with LED walls was recently popularized by Lucasfilm in the hit Star Wars series "The Mandalorian" and since then it has been rapidly gaining popularity. This new XR technology is a modern day replacement for green screen studios. A problem arising from the use of LED walls is associated with the limited dimension of LED walls. LED walls of satisfactory quality are rather expensive. Thus, budget limitations oftentimes dictate the use of smaller LED walls, which do not fil