KR-20260068111-A - CREATING SHOCKWAVES IN THREE-DIMENSIONAL DEPTH VIDEOS AND IMAGES

KR20260068111AKR 20260068111 AKR20260068111 AKR 20260068111AKR-20260068111-A

Abstract

A virtual shockwave generation system comprises an eyewear device including a frame, a temple connected to a lateral side of the frame, and a depth-capture camera. Execution of programming by a processor configures the virtual shockwave generation system to generate individual shockwave images by applying a transformation function to initial 3D coordinates for each of a plurality of initial depth images. The virtual shockwave generation system generates a warped shockwave video comprising a sequence of generated warped shockwave images. The virtual shockwave generation system presents the warped shockwave video through an image display.

Inventors

카츠, 사기
자크, 에얄

Assignees

스냅 인코포레이티드

Dates

Publication Date: 20260513
Application Date: 20190903
Priority Date: 20180917

Claims (15)

As a virtual shockwave generation system, Depth capture camera configured to detect initial video; Image display; A user input device that receives a shockwave effect selection from a user; It includes a processor coupled to the depth capture camera, the image display, and the user input device, and The above processor is: A sequence of initial depth images is generated from individual initial images within the initial video, wherein each of the initial depth images is associated with a time coordinate on the time (T) axis for a presentation time based on the individual initial images within the initial video, and each of the initial depth images is formed as a matrix of vertices, wherein each vertex represents a position within an individual 3D scene, and each vertex has a position attribute, wherein the position attribute of each vertex is based on a 3D position coordinate system and includes an X position coordinate on the X-axis for a horizontal position, a Y position coordinate on the Y-axis for a vertical position, and a Z position coordinate on the Z-axis for a depth position. In response to receiving the shockwave effect selection, at least, for each of the initial depth images, a warped shockwave image is generated by applying a transformation function to the vertices of each individual initial depth image based on the Y and Z position coordinates and the associated time coordinates based on the associated time coordinates of each of the initial depth images, and for each of the initial depth images, the individual shockwave depth image is generated by applying the transformation function to the individual initial depth image based on an individual correlation matrix for the vertices of the individual initial depth image that determines the influence weight of the transformation function for each of the vertices in the individual shockwave region of the vertices of the individual initial depth image, wherein the influence weight is based at least on the vertical position of the vertex. Generate a warped shockwave video including a sequence of generated warped shockwave images; and Configured to present the warped shockwave video through the above image display, Virtual Shockwave Generation System.
In Article 1, The above transformation function transforms individual shockwave regions of vertices grouped together along the Z-axis based on the associated time coordinates of individual initial depth images; and The above transformation function moves the individual Y position coordinates of the vertices vertically upward or downward on the Y-axis within the individual shockwave regions of the vertices. Virtual Shockwave Generation System.
In Article 2, The transformation function for each initial depth pixel moves the individual Y position coordinates of each vertex vertically upward or downward on the Y-axis in the individual shockwave regions of the vertices to vertically vary or oscillate the individual shockwave regions of the vertices; and For each of the above initial depth images, generating the individual shockwave depth image by applying an individual transformation function to the individual initial depth image comprises vertically varying or oscillating the individual shockwave regions of the vertices, and storing the individual initial depth image having the vertical variations or oscillations as the individual shockwave depth image. Virtual Shockwave Generation System.
In Article 1, The warped shockwave video comprising the sequence of the generated warped shockwave images has the appearance of a wavefront that propagates radially from the object emitting the shockwave or along the Z-axis of the warped shockwave images of the warped shockwave video. Virtual Shockwave Generation System.
In Article 1, The above transformation function moves the individual Y position coordinates of the vertices vertically upward or downward in the individual shockwave regions of the vertices based on the wave pattern; and The above wave pattern provides the appearance of a wavefront propagating radially from the depth capture camera, radially from the object emitting the shockwave, or along the Z-axis of the warped shockwave images of the warped shockwave video. Virtual Shockwave Generation System.
As a virtual shockwave generation system, Depth capture camera configured to detect initial video; Image display; A user input device that receives a shockwave effect selection from a user; It includes a processor coupled to the depth capture camera, the image display, and the user input device, and The above processor is: A sequence of initial depth images is generated from individual initial images within the initial video, wherein each of the initial depth images is associated with a time coordinate on the time (T) axis with respect to the presentation time based on the individual initial images within the initial video, and each of the initial depth images is formed as a matrix of vertices, each vertex representing a position within an individual 3D scene. In response to receiving the above shockwave effect selection, at least, for each of the initial depth images, based on the associated time coordinates of each of the initial depth images, individual warped shockwave images are generated by applying a transformation function to the vertices of each individual initial depth image; Generate a warped shockwave video including a sequence of generated warped shockwave images; Presenting the warped shockwave video through the above image display; and It is configured to compute individual correlation matrices for the vertices of the individual initial depth images, which determine the influence weight of the transformation function for each of the vertices in the individual shockwave regions of the vertices of the individual initial depth images, and The above influence weight is based at least on the vertical position of the above vertex; and The processor generates individual shockwave depth images by applying the transformation function to each individual initial depth image, for each of the initial depth images, additionally based on the computed individual relevance matrix. Virtual Shockwave Generation System.
A method for generating a virtual shockwave for use with an electronic device including a depth capture camera and an image display, A step of capturing an initial video using the depth capture camera described above; A step of generating a sequence of initial depth images from individual initial images within the initial video ― each of the initial depth images is associated with a time coordinate on the time (T) axis for a presentation time based on the individual initial images within the initial video, and each of the initial depth images is formed as a matrix of vertices, each vertex represents a position within an individual 3D scene, each vertex has a position attribute, and the position attribute of each vertex is based on a 3D position coordinate system and includes an X position coordinate on the X-axis for a horizontal position, a Y position coordinate on the Y-axis for a vertical position, and a Z position coordinate on the Z-axis for a depth position ―; In response to receiving a shockwave effect selection, the step of generating individual warped shockwave images by applying a transformation function to the vertices of each individual initial depth image based on at least the Y and Z position coordinates and the associated time coordinates for each of the initial depth images, wherein for each of the initial depth images, generating individual shockwave depth images comprises applying the transformation function to the individual initial depth images based additionally on an individual correlation matrix for the vertices of the individual initial depth images that determines the influence weight of the transformation function for each of the vertices in the individual shockwave region of the vertices of the individual initial depth images, wherein the influence weight is based at least on the vertical position of the vertices. A step of generating a warped shockwave video including a sequence of generated warped shockwave images; and A method comprising the step of presenting the warped shockwave video through the image display above. Method for generating a virtual shockwave.
In Article 7, The above transformation function transforms individual shockwave regions of vertices grouped together along the Z-axis based on the associated time coordinates of individual initial depth images; and The above transformation function moves the individual Y position coordinates of the vertices vertically upward or downward on the Y-axis within the individual shockwave regions of the vertices. Method for generating a virtual shockwave.
In Article 8, The transformation function for each initial depth pixel moves the individual Y position coordinates of each vertex vertically upward or downward on the Y-axis in the individual shockwave regions of the vertices to vertically vary or oscillate the individual shockwave regions of the vertices; and For each of the above initial depth images, generating the individual shockwave depth image by applying an individual transformation function to the individual initial depth image comprises vertically varying or oscillating the individual shockwave regions of the vertices, and storing the individual initial depth image having the vertical variations or oscillations as the individual shockwave depth image. Method for generating a virtual shockwave.
In Article 7, Presenting the warped shockwave video comprising the sequence of the warped shockwave images generated above presents the appearance of a wavefront propagating radially from the object emitting the shockwave, or along the Z-axis of the warped shockwave images of the warped shockwave video. Method for generating a virtual shockwave.
In Article 7, The above transformation function moves the individual Y position coordinates of the vertices vertically upward or downward in the individual shockwave regions of the vertices based on the wave pattern; and The above wave pattern provides the appearance of a wavefront propagating radially from the depth capture camera, radially from the object emitting the shockwave, or along the Z-axis of the warped shockwave images of the warped shockwave video. Method for generating a virtual shockwave.
As a non-transient computer-readable medium storing program code for execution on an electronic device including a depth capture camera, an image display, and an electronic processor, When the above program code is executed, the electronic device: A step of capturing an initial video through the depth capture camera described above; A step of generating a sequence of initial depth images from individual initial images within the initial video ― each of the initial depth images is associated with a time coordinate on the time (T) axis for a presentation time based on the individual initial images within the initial video, and each of the initial depth images is formed as a matrix of vertices, each vertex represents a position within an individual 3D scene, each vertex has a position attribute, and the position attribute of each vertex is based on a 3D position coordinate system and includes an X position coordinate on the X-axis for a horizontal position, a Y position coordinate on the Y-axis for a vertical position, and a Z position coordinate on the Z-axis for depth ―; In response to receiving a shockwave effect selection, the step of generating individual warped shockwave images by applying a transformation function to the vertices of each individual initial depth image for each of the initial depth images, based on at least the associated time coordinates of each of the initial depth images — generating individual shockwave depth images for each of the initial depth images comprises applying the transformation function to the individual initial depth images based further on an individual correlation matrix for the vertices of the individual initial depth images, which determines the influence weight of the transformation function for each of the vertices in the individual shockwave region of the vertices of the individual initial depth images, wherein the influence weight is based at least on the vertical position of the vertices —; A step of generating a warped shockwave video comprising a sequence of generated warped shockwave images based at least on the above Y and Z position coordinates and the above associated time coordinates; and A step of presenting the warped shockwave video through the above image display Operating to cause to perform, Non-transient computer-readable media.
In Article 12, The above transformation function transforms individual shockwave regions of vertices grouped together along the Z-axis based on the associated time coordinates of individual initial depth images; and The above transformation function moves the individual Y position coordinates of the vertices vertically upward or downward on the Y-axis within the individual shockwave regions of the vertices. Non-transient computer-readable media.
In Article 13, The transformation function for each initial depth pixel moves the individual Y position coordinates of each vertex vertically upward or downward on the Y-axis in the individual shockwave regions of the vertices to vertically vary or oscillate the individual shockwave regions of the vertices; and For each of the above initial depth images, generating the individual shockwave depth image by applying an individual transformation function to the individual initial depth image comprises vertically varying or oscillating the individual shockwave regions of the vertices, and storing the individual initial depth image having the vertical variations or oscillations as the individual shockwave depth image. Non-transient computer-readable media.
In Article 12, Presenting the warped shockwave video comprising the sequence of the warped shockwave images generated above presents the appearance of a wavefront propagating radially from the object emitting the shockwave, or along the Z-axis of the warped shockwave images of the warped shockwave video. Non-transient computer-readable media.

Description

Creating Shockwaves in Three-Dimensional Depth Videos and Images [0001] This application claims priority to U.S. provisional application serial number 62/732,270, filed September 17, 2018, titled Creating Shockwaves in Three-Dimensional Depth Videos and Images, the contents of which are incorporated herein by reference. [0002] The subject of this article is to wearable devices, e.g., eyewear devices, and mobile devices and technologies that allow a user to generate shockwaves in a three-dimensional space of videos and images. [0003] Currently available portable eyewear devices (e.g., glasses, headwear, and headgear); computing devices, including mobile devices (e.g., tablets, smartphones, and laptops) and personal computers, such as wearable devices, integrate image displays and cameras. Currently, users of computing devices can utilize photo lenses or filters to create effects on two-dimensional photos. Various photo editing applications feature tools such as stickers, emojis, and captions for editing two-dimensional photos. [0004] With the advent of three-dimensional (3D) image and video content, more sophisticated manipulations and interactions are required to transform three-dimensional image and video content (e.g., videos, pictures, etc.). For example, it is desirable to be able to manipulate and interact with three-dimensional image and video content to create graphic effects on three-dimensional images and videos. [0005] Therefore, there is a need to improve the video and image editing effects available for 3D image and video content. [0006] The drawings depict one or more implementations only as examples, not as limitations. In the drawings, similar reference numerals indicate identical or similar elements. [0007] FIG. 1a is a right side view of an exemplary hardware configuration of an eyewear device used in a virtual shockwave generation system, wherein a transformation function is applied to the initial depth images of an initial video to generate warped shockwave images for generating a warped shockwave video. [0008] FIG. 1b is a top cross-sectional view of the right chunk of the eyewear device of FIG. 1a showing the right visible light camera and circuit board of the depth-capture camera. [0009] FIG. 1c is a left side view of an exemplary hardware configuration of the eyewear device of FIG. 1a, showing the left visible light camera of the depth-capture camera. [0010] FIG. 1d is a top cross-sectional view of the left chunk of the eyewear device of FIG. 1c showing the left visible light camera and circuit board of the depth-capture camera. [0011] FIG. 2a is a right side view of another exemplary hardware configuration of an eyewear device utilized in a virtual shockwave generation system, illustrating a depth sensor of a depth-capture camera and a right visible light camera to generate an initial depth image of a sequence of initial depth images (e.g., in an initial video). [0012] FIGS. 2b and FIGS. 2c are rear views of exemplary hardware configurations of an eyewear device including two different types of image displays. [0013] FIG. 3 shows a rear view of the eyewear device of FIG. 2a, showing the infrared camera of the depth sensor, the front of the frame, the rear of the frame, and the circuit board. [0014] Fig. 4 is a cross-sectional view taken through the frame of the eyewear device of Fig. 3 and an infrared camera. [0015] FIG. 5 shows a rear perspective view of the eyewear device of FIG. 2a, showing the infrared emitter of the depth sensor, the infrared camera of the depth sensor, the front of the frame, the rear of the frame, and the circuit board. [0016] FIG. 6 is a cross-sectional view taken through the frame and infrared emitter of the eyewear device of FIG. 5. [0017] FIG. 7 illustrates examples of reflected variations of the emitted pattern of infrared light captured by the infrared camera of the depth sensor of the eyewear device to measure the depth of pixels in the raw image to generate initial depth images from the initial video, and the pattern of infrared light emitted by the infrared emitter of the depth sensor. [0018] FIG. 8a illustrates an example of visible light captured by a visible light camera as a raw image and infrared light captured by an infrared camera of a depth sensor as an infrared image to generate an initial depth image of a three-dimensional scene. [0019] FIG. 8b illustrates an example of visible light captured by the right visible light camera as the right raw image and infrared light captured by the left visible light camera as the left raw image to generate an initial depth image of a three-dimensional scene. [0020] FIG. 9 is a high-level functional block diagram of an exemplary virtual shockwave generation system including an eyewear device having a depth-capture camera that generates initial depth images (e.g., in an initial video) and a user input device (e.g., a touch sensor), a mobile device and a server system connected through various networks. [0021