US-20260129171-A1 - SYSTEMS AND METHODS FOR SPATIAL VIDEO CAPTURE

US20260129171A1US 20260129171 A1US20260129171 A1US 20260129171A1US-20260129171-A1

Abstract

The claims generally describe an apparatus for spatial image and video capture comprising a micro-electromechanical system (MEMS) scanning mirror. The apparatus comprises at least one adaptive MEMS micromirror, a first optical subsystem configured to: receive light from a first portion of an environment to direct light to the at least one adaptive MEMS micromirror; and a second optical subsystem configured to: receive light from a second portion of the environment to direct light to the at least one adaptive MEMS micromirror. The apparatus further comprises control circuitry that positions the MEMS micromirror to direct light from the first optical subsystem and light from the second optical subsystem to the at least one sensor. The apparatus generates image data based at least in part on the received light.

Inventors

Ning Xu
Tao Chen

Assignees

ADEIA IMAGING LLC

Dates

Publication Date: 20260507
Application Date: 20251230

Claims (20)

1 . An apparatus, comprising: at least one adaptive MEMS micromirror; a first optical subsystem configured to: receive light from a first portion of an environment to direct light to the at least one adaptive MEMS micromirror; and a second optical subsystem configured to: receive light from a second portion of the environment to direct light to the at least one adaptive MEMS micromirror; control circuitry configured to: position the MEMS micromirror to direct light from the first optical subsystem to an at least one sensor to generate first sensor data; position the MEMS micromirror to direct light from the second optical subsystem to the at least one sensor to generate second sensor data; and generate image data based at least in part on the first sensor data and the second sensor data.
2 . The apparatus of claim 1 , wherein the first optical subsystem comprises a first telephoto system and the second optical subsystem comprises a second telephoto system.
3 . The apparatus of claim 2 , wherein the first telephoto system comprises a respective focal length, and wherein the respective focal length of the first telephoto system is adjustable by the control circuitry.
4 . The apparatus of claim 2 , wherein the second telephoto system comprises a respective focal length, and wherein the respective focal length of the second telephoto system is adjustable by the control circuitry.
5 . The apparatus of claim 1 , wherein the at least one adaptive MEMS micromirror comprises a plurality of singulated micromirrors.
6 . The apparatus of claim 5 , wherein each pixel on the at least one sensor corresponds to a singulated micromirror of the plurality of singulated micromirrors.
7 . The apparatus of claim 1 , wherein the apparatus is a mobile device.
8 . The apparatus of claim 1 , wherein each the first optical subsystem and second optical subsystem each comprise a respective set of prisms and wherein at least one surface of each the respective set of prisms comprises a metallic coating.
9 . The apparatus of claim 1 , wherein the generating image data based at least in part on the first sensor data and the second sensor data further comprises: determining matching features in the first sensor data and the second sensor data using a feature detection algorithm; and generating the image data based at least in part on the matching features in the first sensor data and the second sensor data.
10 . The apparatus of claim 1 , wherein the image data is a video comprising a plurality of frames and wherein the control circuitry is further configured to: position the MEMS micromirror to direct light from the first optical subsystem to the at least one sensor to generate first sensor data at a first time; position the MEMS micromirror to direct light from the second optical subsystem to the at least one sensor to generate second sensor data at a second time; and generating a plurality of frames by time-multiplexing the first sensor data at the first time and the second sensor data at the second time.
11 . A method performed using a system, wherein the system comprises: at least one adaptive MEMS micromirror; a first optical subsystem configured to: receive light from a first portion of an environment to direct light to the at least one adaptive MEMS micromirror; and a second optical subsystem configured to: receive light from a second portion of the environment to direct light to the at least one adaptive MEMS micromirror; a control circuitry; the method comprising: positioning, using the control circuitry, the MEMS micromirror to direct light from the first optical subsystem to an at least one sensor to generate first sensor data; positioning, using the control circuitry, the MEMS micromirror to direct light from the second optical subsystem to the at least one sensor to generate second sensor data; and generating, using the control circuitry, image data based at least in part on the first sensor data and the second sensor data.
12 . The method of claim 11 , wherein the first optical subsystem comprises a first telephoto system and the second optical subsystem comprises a second telephoto system.
13 . The method of claim 12 , wherein the first telephoto system comprises a respective focal length, and wherein the respective focal length of the first telephoto system is adjustable by a control circuitry of the system.
14 . The method of claim 12 , wherein the second telephoto system comprises a respective focal length, and wherein the respective focal length of the second telephoto system is adjustable by a control circuitry of the system.
15 . The method of claim 11 , wherein the at least one adaptive MEMS micromirror comprises a plurality of singulated micromirrors.
16 . The method of claim 15 , wherein each pixel on the at least one sensor corresponds to a singulated micromirror of the plurality of singulated micromirrors.
17 . The method of claim 11 , wherein the system is a part of a mobile device.
18 . The method of claim 11 , wherein each the first optical subsystem and second optical subsystem each comprise a respective set of prisms and wherein at least one surface of each the respective set of prisms comprises a metallic coating.
19 . The method of claim 11 , wherein the generating image data based at least in part on the first sensor data and the second sensor data further comprises: determining matching features in the first sensor data and the second sensor data using a feature detection algorithm; and generating the image data based at least in part on the matching features in the first sensor data and the second sensor data.
20 . The method of claim 11 , wherein the image data is a video comprising a plurality of frames and wherein the method further comprises: positioning the MEMS micromirror to direct light from the first optical subsystem to the at least one sensor to generate first sensor data at a first time; positioning the MEMS micromirror to direct light from the second optical subsystem to the at least one sensor to generate second sensor data at a second time; and generating a plurality of frames by time-multiplexing the first sensor data at the first time and the second sensor data at the second time.

Description

CROSS-REFERENCE TO RELATED APPLICATION This application is a continuation of U.S. patent application Ser. No. 18/758,411, filed Jun. 28, 2024, the disclosure of which is hereby incorporated by reference herein in its entirety. BACKGROUND This disclosure is related to capturing spatial photos and videos on a mobile device. SUMMARY Some devices, such as smartphones, may be equipped with multiple rear-facing sensors capable of capturing photos and videos. In some approaches, each sensor and corresponding lens serves a distinct purpose, such as standard photography, wide-angle shots, and/or capturing zoomed-in images. As an example, a smartphone with a triple camera setup comprises an ultrawide lens, a wide-angle lens, and a telephoto lens. In this example, the wide-angle lens captures images at a moderately wide angle of view, while the ultra-wide angle of view allows for a much wider field of view, resulting in zoomed-out photos. The telephoto lens allows for higher zoom levels than both the wide-angle lens and the ultra-wide lens and is generally used for close-up photos such as those taken in portrait mode. In some approaches, the multiple rear-facing sensors of a mobile device are used for the capture of three-dimensional images and/or videos that are commonly known as spatial photos and/or videos. For example, a phone may capture spatial photos and/or videos by initiating simultaneous capture from both the wide-angle and ultra-wide sensors while the device is in landscape orientation. However, this approach limits the quality of the resulting spatial photos and/or videos due to differences between the wide and ultra-wide sensors with respect to sensor capabilities. Further, this approach is computationally more difficult, as it requires further processing such as transformations and scaling in order to account for differences in the images captured from each of the sensors. In some approaches, mobile devices attempt to address these issues by scaling the field of view from the ultra-wide sensor to match the field of view from the wide sensor. This approach does not fully solve the problem, though, since the cropping and scaling restrict the picture quality of the resulting spatial photos and/or videos to that of the sensor corresponding to the image with the lower picture quality. In one example, the ultra-wide sensor captures only 12 MP, and cropping the captured images leaves a native resolution of less than 4K, while the wide-angle sensor captures 48 MP and has a much higher resolution than the ultra-wide sensor. Cropping and scaling images captured by both sensors to match each other therefore limits such spatial photos and videos captured by the device to 1080p and 30 frames per second (fps) only, as in common video formats. Additionally, limited distance between the sensors may, in some cases, result in images that lack perceived depth. For example, when in the landscape mode required for capturing spatial photos, the distance between the wide-angle sensor and the ultra-wide sensor is roughly 15 mm, while the average interpupillary distance is 63 mm for humans. The close distance between the two sensors makes the captured spatial photos and/or videos seem to lack perceived depth. For instance, in such cases, the images may lack the depth-of-field variations that contribute to a three-dimensional effect. Furthermore, utilizing two sensors to capture spatial videos also creates issues outside of picture quality, such as increased hardware requirements, increased power consumption, and redundancies related to signal processing pipelines and memory buffers. A pair of identical sensors necessitates duplicate sensors, lenses, and associated circuitry. The operation of the pair of identical sensors is also a significant drain on battery life due to the power requirements of the image sensor, signal processing, and data storage. Because each identical sensor operates independently, each sensor requires separate signal processing pipelines and memory buffers as well, which in turn increases power consumption and demands more from the device's processor, leading to overheating and further battery drain. To overcome these problems, systems, methods, and apparatuses are described herein for generating a spatial photo using an apparatus comprising a micro-electromechanical systems (MEMS) scanning mirror. In some embodiments, the apparatus comprises an enclosure comprising a panel, wherein the panel comprises a first light-permeable element and a second light-permeable element. In some embodiments, the apparatus comprises a light-detecting element disposed within the enclosure and configured to detect light traveling toward an inner surface of the panel of the enclosure, a first mirroring element disposed within the enclosure and configured to reflect light traveling through the first light-permeable element of the panel, and a second mirroring element disposed within the enclosure and configured to reflect light