CA-3034384-C - VIRTUAL, AUGMENTED, AND MIXED REALITY SYSTEMS AND METHODS

Abstract

A virtual, augmented, or mixed reality display system includes a display configured to display virtual, augmented, or mixed reality image data, the display including one or more optical components which introduce optical distortions or aberrations to the image data. The system also includes a display controller configured to provide the image data to the display. The display controller includes memory for storing optical distortion correction information, and one or more processing elements to at least partially correct the image data for the optical distortions or aberrations using the optical distortion correction information.

Inventors

• Jose Felix Rodriguez
• Ricardo Martinez Perez

Assignees

• MAGIC LEAP, INC.

Dates

Publication Date

20260505

Application Date

20170822

Priority Date

20160822

Claims (1)

1. We Claim: 1. A virtual, augmented, or mixed reality dispiay system comprising: a display configured to display distortion-corrected image data, the display comprising one or more optical components which introduce optical distortions or aberrations to the distortion-corrected image data, wherein the display comprises a plurality of at least partially distinct optical paths; and a display controller configured to provide the distortion-corrected image data to the display, the display controller comprising: memory for storing optical distortion correction information, wherein the optical distortion correction information comprises separate optical distortion correction information for respective colors of the optical paths of the display, and one or more processing elements to at least partially correct nondistortion- corrected image data for the optical distortions or aberrations using the optical distortion correction information; wherein the display controller is configured to pre-distort the non-distortioncorrected image data to obtain the distortion-corrected image data by determining a distortion-corrected pixel at a first location (x, y) for the distortion-corrected image data based on one or more non-distortion-corrected pixels near a different second location (x', y') in the non-distortion-corrected image data received by the display controller; and wherein the display controller is configured to use the optical distortion correction information to determine the second location (x', y') in the non-distortioncorrected image data. 200 2. The system of claim 1, wherein the optical distortion correction information is used by the display controller to pre-distort the non-distortion-corrected image data in a way that is at least partially inversely related to the optical distortions or aberrations. 3. The system of claim 1, wherein the (x', y') coordinates of the second location are fractional numbers. 4. The system of claim 3, wherein the display controller is configured to determine the distortion-corrected pixel at the first location (x, y) by interpolating between a plurality of non-distortion-corrected pixels surrounding the second location (x', y'). 5. The system of claim 4, wherein the display controller uses bilinear interpolation. 6. The system of claim 1, wherein the optical distortion correction information is stored in the form of one or more lookup tables, and wherein the one or more lookup tables are stored in a compressed format, and wherein the display controller is configured to ~xpand the one or more lookup tables before correcting for the optical distortions or aberrations using the optic•aI distortion correction information. 7. The system of claim 1, wherein the optical distortion correction information further comprises information for performing one or more image warp operations, and wherein the display controller is further configured to perform the one or more image warp operations on the non-distortion-corrected image data. 8. The system of claim 1, wherein the display is configured to display the colors sequentially. 201 9. A method in a virtual, augmented, or mixed reality display system, the method comprising: providing distortion-corrected image data to be shown on a display, the display comprising one or more optical components which introduce optical distortions or aberrations to the distortion-corrected image data, wherein the display comprises a plurality of at least partially distinct optical paths; storing optical distortion correction information, wherein the optical distortion correction information comprises separate optical distortion correction information for respective colors of the optical paths of the display;· at least partially correcting non-distortion-corrected image data for the optical distortions or aberrations using the optical distortion correction information; and .displaying the distortion-corrected image data to the user with the display; wherein the act of at least partially correcting the non-distortion-corrected image data comprises: pre-distorting the non-distortion-corrected image data to obtain the distortion-corrected image data by determining a distortion-corrected pixel at a first location (x, y) for the distortion-corrected image data based on one or more non-distortion-corrected pixels near a different second location (x', y') in the non-distortion-corrected image data; and using the optical distortion correction information to determine the second location (x', y') in the non-distortion-corrected image data. 10. The method of claim 9, further comprising using the optical distortion correction information to pre-distort the non-distortion-corrected image data in a way that is at least partially inversely related to the optical distortions or aberrations. 202 . ' 11. The method of claim 9, wherein the (x', y') coordinates of the second location are fractional numbers. 12. The method of claim 11, further comprising determining the distortioncorrected pixel at the first location (x, y) by interpolating between a plurality of nondistortion- corrected pixels surrounding the second location (x', y'). 13. The method of claim 12, further comprising using bilinear interpolation. 14. The method of claim 9, further comprising: • storing the optical distortion corr.ection information in the form of one or niore lookup tables; and storing the one or more lookup tables in a compressed format, and expanding the one or more lookup tables before correcting for the optical distortions or aberrations using the optical distortion correction information . . 15. The method of claim 9, wherein the optical distortion correction information further comprises information for performing one or more image warp operations, and further comprising performing the one or more image warp operations on the non-distortion-corrected image data. 16. The method of claim 9, wherein the display is configured to display the colors sequentially. 203

Description

VIRTUAL, AUGMENTED, AND MIXED REALITY SYSTEMS AND METHOD~ Cross-Reference to Related Applications [0001] This application claims priority to U.S. Provisional Application Serial Number 62/377,829, filed on August 22, 2016 and entitled "MIXED REALITY SYSTEMS AND METHODS," U.S. Provisional Application Serial Number 62/377,804, filed on August 22, 2016 and entitled "MIXED REALITY SYSTEMS AND METtiODS,'' .an~ U.S. Provisional Application Serial Number 62/462,279, flied on February 22, 2017 and entitled "VIRTUAL, AUGMENTED, AND MIXED REALITY SYSTEMS AND METHODS." The present application is related to U.S. Utility Patent ApplJcation Serial No. 14/555,585 filed on November 27, 2014 and entitled "VIRTUAL AND AUGMENTED REALITY SYSTEMS AND METHODS." Described in the aforementioned patent applications are various embodiments of virtual, augmented, and mixed reality systems and methods. Described her~in are further embodiments of virtual, augmented, and mixed reality systems and methods. Copyright Notice [0002] A portion of the disclosure of this patent document contains 111aterial that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. 1 Q • I Field of the Invention [0003] The present disclosure relates to virtual reality, augmented reality, and mixed reality imaging, visualization, and display systems and methods. Background 5 (0004] Modern computing and display technologies have facilitated the development of virtual reality (''VR"), augmented reality ("AR"), and mixed reality ("MR") systems. VR systems create a simulated environment for a user to experience. This can be done by presenting computer-generated imagery to the user through a head-mounted display. This imagery creates a sensory experience 1 O which immerses the user in the simulated environment. A VR scenario typically involves presentation of only computer-generated imagery rather than also including actual real-world imagery. [0005] AR systems generally supplement a real-world environment with simulated elements. For example, AR systems may provide a user with a view of the 15 surrounding real-world environment via a head-mounted display. However, computer-generated imagery can also be presented on the display to enhance the real-world environment. This computer-generated imagery can include elements which are contextually-related to the real-world environment. Such elements can include simulated text, images, objects, etc. MR systems also introduce simulated 20 objects into a real-world environment, but these objects typically feature a greater degree of interactivity than in AR systems. The simulated elements can often times be interactive in real time. [0006] Figure 1 depicts an example AR/MR scene 1 where a user sees a realworld park setting 6 featuring people, trees, buildings in the background, and a 2 concrete platform 20. In addition to these items, computer-generated imagery is also presented to the user. The computer-generated imagery can include, for example, a robot statue 10 standing upon the real-world platform 20, and a cartoon-like avatar character 2 flying by which seems to be a personification of a bumble bee, even 5 though these elements 2, 10 are not actually present in the real-world environment. [0007] Various optical systems generate images at various depths for displaying VR, AR, or MR scenarios. Some such optical systems are described in U.S. Utility Patent Application Serial No. 14/555,585 filed on November 27, 2014. Other such optical systems for displaying MR experiences are described in U.S. Utility Patent 10 Application Serial No. 14/738,877. (0008] Because the human visual perception system is complex, it is challenging to produce a VR/AR/MR technology that facilitates a comfortable, natural-feeling, rich presentation of virtual image elements amongst other virtual or real-world imagery elements. Improved techniques are needed for processing image data in 15 such systems, including, for example, techniques for providing control data to control how the image data is displayed, techniques for correcting optical distortions in the image data, techniques for displaying and blending image data from many depth planes, and techniques for warping image data based on the head pose of a user. VR/AR/MR technology also has size and portability issues, battery life issues, 20 system over-heating issues, and other system and optical challenges. Improved techniques are needed for addressing these issues, including, for example, overheat cause identification, time domain power management, discrete imaging mode, and 3 • WO 2018/0j9270 PCT/US2017/048069 eye/gaze tracking based rendering modification. The systems and methods described herein are configured to address these and other challenges. (0009] What