US-12620178-B1 - Supplementing depth information for anchoring

Abstract

According to various implementations, a method includes obtaining pose data that indicates a plurality of poses of the electronic device within a physical environment. The method includes obtaining a representation of a three-dimensional (3D) environment. The representation of the 3D environment includes a plurality of planes. Each of the plurality of planes defines a plurality of points in xy space. The representation of the 3D environment does not include z space (e.g., depth) information. The method includes anchoring the representation of the 3D environment to a physical anchor point within the physical environment based on the pose data. The method includes anchoring a computer-generated object to the representation of the 3D environment based on the pose data. For example, the pose data includes z space (e.g., depth) information, which is used to anchor the computer-generated object to the representation of the 3D environment.

Inventors

• Rudy Poot
• Alvin Chung
• Tomas Alvarez Rodriguez

Assignees

• APPLE INC.

Dates

Publication Date

20260505

Application Date

20230502

Claims (20)

1. 1 . A method comprising: at an electronic device with one or more processors, a non-transitory memory, and a display: obtaining pose data that indicates a plurality of poses of the electronic device within a physical environment; obtaining a representation of a three-dimensional (3D) environment, wherein the representation of the 3D environment includes a first plurality of planes, wherein each of the first plurality of planes defines a plurality of points in xy space, and wherein the representation is a modified version of the 3D environment that lacks z space information; anchoring the representation of the 3D environment to a physical anchor point within the physical environment based on z space information indicated in the pose data; and anchoring a computer-generated object to the representation of the 3D environment based on the pose data.
2. 2 . The method of claim 1 , wherein none of the first plurality of planes defines a point in z space.
3. 3 . The method of claim 2 , wherein the pose data includes 3D positional information regarding the physical environment, wherein the 3D positional information includes a point in the z space, and wherein anchoring the computer-generated object to the representation of the 3D environment is based on the point in the z space.
4. 4 . The method of claim 1 , wherein obtaining the pose data includes generating a 3D map that characterizes the physical environment.
5. 5 . The method of claim 1 , wherein obtaining the pose data includes generating a 3D point cloud that characterizes the physical environment.
6. 6 . The method of claim 1 , wherein anchoring the representation of the 3D environment to the physical anchor point includes: rendering the representation of the 3D environment based on the pose data in order to generate a first portion of display data, and displaying, on the display, the first portion of the display data; and wherein anchoring the computer-generated object to the representation of the 3D environment includes: rendering the computer-generated object based on the pose data in order to generate a second portion of the display data, and displaying, on the display, the second portion of the display data while displaying the first portion of the display data.
7. 7 . The method of claim 1 , wherein the representation of the 3D environment corresponds to a 3D representation of a virtual environment.
8. 8 . The method of claim 1 , wherein the representation of the 3D environment corresponds to a 3D representation of a physical environment.
9. 9 . The method of claim 1 , wherein anchoring the representation of the 3D environment to the physical anchor point is substantially concurrent with anchoring the computer-generated object to the representation of the 3D environment.
10. 10 . The method of claim 1 , wherein the electronic device further comprises an image sensor that captures image data of the physical environment, wherein the electronic device further comprises a positional sensor that generates positional sensor data characterizing a position of the electronic device, and wherein obtaining the pose data includes determining the pose data based on the image data and the positional sensor data.
11. 11 . The method of claim 10 , further comprising: detecting, based on the positional sensor data, a positional change of the electronic device; and in response to detecting the positional change: maintaining anchoring the representation of the 3D environment to the physical anchor point by anchoring a second plurality of planes of the representation of the 3D environment based on the pose data, wherein each of the second plurality of planes defines a plurality of points in the xy space; and maintaining anchoring the computer-generated object to the representation of the 3D environment by anchoring the computer-generated object to the second plurality of planes of the representation of the 3D environment.
12. 12 . The method of claim 10 , wherein determining the pose data includes applying simultaneous localization and mapping (SLAM) to the image data and the positional sensor data.
13. 13 . The method of claim 1 , wherein a corresponding 3D environment defines a plurality of points in xyz space, and wherein the corresponding 3D environment is more graphically complex than the representation of the 3D environment.
14. 14 . The method of 13 , wherein rendering the corresponding 3D environment is associated with a first amount of resource utilization by the electronic device, and wherein rendering the representation of the 3D environment is associated with a second amount of resource utilization by the electronic device that is less than the first amount of resource utilization.
15. 15 . The method of claim 1 , further comprising: detecting a first user input that specifies the physical anchor point, wherein anchoring the representation of the 3D environment to the physical anchor point is in response to detecting the first user input.
16. 16 . The method of claim 15 , further comprising detecting a second user input that requests anchoring the computer-generated object to the representation of the 3D environment, wherein anchoring the computer-generated object to the representation of the 3D environment is in response to detecting the second user input.
17. 17 . An electronic device comprising: a display; a non-transitory memory; and one or more processors to: obtain pose data that indicates a plurality of poses of the electronic device within a physical environment; obtain a representation of a 3D environment, wherein the representation of the 3D environment includes a first plurality of planes, wherein each of the first plurality of planes defines a plurality of points in xy space, and wherein the representation is a modified version of the 3D environment that lacks z space information; anchor the representation of the 3D environment to a physical anchor point within the physical environment based on z space information indicated in the pose data; and anchor a computer-generated object to the representation of the 3D environment based on the pose data.
18. 18 . The electronic device of claim 17 , wherein none of the first plurality of planes defines a point in z space, wherein the pose data indicates a point in the z space, and wherein anchoring the computer-generated object to the representation of the 3D environment is based on the point in the z space.
19. 19 . The electronic device of claim 17 , wherein the electronic device further comprises an image sensor that captures image data of the physical environment, wherein the electronic device further comprises a positional sensor that generates positional sensor data characterizing a position of the electronic device, and wherein obtaining the pose data includes determining the pose data based on the image data and the positional sensor data.
20. 20 . A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which, when executed by an electronic device including a display, cause the electronic device to: obtain pose data that indicates a plurality of poses of the electronic device within a physical environment; obtain a representation of a 3D environment, wherein the representation of the 3D environment includes a first plurality of planes, wherein each of the first plurality of planes defines a plurality of points in xy space, and wherein the representation is a modified version of the 3D environment that lacks z space information; anchor the representation of the 3D environment to a physical anchor point within the physical environment based on z space information indicated in the pose data; and anchor a computer-generated object to the representation of the 3D environment based on the pose data.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application is claims priority to U.S. Provisional Patent App. No. 63/337,860, filed on May 3, 2022, which is hereby incorporated by reference in its entirety. TECHNICAL FIELD The present disclosure relates to displaying an environment, and in particular anchoring the environment to a physical environment. BACKGROUND Anchoring an environment (e.g., a virtual 3D environment) includes rendering the environment such that the environment appears world-locked to a physical anchor point of a physical environment. Because rendering is a computationally expensive process, certain techniques include rendering a representation of the environment, in order to reduce computational demands associated with the anchoring. For example, the representation of the environment includes less graphical information than the environment. However, because of the lower amount of graphical information, it is challenging to efficiently anchor additional content to the anchored representation of the environment. SUMMARY In accordance with some implementations, a method is performed at an electronic device including one or more processors, a non-transitory memory, and a display. The method includes obtaining pose data that indicates a plurality of poses of the electronic device within a physical environment. The method includes obtaining a representation of a 3D environment. The representation of the 3D environment includes a first plurality of planes. Each of the first plurality of planes defines a plurality of points in xy space. The method includes anchoring the representation of the 3D environment to a physical anchor point within the physical environment based on the pose data. The method includes anchoring a computer-generated object to the representation of the 3D environment based on the pose data. In accordance with some implementations, a method is performed at an electronic device including one or more processors, a non-transitory memory, and a display. The one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors and the one or more programs include instructions for performing or causing performance of the operations of any of the methods described herein. In accordance with some implementations, a non-transitory computer readable storage medium has stored therein instructions which when executed by one or more processors of an electronic device, cause the device to perform or cause performance of the operations of any of the methods described herein. In accordance with some implementations, an electronic device includes means for performing or causing performance of the operations of any of the methods described herein. In accordance with some implementations, an information processing apparatus, for use in an electronic device, includes means for performing or causing performance of the operations of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS For a better understanding of the various described implementations, reference should be made to the Description, below, in conjunction with the following drawings in which like reference numerals refer to corresponding parts throughout the figures. FIG. 1A is an example of a 3D environment. FIG. 1B is an example of a representation of the 3D environment. FIGS. 2A-2H are an example of an electronic device anchoring a computer-generated object to the representation of the 3D environment based on pose data in accordance with some implementations. FIG. 3 is an example of a block diagram of an electronic device in accordance with some implementations. FIG. 4 is a block diagram of an example of a portable multifunction device in accordance with some implementations. FIG. 5 is an example of a flow diagram of a method of anchoring a computer-generated object to a representation of a 3D environment based on pose data in accordance with some implementations. DESCRIPTION In some circumstances, a device anchors an environment to a physical anchor point of a physical (e.g., real-world) environment, across a plurality of poses of the device. For example, in an augmented reality (AR) application, the device renders a 3D virtual house such that the rendered 3D virtual house appears anchored to a physical wall. Because rendering is a computationally expensive process, some techniques include rendering a simplified representation of the environment, in order to reduce computational demands associated with the anchoring. For example, the simplified representation of the environment includes information in two dimensions (e.g., xy space), whereas the environment includes information in three dimensions (e.g., xyz space). As another example, the simplified representation of the environment includes multiple 2D planes (e.g., in xy space), whereas the environment corresponds to a 3D mesh (e.g., in xyz space). In other words, in contrast to the environment, the simplified r