Search

EP-3483703-B1 - SYSTEMS, METHODS, AND TOOLS FOR SPATIALLY-REGISTERING VIRTUAL CONTENT WITH PHYSICAL ENVIRONMENT IN AUGMENTED REALITY PLATFORMS

EP3483703B1EP 3483703 B1EP3483703 B1EP 3483703B1EP-3483703-B1

Inventors

  • DAVIES, PAUL ROBERT
  • LEE, DAVID
  • EVANS, GABRIEL JOSEPH

Dates

Publication Date
20260506
Application Date
20181106

Claims (14)

  1. A system (100) comprising: an alignment tool (106) having a pointer (204) and a fiducial marker (112), the alignment tool configured to be carried by an operator within a physical workspace (102); and an augmented-reality, AR, imaging device (104) including one or more sensors (406) and one or more processors (410), the one or more processors configured to track the fiducial marker (112) in the physical workspace (102) using the one or more sensors, and determine positional coordinates of the pointer (204) within a physical coordinate system that maps the physical workspace (102) at each of multiple physical reference locations (714, 716, 718) within the physical workspace (102) based on a position and orientation of the fiducial marker (112) that is tracked, each of the multiple physical reference locations (714, 716, 718) being associated with a different corresponding virtual reference point (618, 620, 622) within a virtual model (604), wherein the virtual reference points (618, 620, 622) have different respective positional coordinates within a virtual coordinate system of the virtual model (604), wherein the one or more processors (410) are further configured to generate a transfer function that includes rotation and translation of the virtual coordinate system to register the virtual coordinate system to the physical coordinate system by reducing errors between individual point pairs formed by the positional coordinates of the virtual reference points (618, 620, 622) and the respective positional coordinates of the associated physical reference locations (714, 716, 718), and to display virtual content on a display (606) according to the transfer function such that the virtual content is spatially-registered with the physical workspace (102), wherein the AR imaging device (104) is an optical see-through head-mounted device configured to be worn by the operator that carries the alignment tool (106) within the physical workspace (102), the display (606) being integrated onto the AR imaging device and visible to the operator wearing the AR imaging device, and wherein the physical workspace (102) is an interior space of an aircraft (10), and the virtual content that is displayed on the display (606) is a virtual representation of one or more parts of the aircraft.
  2. The system (100) of any preceding claim, wherein the positional coordinates of the physical reference locations (714, 716, 718) are defined along three mutually-perpendicular axes in the physical coordinate system.
  3. The system (100) of any preceding claim, wherein the one or more processors (410) are configured to generate the physical coordinate system based on sensor data acquired by the one or more sensors (406), the positional coordinates of the pointer (204) at the physical reference locations (714, 716, 718) being defined within the physical coordinate system.
  4. The system (100) of any preceding claim, wherein the alignment tool (106) includes a selection button (220) and a wireless communication circuit (404), the alignment tool configured to wirelessly communicate a data acquisition command signal responsive to the operator pressing the selection button.
  5. The system (100) of claim 4, wherein the one or more processors (410) of the AR imaging device (104) are configured to determine the positional coordinates of the pointer (204) at each of the physical reference locations (714, 716, 718) responsive to receiving the data acquisition command signal from the alignment tool (106).
  6. The system (100) of any preceding claim, wherein the physical reference locations (714, 716, 718) include at least three locations within the physical workspace (102) that are spaced apart from one another by at least two meters.
  7. The system (100) of any preceding claim, wherein the alignment tool (106) further includes a frame (202) and a handle (132) that is attached to and extends from the frame, the handle configured to be held by the operator carrying the alignment tool, the frame including a front side (206) and a rear side (208) that is opposite the front side, the fiducial marker (112) being mounted on the front side of the frame, the pointer (204) of the alignment tool disposed rearward of the rear side of the frame and extending away from the frame to a tip (218) of the pointer at a distal end (216) of the pointer, the tip located at a fixed, predetermined position relative to the fiducial marker such that the AR imaging device (104) determines a position of the tip within the physical workspace (102) based on the position and orientation of the fiducial marker.
  8. A method (500) comprising: tracking (506), using an augmented-reality, AR, imaging device (104), a fiducial marker (112) on an alignment tool (106) carried by an operator within a physical workspace (102); determining (510) positional coordinates of a pointer (204) of the alignment tool within a physical coordinate system that maps the physical workspace (102) at each of multiple physical reference locations (714, 716, 718) within the physical workspace (102), the positional coordinates determined based on a position and orientation of the fiducial marker tracked by the AR imaging device (104), each of the multiple physical reference locations (714, 716, 718) within the physical workspace being associated with a different corresponding virtual reference point (618, 620, 622) within a virtual model (604), wherein the virtual reference points (618, 620, 622) have different respective positional coordinates within a virtual coordinate system of the virtual model (604); generating (516) a transfer function that includes rotation and translation of the virtual coordinate system to register the virtual coordinate system to the physical coordinate system by reducing errors between individual point pairs formed by the positional coordinates of the virtual reference points (618, 620, 622) and the respective positional coordinates of the associated physical reference locations (714, 716, 718); and displaying (518) virtual content on a display (606) according to the transfer function such that the virtual content is spatially-registered with the physical workspace (102), wherein the AR imaging device (104) is an optical see-through head-mounted device configured to be worn by the operator that carries the alignment tool (106) within the physical workspace (102), the display (606) being integrated onto the AR imaging device and visible to the operator wearing the AR imaging device, and wherein the physical workspace (102) is an interior space of an aircraft (10), and the virtual content that is displayed on the display (606) is a virtual representation of one or more parts of the aircraft.
  9. The method of claim 8, wherein the virtual content that is displayed on the display is different from the virtual model (604).
  10. The method of claim 8 or claim 9, further comprising: mapping the physical workspace (102) within the physical coordinate system using the AR imaging device (104) prior to tracking the fiducial marker on the alignment tool.
  11. The method of any of claims 8-10, wherein the positional coordinates of the pointer tip at the physical reference locations (714, 716, 718) are each collected responsive to receiving a selection by the operator.
  12. The method of any of claims 8-11, wherein the physical reference locations (714, 716, 718) include at least three locations within the physical workspace (102).
  13. The method of claim 8, wherein the positional coordinates of the physical reference locations (714, 716, 718) are defined along three mutually-perpendicular axes in the physical coordinate system.
  14. The method of claim 8, further comprising: determining the positional coordinates of the pointer (204) at each of the physical reference locations (714, 716, 718) responsive to receiving a data acquisition command signal from the alignment tool (106).

Description

FIELD Embodiments of the present disclosure generally relate to augmented reality platforms, and, more specifically, to aligning virtual content with physical environments or workspaces, such as an interior space of an aircraft or other vehicles, in augmented reality platforms. BACKGROUND Augmented reality platforms are computer-based systems that superimpose virtual content onto a display showing a live view of a physical, real-world environment to a user, thereby providing a composite view of both the physical environment and the virtual content. The live view may be provided as a video feed on a display or by using translucent, see-through displays or lenses, such that the user is able to see the physical environment through the display. Augmented reality can be useful in many different applications, such as gaming, education, and military. One specific useful application of augmented reality is for providing instructional tasks. For example, the overlaid virtual content may visually guide an operator when performing certain tasks, such as vehicle, computer, or other machine assembly, vehicle, computer, or other machine repairs, medical procedures, furniture assembly, and the like. The virtual content in the composite view typically needs to accurately align with the physical environment in order to provide supportive guidance for the instructional tasks, even as the operator moves within the physical environment. For example, if the virtual content does not accurately align with the physical environment, the guidance provided by the virtual content during performance of the instructional task may be confusing and misleading to the user, and may result in costly errors. One known method for aligning virtual content with the physical, real-world environment in an augmented reality platform requires technical skill of an operator. For example, a user may be required to manually translate and angularly orient a virtual object via the use of a keyboard, touchpad, controller device, mouse, hand gestures, or the like, until the virtual object aligns with a physical monument in the physical environment. Moreover, such manual alignment can be tedious and time-consuming, as well as imprecise and inaccurate because the process relies on the skill of the operator and is prone to human error. European patent application publication No. EP 2 728 548 A2 discusses an automated frame of reference calibration for augmented reality. European patent application publication No. EP 3 095 595 A1 discusses a remote advanced repair guidance. United States patent application publications US 2008/123910 A1 and US 2017/258526 A1 disclose methods and systems in augmented reality for coordinate registration and accuracy evaluation of image guided surgery systems. European patent application publication EP 2 998 764 A1 discloses a method for defining measuring points used in a mobile laser tracker, applicable to buildings, bridges, machines, and vehicles (such as the interior of the railway car), ships, and aircraft. SUMMARY The invention relates to a system and a method as defined by the independent claims. The dependent claims define further embodiments of the invention. The embodiments disclosed herein take these and other factors into consideration. Certain embodiments of the present disclosure provide a system for aligning virtual content with a physical workspace or environment in an augmented reality platform as set forth in the appended claims. The system includes an alignment tool and an augmented-reality (AR) imaging device. The alignment tool has a pointer and a fiducial marker. The alignment tool is configured to be carried by an operator within a physical workspace. The AR imaging device includes one or more sensors and one or more processors. The one or more processors are configured to track the fiducial marker in the physical workspace using the one or more sensors, and determine positional coordinates of the pointer at physical reference locations within the physical workspace based on a position and orientation of the fiducial marker that is tracked. The physical reference locations are associated with different virtual reference points within a virtual model. The one or more processors are further configured to generate a transfer function to fit positional coordinates of the virtual reference points with the positional coordinates of the associated physical reference locations. The one or more processors display virtual content on a display according to the transfer function such that the virtual content is spatially-registered with the physical workspace. Certain embodiments of the present disclosure provide a method for aligning virtual content in an augmented reality platform as set forth in the appended claims. The method includes tracking, using an augmented-reality (AR) imaging device, a fiducial marker on an alignment tool carried by an operator within a physical workspace. The method includes determining p