Search

JP-2026076149-A - Methods and devices for managing user-facing interactions with physical objects.

JP2026076149AJP 2026076149 AJP2026076149 AJP 2026076149AJP-2026076149-A

Abstract

[Problem] To provide a system, method, and device for managing user interface interactions with physical objects. [Solution] A first graphical element associated with a first set of output modalities is displayed in an XR environment. While the first graphical element is displayed, the movement of a physical object is detected. In response to the detection of this movement, if the movement causes the physical object to exceed a distance threshold for the first graphical element among the first graphical elements, the first output modality associated with the first graphical element is selected as the current output modality of the physical object. If the movement causes the physical object to exceed a distance threshold for the second graphical element among the first graphical elements, the second output modality associated with the second graphical element is selected as the current output modality of the physical object. [Selection Diagram] Figure 9A

Inventors

  • リンドマイヤー ウィリアム ディー
  • ジャクソン ベンジャミン ジー
  • コバヤシ トニー
  • ボー ブレントン エー
  • サンドストローム マシュー ジェイ
  • パストラーナ ヴィセンテ イスラエル

Assignees

  • アップル インコーポレイテッド

Dates

Publication Date
20260511
Application Date
20251218
Priority Date
20210727

Claims (20)

  1. It is a method, In a computing system comprising non-temporary memory and one or more processors, and communicatively coupled to a display device and one or more input devices, The display device is used to display a first set of graphical elements associated with a first set of output modalities within an augmented reality (XR) environment. While the first set of graphical elements is being displayed, the first movement of a physical object is detected, In response to detecting the first movement of the physical object, In accordance with the determination that the physical object has exceeded a distance threshold for a first graphical element among the first plurality of graphical elements due to the first movement of the physical object, the first output modality associated with the first graphical element is selected as the current output modality of the physical object. In accordance with the determination that the physical object has exceeded the distance threshold for a second graphical element among the first plurality of graphical elements due to the first movement of the physical object, the second output modality associated with the second graphical element is selected as the current output modality of the physical object. Methods that include...
  2. The method according to claim 1, wherein the first and second output modalities cause different visual changes within the XR environment.
  3. According to the determination that the physical object has exceeded the distance threshold for the first graphical element among the first plurality of graphical elements due to the first movement of the physical object, Maintaining the display of the first graphical element adjacent to the physical object, To stop displaying the remaining first graphical elements that do not include the first graphical element, In accordance with the determination that the physical object has exceeded the distance threshold for the second graphical element among the first plurality of graphical elements due to the first movement of the physical object, Maintaining the display of the second graphical element adjacent to the physical object, To stop displaying the remaining first graphical elements that do not include the second graphical element, The method according to claim 1 or 2, further comprising:
  4. After selecting the first output modality associated with the first graphical element as the current output modality of the physical object, the second movement of the physical object is detected. In response to detecting a second movement of the physical object, the first graphical element is moved based on the second movement of the physical object in order to maintain the display of the first graphical element adjacent to the physical object. The method according to claim 3, further comprising:
  5. After stopping the display of the remaining first set of graphical elements, the system obtains a touch input instruction directed at the physical object. In response to receiving the aforementioned touch input instruction, the first set of graphical elements are redisplayed within the XR environment via the display device. The method according to claim 3 or 4, further comprising:
  6. The process further includes obtaining touch input instructions directed to the physical object before displaying the first set of graphical elements, Displaying the first plurality of graphical elements within the XR environment is done in response to receiving the touch input instruction, The method according to any one of claims 1 to 5, including the method described in any one of claims 1 to 5.
  7. In accordance with the determination that the physical object does not exceed the distance threshold with respect to the first graphical element or the second graphical element as a result of the first movement of the physical object, Maintaining the initial output modality as the current output modality of the physical object, Maintaining the display of the first set of graphical elements, The method according to any one of claims 1 to 6, further comprising:
  8. After selecting the first output modality associated with the first graphical element as the current output modality of the physical object, the subsequent marking input by the physical object is detected. In response to detecting the subsequent marking input, one or more marks are displayed in the XR environment via the display device based on the subsequent marking input and the first output modality. The method according to any one of claims 1 to 7, further comprising:
  9. In response to the detection of the subsequent marking input, Based on the determination that an input associated with how strongly the physical object is pressed against a physical surface corresponds to a first input value, the display device displays one or more marks having a first appearance, which is associated with the parameters of the one or more marks corresponding to the first input value, within the XR environment, based on the subsequent marking input and the first output modality. Based on the determination that the input associated with how strongly the physical object is pressed against the physical surface corresponds to a second input value, the display device displays one or more marks having a second appearance, which is associated with the parameters of the one or more marks corresponding to the second input value, within the XR environment, based on the subsequent marking input and the first output modality. The method according to claim 8, further comprising:
  10. The method according to claim 9, wherein the parameter corresponds to one of the radius, width, thickness, intensity, translucency, opacity, color, or texture of the one or more marks in the XR environment.
  11. In response to the detection of the subsequent marking input, In accordance with the determination that an input associated with how tightly the physical object is being grasped by the user corresponds to a first input value, the display device, based on the subsequent marking input and the first output modality, displays one or more marks having a first appearance, which are associated with the parameters of the one or more marks corresponding to the first input value, within the XR environment. In accordance with the determination that the input associated with how tightly the physical object is being held by the user corresponds to a second input value, the display device, based on the subsequent marking input and the first output modality, displays one or more marks having a second appearance, which is associated with the parameters of the one or more marks corresponding to the second input value, within the XR environment. The method according to claim 8, further comprising:
  12. The method according to claim 11, wherein the parameter corresponds to one of the radius, width, thickness, intensity, translucency, opacity, color, or texture of the one or more marks in the XR environment.
  13. Before displaying the first set of graphical elements, which are functions of the grip pose, the grip pose associated with the current manner in which the physical object is held by the user is obtained. In response to acquiring the aforementioned gripping position, In accordance with the determination that the grip posture corresponds to a first grip posture, the display device displays the first set of graphical elements associated with the first set of output modalities in the XR environment. In accordance with the determination that the grip posture corresponds to a second grip posture different from the first grip posture, the display device displays a second set of graphical elements associated with a second set of output modalities in the XR environment. The method according to any one of claims 1 to 12, further comprising:
  14. After displaying the first set of graphical elements associated with the first set of output modalities within the XR environment, the change in grip posture from the first grip posture to the second grip posture is detected. In response to detecting the change in the grip posture, the display of the first set of graphical elements in the XR environment is replaced with the second set of graphical elements associated with the second set of output modalities in the XR environment. The method according to claim 13, further comprising:
  15. Before displaying the first set of graphical elements, information is obtained indicating whether the first or second end of the physical object is facing outwards. In response to obtaining the information indicating whether the first end or the second end of the physical object is facing outward, In accordance with the determination that the first end of the physical object is facing outward, the display device displays the first set of graphical elements associated with the first set of output modalities in the XR environment. In accordance with the determination that the second end of the physical object is facing outward, the display device displays a second set of graphical elements associated with a second set of output modalities in the XR environment. The method according to any one of claims 1 to 14, further comprising:
  16. After displaying the first set of graphical elements associated with the first set of output modalities within the XR environment, the change from the outward-facing first end of the physical object to the outward-facing second end of the physical object is detected. In response to detecting the change from the outward-facing first end of the physical object to the outward-facing second end of the physical object, the display device displays the second set of graphical elements associated with the second set of output modalities within the XR environment. The method according to claim 15, further comprising:
  17. It is a device, One or more processors, Non-temporary memory and An interface for communicating with a display device and one or more input devices, One or more programs stored in the non-temporary memory, which, when executed by the one or more processors, cause the device to perform the method described in any one of claims 1 to 16, A device equipped with the following features.
  18. A non-temporary memory for storing one or more programs, wherein when the one or more programs are executed by one or more processors of a device having interfaces for communicating with a display device and one or more input devices, the non-temporary memory causes the device to execute the method according to any one of claims 1 to 16.
  19. It is a device, One or more processors, Non-temporary memory and An interface for communicating with a display device and one or more input devices, Means for causing the device to perform the method described in any one of claims 1 to 16, A device equipped with the following features.
  20. It is a method, In a computing system comprising non-temporary memory and one or more processors, and communicatively coupled to a display device and one or more input devices, The display device is used to display a user interface, While the aforementioned user interface is displayed, the system detects marking input by a physical object, In response to the detection of the aforementioned marking input, In accordance with the determination that the marking input is directed toward a physical surface, a mark is displayed in the user interface based on the marking input via the display device, wherein the parameters of the mark displayed based on the marking input are determined based on how strongly the physical object is pressed against the physical surface. In accordance with the determination that the marking input is not directed toward the physical surface, the mark is displayed in the user interface via the display device based on the marking input, wherein the parameters of the mark displayed based on the marking input are determined based on how tightly the physical object is being grasped by the user. Methods that include...

Description

This disclosure generally relates to systems, methods, and methods for interacting with and manipulating user interfaces, and more particularly, for managing user interface-directed interactions with physical objects. Typically, users can interact with user interfaces through various input modalities, such as touch input, voice input, and stylus/peripheral device input. However, the workflow for performing actions within the user interface may remain the same regardless of the input modality. This can lead to missed opportunities to enhance the user experience based on input modality and other factors. This disclosure may have a more detailed description by reference to several exemplary implementations, some of which are shown in the accompanying drawings, as can be understood by those skilled in the art. This is a block diagram of an exemplary operating architecture for several implementation configurations. This is a block diagram of an exemplary controller relating to several implementation configurations. This is a block diagram of an exemplary electronic device relating to several implementation configurations. This is a block diagram of an exemplary control device relating to several implementation configurations. This is a block diagram of the first part of an exemplary content delivery architecture relating to several implementation forms. The following are illustrative data structures related to several implementation configurations. This is a block diagram of the second part of an exemplary content delivery architecture relating to several implementation forms. This shows a sequence of instances for a first content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a first content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a first content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a first content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a first content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a first content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a first content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a first content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a first content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a first content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a first content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a first content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a first content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a first content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a first content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a first content delivery scenario, relating to several implementation forms. This shows a sequence of instances for a second content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a second content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a second content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a second content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a second content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a second content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a second content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a second content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a second content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a second content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a second content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a second content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a second content delivery scenario, relating to several implementation forms.This shows a sequence of instances for a second cont