US-12625560-B2 - Interaction method and apparatus for light field display, and light field display system

US12625560B2US 12625560 B2US12625560 B2US 12625560B2US-12625560-B2

Abstract

Provided in the present invention are an interaction method and apparatus for a light field display, and a light field display system. The method includes: establishing a virtual object on a three-dimensional interaction interface, which is displayed by a target light field display, so as to map an interaction gesture of a user for the three-dimensional interaction interface; identifying the interaction gesture of the user for the three-dimensional interaction interface, and mapping the interaction gesture to the virtual object, so as to perform, on a target object in the three-dimensional interaction interface, an interaction operation corresponding to the interaction gesture; and adjusting the display content of the three-dimensional interaction interface on the basis of the interaction operation.

Inventors

Xuefeng Wang
Hao Zhang
Jinghua Miao
Qingwen Fan
Lili Chen

Assignees

BOE TECHNOLOGY GROUP CO., LTD.

Dates

Publication Date: 20260512
Application Date: 20230516
Priority Date: 20220622

Claims (17)

1 . An interactive method for light field display, comprising: creating a virtual object on a stereoscopic interactive interface displayed by a target light field display to map a user's interactive gesture on the stereoscopic interactive interface; performing an interactive operation corresponding to the interactive gesture on a target object in the stereoscopic interactive interface, by recognizing the user's interactive gesture for the stereoscopic interactive interface and mapping the interactive gesture to the virtual object; and adjusting a display content of the stereoscopic interactive interface based on the interactive operation, wherein the virtual object comprises a virtual item, and the performing the interactive operation corresponding to the interactive gesture on the target object in the stereoscopic interactive interface, by recognizing the user's interactive gesture for the stereoscopic interactive interface and mapping the interactive gesture to the virtual object comprises: recognizing the user's interactive gesture on the stereoscopic interactive interface based on a convex hull detection algorithm; and mapping the interaction gesture to the virtual item by mapping a first position of a hand to a position of the virtual item based on a type of the interaction gesture, wherein the type of the interaction gesture is associated with the first position of the hand, wherein the type of the interactive gesture comprising an open hand or a clenched fist, the first position comprises a palm center, and mapping the first position of the hand to the position of the virtual item based on the type of the interactive gesture comprises: mapping a coordinate of the palm center to a coordinate of a center point of the virtual item, wherein the light field display comprises an image acquisition device for acquiring the interactive gesture, and an installation position of the image acquisition device is determined according to an optimal viewing distance of the light field display; and when mapping the interactive gesture to the virtual object, a coordinate mapping is performed according to the installation position of the image acquisition device and a corresponding coordinate transformation formula, wherein the light field display comprises a light field display whose optimal viewing distance exceeds a preset distance, and the image acquisition device is installed at a top of the light field display; and when mapping the interactive gesture to the virtual object, the coordinate mapping is performed according to a first coordinate transformation formula; and the first coordinate transformation formula comprises: x n = x 1 y n = h + y 1 * cos ⁢ α - z 1 * sin ⁢ α z n = y 1 * sin ⁢ α + z 1 * cos ⁢ α wherein, (x n , y n , z n ) represent the coordinate of the virtual object, h represents the height of the light field display, (x 1 , y 1 , z 1 ) represent the coordinate of the interactive gesture, and α represents an angle between a right ahead direction of the image acquisition device and a horizontal direction, or, wherein the light field display comprises a light field display whose optimal viewing distance does not exceed a preset distance, and the image acquisition device is installed at a bottom of the light field display: when mapping the interactive gesture to the virtual object, the coordinate mapping is performed according to a second coordinate transformation formula; the second coordinate transformation formula comprises: x n = x 2 y n = y 2 * cos ⁢ β + z 2 * sin ⁢ β z n = - y 2 * sin ⁢ β + z 2 * cos ⁢ β wherein, (x n , y n , z n ) represent the coordinate of the virtual object, (x 2 , y 2 , z 2 ) represent the coordinate of the interactive gesture, and β represents an angle between a right ahead direction of the image acquisition device and a horizontal direction.
2 . The interactive method for light field display according to claim 1 , wherein the type of the interactive gesture comprises a single finger gesture, the first position comprises a fingertip of the single finger, and the mapping first position of the hand to the position of the virtual item based on the type of the interactive gesture comprises: mapping a coordinate of the fingertip of the single finger to a coordinate of a center point of the virtual item.
3 . The interactive method for light field display according to claim 1 , wherein the virtual object comprises a virtual hand model, and the performing the interactive operation corresponding to the interactive gesture on the target object in the stereoscopic interactive interface, by recognizing the user's interactive gesture for the stereoscopic interactive interface and mapping the interactive gesture to the virtual object comprises: recognizing the user's interactive gesture on the stereoscopic interactive interface based on a joint detection algorithm; and mapping the interaction gesture to the virtual hand model by mapping a coordinate of each joint position corresponding to the interactive gesture to a coordinate of each joint position of the virtual hand model.
4 . An interactive method for light field display, comprising: creating a virtual object on a stereoscopic interactive interface displayed by a target light field display to map a user's interactive gesture on the stereoscopic interactive interface; performing an interactive operation corresponding to the interactive gesture on a target object in the stereoscopic interactive interface, by recognizing the user's interactive gesture for the stereoscopic interactive interface and mapping the interactive gesture to the virtual object; and adjusting a display content of the stereoscopic interactive interface based on the interactive operation, wherein the interactive gesture comprises a single finger gesture; and the performing the interactive operation corresponding to the interactive gesture on the target object in the stereoscopic interactive interface comprises: detecting a duration of the single finger gesture touching the target object in the stereoscopic interactive interface; determining whether the duration reaches a preset duration; in a case that the duration reaches a preset duration, determining that the interaction operation corresponding to the single finger gesture is a long press operation; and in a case that the duration does not reach the preset duration, determining that the interaction operation corresponding to the single finger gesture is a click operation, wherein the target object comprises a first-level menu item, the stereoscopic interactive interface comprises at least one first-level menu item, and the at least one first-level menu item is displayed in a first preset area of the stereoscopic interactive interface according to a preset spatial display stack order, and the interactive operation further comprises a grab operation; the adjusting the display content of the stereoscopic interactive interface based on the interactive operation comprises: moving a second first-level menu item based on a grab operation on the second first-level menu item, generating a second second-level menu display area corresponding to the second first-level menu item, and displaying the second second-level menu display area in a second preset area of the stereoscopic interactive interface at the highest spatial display level; wherein the highest spatial display level is a spatial display level closest to the user.
5 . The interactive method for light field display according to claim 4 , wherein the target object comprises a first-level menu item, the stereoscopic interactive interface comprises at least one first-level menu item, and the at least one first-level menu item is displayed in a first preset area of the stereoscopic interactive interface according to a preset spatial display stack order; the adjusting the display content of the stereoscopic interactive interface based on the interactive operation comprises: in response to a click operation on a first first-level menu item, adjusting a spatial display level of the first first-level menu item to a highest spatial display level, and/or adjusting a display color of the first first-level menu item; wherein the highest spatial display level is a spatial display level closest to the user.
6 . The interactive method for light field display according to claim 4 , wherein the adjusting the display content of the stereoscopic interactive interface based on the interactive operation further comprises: moving the second second-level menu display area to a third preset area of the stereoscopic interactive interface to close the second second-level menu display area, based on a grab operation on the second second-level menu display area.
7 . The interactive method for light field display according to claim 4 , wherein the adjusting the display content of the stereoscopic interactive interface based on the interactive operation further comprises: in a case that the second second-level menu display area is displayed in the second preset area of the stereoscopic interactive interface, in response to a click operation on a third first-level menu item, adjusting the spatial display level of the third first-level menu item to the highest spatial display level, and/or adjusting the display color of the third first-level menu item; and based on a grab operation on the third first-level menu item, moving the third first-level menu item and generating a third second-level menu display area corresponding to the third first-level menu item, and displaying the third second-level menu display area in a third preset area of the stereoscopic interactive interface at the highest spatial display level.
8 . The interactive method for light field display according to claim 7 , wherein the adjusting the display content of the stereoscopic interactive interface based on the interactive operation further comprises: in response to the virtual object moving away from the user, hiding the third and second-level menu display areas, and displaying a second-level menu display areas of a next lower level; and in response to the virtual object moving toward the user, displaying the hidden third second-level menu display areas again.
9 . The interactive method for light field display according to claim 4 , wherein the adjusting the display content of the stereoscopic interactive interface based on the interactive operation further comprises: in response to a click operation on any second-level menu item in the second second-level menu display area, displaying an interface corresponding to the second-level menu item in full screen mode; in the full-screen state, in response to a long press operation on the first preset area of the stereoscopic interactive interface, restoring display of the first-level menu items hidden under the interface displayed in the full-screen mode; and in response to a click operation on any first-level menu item, displaying the interface displayed in full screen mode in thumbnail mode.
10 . The interactive method for light field display according to claim 9 , wherein the adjusting the display content of the stereoscopic interactive interface based on the interactive operation further comprises: moving any thumbnail based on a grab operation to change a display position of the thumbnail; or, in response to moving any thumbnail to a third preset area of the stereoscopic interactive interface through a grab operation, closing the thumbnail; or, in response to a long press operation on any thumbnail, generating an operation option for the thumbnail; or, in response to the virtual object penetrating any thumbnail in a direction away from the user, hiding the thumbnail; and in response to the virtual object moving in a direction close to the user to leave the thumbnail, restoring display of the thumbnail.
11 . A non-transitory computer-readable storage medium, wherein a computer program is stored on the non-transitory computer-readable storage medium, and when the computer program is executed by one or more processors, the interactive method for light field display according to claim 1 is implemented.
12 . A light field display, comprising a non-transitory memory and one or more processors, wherein a computer program is stored in the non-transitory memory, and when the computer program is executed by the one or more processors, the interactive method for light field display according to claim 1 is implemented.
13 . A light field display system, comprising: the light field display according to claim 12 ; and an image acquisition device, connected to the light field display and is configured to acquire an interactive gesture.
14 . A non-transitory computer-readable storage medium, wherein a computer program is stored on the non-transitory computer-readable storage medium, and when the computer program is executed by one or more processors, the interactive method for light field display according to claim 4 is implemented.
15 . A light field display, comprising a non-transitory memory and one or more processors, wherein a computer program is stored in the non-transitory memory, and when the computer program is executed by the one or more processors, the interactive method for light field display according to claim 4 is implemented.
16 . A light field display system, comprising: the light field display according to claim 15 ; and an image acquisition device, connected to the light field display and is configured to acquire an interactive gesture.
17 . A light field display system, comprising a light field display and an image acquisition device, connected to the light field display and is configured to acquire an interactive gesture, wherein the light field display comprises a memory and one or more processors, wherein a computer program is stored in the memory, and when the computer program is executed by the one or more processors, an interactive method for light field display is implemented, the method comprising: creating a virtual object on a stereoscopic interactive interface displayed by a target light field display to map a user's interactive gesture on the stereoscopic interactive interface; performing an interactive operation corresponding to the interactive gesture on a target object in the stereoscopic interactive interface, by recognizing the user's interactive gesture for the stereoscopic interactive interface and mapping the interactive gesture to the virtual object; and adjusting a display content of the stereoscopic interactive interface based on the interactive operation, wherein in a case that the light field display is a light field display with an optimal viewing distance exceeding a preset distance, the image acquisition device is installed at a top of the light field display, and a longitudinal field angle of the image acquisition device is set in a range of 90° to 120°, and an angle between a right ahead direction of the image acquisition device and a horizontal direction is set in a range of 45° to 60°; and in a case that the light field display is a light field display whose optimal viewing distance does not exceed a preset distance, the image acquisition device is installed at a bottom of the light field display, the longitudinal field angle of the image acquisition device is set in a range of 60° to 90°, and the angle between the right ahead direction of the image acquisition device and the horizontal direction is set by the following relationship: α = 90 ⁢ ° - fov 2 - arctan ⁢ l h wherein, h represents a height of the light field display, α represents the angle between the right ahead direction of the image acquisition device and the horizontal direction, fov represents the longitudinal field of view of the image acquisition device, and l represents the optimal viewing distance.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS The present application is a national phase application of International Application No. PCT/CN2023/094572, filed on May 16, 2023, which claims priority from the Chinese application No. 202210713570.X, filed on Jun. 22, 2022 and entitled “INTERACTION METHOD AND APPARATUS FOR LIGHT FIELD DISPLAY, AND LIGHT FIELD DISPLAY SYSTEM”, and the entire contents thereof are incorporated herein by reference. TECHNICAL FIELD The present disclosure relates to the field of 3D display technology, and in particular to an interaction method and device for a light field display, and a light field display system. BACKGROUND With the advancement of display technology, people are no longer satisfied with watching 2D content, and 3D light field displays have emerged. It should be noted that, information disclosed in the above background portion is provided only for better understanding of the background of the present disclosure, and thus it may contain information that does not form the prior art known by those ordinary skilled in the art. SUMMARY The purpose of the present disclosure is to provide an interactive solution suitable for a 3D light field display. In a first aspect, an embodiment of the present disclosure provides an interactive method for a light field display, including: creating a virtual object on a stereoscopic interactive interface displayed by a target light field display to map a user's interactive gesture on the stereoscopic interactive interface;performing an interactive operation corresponding to the interactive gesture on a target object in the stereoscopic interactive interface, by recognizing the user's interactive gesture for the stereoscopic interactive interface and mapping the interactive gesture to the virtual object; andadjusting a display content of the stereoscopic interactive interface based on the interactive operation. In some implementations, the virtual objection includes a virtual item, and the performing the interactive operation corresponding to the interactive gesture on the target object in the stereoscopic interactive interface, by recognizing the user's interactive gesture for the stereoscopic interactive interface and mapping the interactive gesture to the virtual object includes: recognizing the user's interactive gesture on the stereoscopic interactive interface based on a convex hull detection algorithm; andmapping the interaction gesture to the virtual item by mapping a first position of a hand to a position of the virtual item based on a type of the interaction gesture, wherein the type of the interaction gesture is associated with the first position of the hand. In some implementations, the type of the interactive gesture including an open hand or a clenched fist, the first position includes a palm center, and mapping the first position of the hand to the position of the virtual item based on the type of the interactive gesture includes: mapping a coordinate of the palm center to a coordinate of a center point of the virtual item. In some implementations, the type of the interactive gesture includes a single finger gesture, the first position includes a fingertip of the single finger, and the mapping first position of the hand to the position of the virtual item based on the type of the interactive gesture includes: mapping a coordinate of the fingertip of the single finger to a coordinate of a center point of the virtual item. In some implementations, the virtual object includes a virtual hand model, and the performing the interactive operation corresponding to the interactive gesture on the target object in the stereoscopic interactive interface, by recognizing the user's interactive gesture for the stereoscopic interactive interface and mapping the interactive gesture to the virtual object includes: recognizing the user's interactive gesture on the stereoscopic interactive interface based on a joint detection algorithm; andmapping the interaction gesture to the virtual hand model by mapping a coordinate of each joint position corresponding to the interactive gesture to a coordinate of each joint position of the virtual hand model. In some implementations, the light field display includes an image acquisition device for acquiring the interactive gesture, and an installation position of the image acquisition device is determined according to an optimal viewing distance of the light field display; and when mapping the interactive gesture to the virtual object, a coordinate mapping is performed according to the installation position of the image acquisition device and a corresponding coordinate transformation formula. In some implementations, the light field display includes a light field display whose optimal viewing distance exceeds a preset distance, and the image acquisition device is installed at a top of the light field display; and when mapping the interactive gesture to the virtual object, the coordinate mapping is performed acco