KR-102960644-B1 - Positioning and tracking method and platform, head-mounted display system and computer-readable storage medium

KR102960644B1KR 102960644 B1KR102960644 B1KR 102960644B1KR-102960644-B1

Abstract

Embodiments of the present invention provide a positioning and tracking method, a positioning and tracking platform, a head-mounted display system, and a computer-readable storage medium. The method comprises acquiring one or more images of odd frames collected by one or more tracking cameras placed on a head-mounted display device during a first exposure period set in advance, and one or more images of even frames collected by one or more tracking cameras placed on a head-mounted display device during a second exposure period set in advance, wherein at least one or more images of even frames include blobs corresponding to a plurality of light-emitting bodies placed on a gamepad, and the gamepad is communication-connected with the head-mounted display device (S201); degrees of freedom (DoF) information of the head-mounted display device is determined according to one or more images of odd frames and posture information of the head-mounted display device (S202); and DoF information of the gamepad is determined according to one or more images of even frames, posture information of the gamepad, and DoF information of the head-mounted display device (S203).

Inventors

우, 타오

Assignees

칭다오 피코 테크놀로지 씨오., 엘티디.

Dates

Publication Date: 20260507
Application Date: 20210824
Priority Date: 20200916

Claims (20)

In positioning-and-tracking methods, A step of acquiring one or more images of odd frames collected during a first exposure period preset by one or more tracking cameras disposed in a head-mounted display device, and one or more images of even frames collected during a second exposure period preset by the one or more tracking cameras disposed in the head-mounted display device; A step of determining degrees of freedom (DoF) information of the head-mounted display device according to one or more images of odd frames and posture information of the head-mounted display device; and The method includes the step of determining the DoF information of the gamepad according to one or more images of an even frame, posture information of the gamepad, and DoF information of the head-mounted display device. The above method is, A step of controlling the light-emitting element of the gamepad to turn on according to a preset lighting period when the one or more images of an even frame are collected by the one or more tracking cameras disposed in the head-mounted display device during the preset second exposure period - the intermediate time of the preset second exposure period corresponds to the intermediate time of the preset lighting period, and the preset lighting period is greater than the preset second exposure period - Includes more, The one or more images of the above even frames include at least a blob corresponding to a plurality of light-emitting bodies disposed on the gamepad, and the gamepad is connected to the head-mounted display device. method.
In paragraph 1, The above preset second exposure period is shorter than the above first exposure period, method.
In paragraph 1, The first exposure period is adaptively adjusted according to the intensity of ambient light. method.
In paragraph 1, At least two tracking cameras are positioned on the head-mounted display device, and The exposure period of each tracking camera has the same center point, method.
In paragraph 4, The step of acquiring one or more images of odd frames collected during a first exposure period preset by one or more tracking cameras disposed in the head-mounted display device and one or more images of even frames collected during a second exposure period preset by the one or more tracking cameras disposed in the head-mounted display device is The step of collecting a plurality of images of odd frames having a preset first exposure period by the at least two tracking cameras, and The method includes the step of collecting a plurality of even frames having a preset second exposure period by the at least two tracking cameras. Each image of the odd frames above corresponds to a respective tracking camera, and Each image in the above even frames corresponds to each tracking camera, method.
In paragraph 1, The step of determining the DoF information of the head-mounted display device according to the one or more images of the odd frames and the posture information of the head-mounted display device is: A step of calculating the DoF of the translational motion of the head-mounted display device using a simultaneous positioning and mapping (SLAM) algorithm based on spatial structure information in one or more images of odd frames to acquire the DoF of the translational motion of the head-mounted display device; A step of obtaining the DoF of the rotational motion of the head-mounted display device according to the attitude information of the head-mounted display device collected by a first inertial measurement unit (IMU) integrated into the head-mounted display device; and A step of determining the DoF information of the head-mounted display device based on the DoF of the translational motion and the DoF of the rotational motion. A method including
In paragraph 1, After the step of determining the DoF information of the head-mounted display device according to the one or more images of the odd frame and the posture information of the head-mounted display device, The above method is, A step of performing pre-integration on the DoF information of the head-mounted display device by referring to the attitude information of the head-mounted display device to obtain high-frequency DoF information of the head-mounted display device. A method that includes more.
In paragraph 1, The step of determining the DoF information of the gamepad according to the one or more images of the even frames, the posture information of the gamepad, and the DoF information of the head-mounted display device is: A step of performing blob detection in one or more images of an even frame to determine position information of all blobs in one or more images of an even frame; A step of determining the three-dimensional coordinates of a light source on the gamepad corresponding to a blob in one or more images of an even frame according to the position information of all blobs in one or more images of the even frame; and A step of determining the DoF information of the gamepad according to the 3D coordinates of the light source of the gamepad corresponding to the blob of one or more images of the even frame, the pose information of the gamepad, and the DoF information of the head-mounted display device. A method including
In paragraph 8, The step of performing blob detection in one or more images of an even frame to determine position information of all blobs in one or more images of an even frame is: A step of performing blob detection on each of the one or more images of an even frame collected by the one or more tracking cameras to determine position information of all blobs in each of the one or more images of the even frame; A step of determining a correspondence between the blobs in the one or more images of the even frame and the light source on the gamepad according to the distribution rule of each blob in the one or more images of the even frame; and It includes the step of determining the 3D coordinates of the corresponding light source on the game pad based on the 2D coordinates of the corresponding light source on the game pad, and All of the above blob position information includes at least four blob position information, method.
In paragraph 8, The step of determining the DoF information of the gamepad according to the 3D coordinates of the light source of the gamepad corresponding to the blob of one or more images of the even frame, the pose information of the gamepad, and the DoF information of the head-mounted display device is: A step of determining the DoF of translational motion according to the three-dimensional coordinates of the light source on the gamepad corresponding to the blob in one or more images of an even frame; A step of determining the DOF of the rotational movement of the gamepad according to the posture information of the gamepad; A step of determining the DoF information of the gamepad according to the DoF of the translational motion and the DoF of the rotational motion; and A step of performing coordinate transformation for the DoF information of the gamepad according to the DoF information of the head-mounted display device A method including
In paragraph 8, After the step of determining the DoF information of the gamepad according to the 3D coordinates of the light source of the gamepad corresponding to the blob of one or more images of the even frame, the pose information of the gamepad, and the DoF information of the head-mounted display device, The above method is, A step of performing pre-integration on the DoF information of the game pad by referring to the posture information of the game pad to obtain high-frequency DoF information of the game pad. A method that includes more.
In paragraph 1, The one or more images of the odd frames include at least a hand or hands, and The above method is, Step of determining DoF information of the hand or hands according to one or more images of the odd frames A method that includes more.
In Paragraph 12, The step of determining the DoF information of the hand or hands according to the one or more images of the odd frames is: A step of inputting one or more images of odd frames into a preset convolutional neural network model to obtain the locations of multiple key points of the hand or the hands; and A step of determining the DoF information of the hand or hands according to the location of a plurality of key points of the hand or hands. A method including
In Paragraph 13, The method further includes the step of training the aforementioned preset convolutional neural network model, and The above method is, A step of acquiring a plurality of scenario images including the above-mentioned hand or hands; A step of marking the locations of multiple key points of the hand or hands in the plurality of scenario images and forming an image training sample set with the plurality of marked scenario images; and Step of training the convolutional neural network model according to the above image training sample set including, method.
In positioning-and-tracking platforms, An image acquisition module configured to acquire one or more images of odd frames collected during a first exposure period preset by one or more tracking cameras disposed in a head-mounted display device, and one or more images of even frames collected during a second exposure period preset by one or more tracking cameras disposed in the head-mounted display device; wherein the one or more images of even frames include at least a blob corresponding to a plurality of light emitters disposed on a gamepad, the gamepad is communicationally connected to the head-mounted display device, and when the one or more images of even frames are collected during the second exposure period preset by one or more tracking cameras disposed in the head-mounted display device, the light emitters of the gamepad are controlled to turn on according to a preset lighting period, wherein the intermediate time of the second exposure period preset corresponds to the intermediate time of the preset lighting period, and the preset lighting period is greater than the second exposure period preset -; and A positioning-and-tracking module configured to determine DoF information of a head-mounted display device according to one or more images of the odd frames and attitude information of the head-mounted display device. Includes, The one or more images of the above even frames include at least a blob corresponding to a plurality of light-emitting bodies placed on a gamepad, and The above gamepad is connected to the head-mounted display device via communication, and The positioning-and-tracking module is further configured to determine the DoF information of the gamepad according to the one or more images of the even frames, the posture information of the gamepad, and the DoF information of the head-mounted display device. Positioning-and-tracking platform.
In positioning-and-tracking platforms, processor; and memory Includes, The above memory stores computer instructions, and When the above computer instruction is executed by the processor, the processor performs a method according to any one of claims 1 to 14, Positioning-and-tracking platform.
In head-mounted display systems, Head-mounted display device; A gamepad connected to the head-mounted display device; and Positioning-and-tracking platform Includes, At least two tracking cameras are positioned on the head-mounted display device, and The above gamepad is provided with a plurality of light-emitting bodies, and When one or more images of even frames are collected by one or more tracking cameras disposed in the head-mounted display device during a preset second exposure period, the light-emitting element of the gamepad is controlled to turn on according to a preset lighting period - the midpoint of the preset second exposure period corresponds to the midpoint of the preset lighting period, and the preset lighting period is greater than the preset second exposure period. Head-mounted display system.
In Paragraph 17, The exposure period of each tracking camera placed in the head-mounted display device has the same center point, Head-mounted display system.
In Paragraph 17, Four tracking cameras are positioned on the head-mounted display device, and The four tracking cameras are positioned on the head-mounted display device according to a preset first position constraint rule, and The above-mentioned preset first position constraint rule is used to ensure that each tracking camera meets the requirements of the head-mounted display device in the field of view (FOV), Head-mounted display system.
In Paragraph 17, A plurality of light-emitting bodies are placed on the gamepad according to a preset second position constraint rule, and The placement according to the above-mentioned preset second position constraint rule is, A condition in which no matching or adhesion of local pattern information occurs in images collected within a preset distance range from the tracking camera to the gamepad, and Condition in which a blob corresponding to at least four light sources is displayed in all images collected by one or more of the tracking cameras at any angle satisfying, Head-mounted display system.

Description

Positioning and tracking method and platform, head-mounted display system and computer-readable storage medium Embodiments of the present invention relate to the field of video processing technology, and in particular to a localization and tracking method, a localization and tracking platform, a head-mounted display system, and a computer-readable storage medium. This application claims priority to Chinese patent application No. 202010974857.9, filed with the Chinese National Intellectual Property Administration (CNIPA) on September 16, 2020, with the title "Localization and Tracking Method and Platform, and Head-Mounted Display System," the entirety of which is incorporated herein by reference. Localization (also known as positioning) technology has been widely applied in fields such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), and is an important part of man-machine interaction. Currently, to implement gamepad tracking, electromagnetic or ultrasonic sensors must generally be placed on the gamepad to achieve positioning and tracking. To implement bare-hand tracking, an infrared camera or depth camera must be added to the head-mounted display device. Consequently, positioning and tracking of the head-mounted display device, gamepad, and bare hands cannot be implemented based on the same camera, leading to issues such as high power consumption, low stability, and high operating costs. Therefore, a positioning and tracking solution is required to realize synchronous tracking between a head-mounted display device and a gamepad. To more clearly explain the technical solution of the embodiments of the present invention, the drawings used to describe the embodiments are briefly introduced below. It should be understood that the drawings below are merely illustrative of some embodiments of the present invention and should not be construed as limiting the scope. A person skilled in the art may obtain other drawings from these drawings without inventive effort. FIG. 1 is a schematic diagram of a hardware configuration that can be applied to implement a head-mounted display system according to some embodiments of the present invention. FIG. 2 is a flowchart of a positioning and tracking method according to some embodiments of the present disclosure. FIG. 3 is a schematic diagram of a gamepad according to some embodiment of the present disclosure; FIG. 4 is a structural block diagram of a positioning and tracking platform according to some embodiments of the present invention. FIG. 5 is a structural block diagram of a positioning and tracking platform according to some embodiments of the present invention. Hereinafter, various embodiments of the present disclosure are described in detail with reference to the attached drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical representation, and numerical values of the components and operations described in these embodiments do not limit the scope of the present invention. The following description of at least one exemplary embodiment is for illustrative purposes only and should not be construed as limiting the present disclosure and its application or use. Techniques, methods, and devices known to a person of ordinary knowledge in the relevant technical field may not be discussed in detail, but where appropriate, techniques, methods, and devices should be considered part of the specification. In all examples described and discussed herein, any specific value should be interpreted merely as exemplary and not as a limitation. Accordingly, other examples of the exemplary embodiments may have different values. However, since similar reference numerals and characters in the following drawings indicate similar items, items defined in one drawing are omitted in the following drawings. <Hardware Configuration> FIG. 1 is a block diagram of the hardware configuration of a head-mounted display system (100) according to some embodiment of the present invention. Referring to FIG. 1, a head-mounted display system (100) includes a head-mounted display device (1000), a gamepad (2000), and a localization and tracking platform (3000). The localization and tracking platform (3000) is connected to the head-mounted display device (1000) and the gamepad (2000) via communication to realize localization and tracking of the head-mounted display device (1000) and the gamepad (2000), respectively. The head-mounted display device (1000) may be, for example, a VR device, an AR device, an MR device, etc. In one embodiment, the head-mounted display device (1000) may include a processor (1100), memory (1200), interface apparatus (1300), communication apparatus (1400), display apparatus (1500), input apparatus (1600), audio apparatus (1700), IMU (1800), one or more tracking cameras (1900), as shown in FIG. 1. The processor (1100) may be, for example, a Central Processing Unit (CPU), a Microprocessor Microprogrammed Control