CN-122018148-A - Multi-light source and light spot matching method for head-mounted vision tracking system

CN122018148ACN 122018148 ACN122018148 ACN 122018148ACN-122018148-A

Abstract

The invention provides a multi-light source and light spot matching method of a head-mounted vision tracking system. The method is applied to VR head-mounted display equipment comprising a camera and N light sources, a convex hull graph to be matched is constructed based on M light spots by acquiring the light spot images comprising the M light spots shot by the camera, a reference convex hull graph set which is constructed by selecting the M light sources from the N light sources and utilizing the corresponding light spots is acquired, the reference convex hull graph with the highest similarity with the convex hull graph to be matched is determined in the set to serve as a target convex hull graph, and then the corresponding relation between the light spots and the light sources is determined. According to the invention, the light spots and the light sources can be matched and converted into the similarity matching of the convex hull patterns, the corresponding relation can be still determined under the condition that the number of the light spots is smaller than that of the light sources, dependence on additional codes or complex driving is reduced, the matching stability and applicability are improved, and reliable input is provided for follow-up eye tracking.

Inventors

CHI JIANNAN
LI FANFAN
XU HAN
JIANG XIN
WANG YIXU
GU QINGFENG
CAO MING

Assignees

北京科技大学
北京科技大学顺德创新学院

Dates

Publication Date: 20260512
Application Date: 20260325
Priority Date: 20260130

Claims (11)

1. A method for matching multiple light sources with light spots of a head-mounted gaze tracking system, the method being applied to a VR head-mounted display device, the device comprising a camera and N light sources, the light sources illuminating behind a user's eyeball, reflected light formed on a cornea surface entering the camera to form a light spot, the method comprising: acquiring a light spot image shot by the camera, wherein the light spot image comprises M light spots, M and N are positive integers, and 1<M is less than or equal to N; Constructing a convex hull graph according to the M light spots to obtain a convex hull graph to be matched; Obtaining a reference convex hull graph set corresponding to the M light spots, wherein M light sources are selected from the N light sources, and a convex hull graph is constructed by utilizing the light spots corresponding to the M light sources to obtain a reference convex hull graph; Determining a reference convex hull graph with highest similarity with the convex hull graph to be matched in the reference convex hull graph set to obtain a target convex hull graph; and determining the corresponding relation between the light spots in the light spot image and the light source according to the target convex hull graph.
2. The method for matching multiple light sources to a light spot of a head-mounted gaze tracking system of claim 1, wherein after the step of obtaining an image of the light spot, prior to the step of obtaining the convex hull pattern to be matched, the method further comprises: determining a pupil center of an eyeball of a user and an optical center of the camera; Connecting the pupil center and the optical center to form a central optical axis; and converting coordinates of the light spots in the light spot image to enable an imaging plane where the converted coordinates of the light spots are located to be perpendicular to the central optical axis.
3. The method for matching multiple light sources and light spots in a head-mounted gaze tracking system according to claim 1, wherein determining a reference convex hull pattern with highest similarity to the convex hull pattern to be matched in the reference convex hull pattern set, and obtaining a target convex hull pattern comprises: respectively calculating the feature vector of each reference convex hull graph in the reference convex hull graph set and the feature vector of the convex hull graph to be matched; Calculating the similarity of the reference convex hull graph and the convex hull graph to be matched according to the feature vector of the reference convex hull graph and the feature vector of the convex hull graph to be matched; and in the reference convex hull graph set, determining a reference convex hull graph corresponding to the maximum similarity as the target convex hull graph.
4. The method for matching multiple light sources to light spots of a head-mounted gaze tracking system according to claim 3, wherein when M >2, and the M light spots are all located at the convex hull boundary and are used as convex hull vertices, the convex hull pattern is a convex polygon formed by using the M light spots as vertices, and the step of calculating the feature vector of the convex hull pattern comprises: Sequencing each vertex of the convex hull graph according to a preset sequencing rule to obtain a vertex sequence; Calculating a convex hull graph inner angle vector (a 1 ,a 2 ,a 3 ,......,a M ) and a convex hull graph side length vector (d 1 ,d 2 ,d 3 ,......,d M ) according to the order of the vertexes in the vertex sequence; Calculating an interior angle ratio feature vector (a 1 /a 2 ,a 2 /a 3 ,......,a M-1 /a M ) from the convex hull graph interior angle vector; And calculating a side length ratio feature vector (d 1 /d 2 ,d 2 /d 3 ,......,d M-1 /d M ) according to the side length vector of the convex hull graph, wherein the interior angle ratio feature vector and the side length ratio feature vector are feature vectors of the convex hull graph.
5. The method for matching multiple light sources to light spots of a head-mounted gaze tracking system of claim 4, wherein the step of calculating the similarity of the reference convex hull pattern to the convex hull pattern to be matched comprises: performing dot product calculation on the interior angle ratio feature vector of the convex hull graph to be matched and the interior angle ratio feature vector of the reference convex hull graph to obtain a first dot product value; Performing dot product calculation on the side length ratio feature vector of the convex hull graph to be matched and the side length ratio feature vector of the reference convex hull graph to obtain a second dot product value; and calculating the similarity according to the first dot product value and the second dot product value, wherein the larger the first dot product value and the second dot product value are, the larger the similarity is.
6. The method for matching multiple light sources to a light spot of a head-mounted gaze tracking system of claim 3, When m=2, the convex hull pattern is a line segment connecting two light spots; The reference convex hull graph is constructed by light spots corresponding to two adjacent light sources; The reference convex hull graph set comprises a first set, a second set and a third set, wherein in the horizontal direction, a light source corresponding to the first set is positioned on a first side of the camera, a light source corresponding to the second set is positioned on a second side of the camera, and a light source corresponding to the third set is positioned on two sides of the camera; If the two light spots are positioned on the same side of the pupil center of the eyeball of the user in the horizontal direction, the slope of the line segment is the feature vector of the convex hull graph, and in the first set or the second set, the reference convex hull graph corresponding to the maximum similarity is determined to be the target convex hull graph; and if the two light spots are positioned on two sides of the pupil center in the horizontal direction, determining that the position relationship between the midpoint of the line segment and the pupil center in the vertical direction is the characteristic vector of the convex hull graph, and determining that the reference convex hull graph corresponding to the maximum similarity is the target convex hull graph in the third set.
7. The method for matching multiple light sources to light spots of a head-mounted gaze tracking system of claim 1, wherein the step of obtaining a reference convex hull pattern set corresponding to M light spots comprises: selecting M light sources from the N light sources; Different types of reference light spot images corresponding to the M light sources are obtained, wherein the reference light spot images comprise M light spots corresponding to the M light sources, the reference light spot images are segmented in the horizontal direction to obtain different areas, and pupil centers of eyeballs of a user are respectively located in the different areas in the different types of reference light spot images; Constructing a convex hull pattern by utilizing the M light spots in the reference light spot image aiming at each type of the reference light spot image to obtain a type of reference convex hull pattern, and And forming the reference convex hull graph set by selecting all combinations of M light sources from the N light sources to obtain various reference convex hull graphs.
8. The method for matching multiple light sources and light spots of a head-mounted gaze tracking system according to claim 7, wherein the reference light spot images are equally divided in a horizontal direction to obtain three different areas, the three types of reference light spot images corresponding to the M light sources, the reference convex hull pattern set comprises three reference convex hull pattern subsets corresponding to the three types of reference light spot images, and the step of determining the reference convex hull pattern with the highest similarity to the convex hull pattern to be matched in the reference convex hull pattern set comprises: Dividing the facula image into three areas along the horizontal direction; selecting a reference convex hull graph subset from the reference convex hull graph set according to the area of the pupil center of the eyeball of the user in the facula image; And determining the reference convex hull graph with the highest similarity with the convex hull graph to be matched in the reference convex hull graph subset.
9. The method for matching multiple light sources and light spots of a head-mounted gaze tracking system according to claim 1, wherein the N light sources include a first light source group and a second light source group located at two sides of the camera, and determining a reference convex hull pattern with highest similarity to the convex hull pattern to be matched in the reference convex hull pattern set, and the step of obtaining a target convex hull pattern includes: sequencing the M light spots from small to large according to an X coordinate to obtain a light spot sequence, wherein N is more than 3 and M is more than or equal to 3; sequentially calculating the difference value of the X coordinates of two adjacent light spots in the light spot sequence to obtain a difference value sequence; When the maximum difference value in the difference value sequence is greater than 2 times of at least one adjacent difference value, determining that a light spot boundary line is formed between two light spots corresponding to the maximum difference value, wherein the light spot boundary line divides the light spot sequence into a first light spot group and a second light spot group, a light source corresponding to the first light spot group is positioned in the first light source group, and a light source corresponding to the second light spot group is positioned in the second light source group; screening a reference convex hull graph corresponding to the light source group and the light spot group divided by the light spot dividing line from the reference convex hull graph set; and determining a reference convex hull pattern with the highest similarity with the convex hull pattern to be matched in the screened reference convex hull patterns to obtain a target convex hull pattern.
10. A device for matching multiple light sources with light spots of a head-mounted gaze tracking system, the device being applied to a VR head-mounted display device, the device comprising a camera and N light sources, the light sources illuminating behind a user's eyeball, reflected light formed on the surface of the cornea entering the camera to form a light spot, the device comprising: the first acquisition module is used for acquiring a light spot image shot by the camera, wherein the light spot image comprises M light spots, M and N are positive integers, and 1<M is less than or equal to N; The construction module is used for constructing a convex hull graph according to the M light spots to obtain a convex hull graph to be matched; the second acquisition module is used for acquiring a reference convex hull graph set corresponding to the M light spots, wherein M light sources are selected from the N light sources, and the convex hull graph is constructed by utilizing the light spots corresponding to the M light sources to obtain the reference convex hull graph; The matching module is used for determining a reference convex hull graph with highest similarity with the convex hull graph to be matched in the reference convex hull graph set to obtain a target convex hull graph; And the determining module is used for determining the corresponding relation between the light spots in the light spot image and the light source according to the target convex hull graph.
11. A VR head mounted display device, comprising: the device comprises a camera and N light sources, wherein the light sources irradiate behind the eyeballs of a user, and reflected light formed on the surface of the cornea enters the camera to form light spots; a controller for performing the method of matching a light source to a spot of light of any one of claims 1 to 9.

Description

Multi-light source and light spot matching method for head-mounted vision tracking system The application claims priority of China patent application with the application number 202610135912.2 and the application name of ' VR head-mounted display device, matching method and device of light source and light spot ', which is submitted to the national intellectual property office in 2026, 1 month and 30 '. Technical Field The invention relates to the technical field of data processing, in particular to a multi-light source and light spot matching method of a head-mounted vision tracking system. Background With the rapid development of Virtual Reality (VR) head-mounted display devices, eye movement-based gaze tracking technology has extremely high application value in the fields of interactive operation, gaze point rendering, fatigue detection, user behavior analysis and the like. In order to realize high-precision eye tracking, a near-eye infrared imaging mode is generally adopted in the industry at present, namely, a plurality of infrared LED light sources are arranged around an infrared camera, and three-dimensional gestures and sight line directions of eyeballs are obtained by capturing highlight light spots formed by cornea reflection on the surfaces of the eyeballs and pupil edge information and combining imaging geometric relations. In a typical VR head mounted display device, an infrared camera is mounted near the lens, typically with several infrared light sources regularly or annularly disposed therearound, such as 4, 6, 8 or more. The light generated by the infrared light source irradiates behind the eyeball of the user, forms specular reflection on the outer surface of the cornea, and forms a light spot after the reflected light enters the camera. In theory, each LED light source should form a corresponding light spot, and the corresponding relation between the light spot and the light source has a decisive effect on the solution of the cornea center, which is a key link in the whole eye movement tracking link. The inventor researches and discovers that in the existing eye tracking technology, in order to realize the corresponding relation between an infrared light source and a cornea reflection light spot, a technical route of light source coding is generally adopted. According to the scheme, different light sources form distinguishable light spot characteristics in imaging by applying different light emitting modes to the light sources, so that matching is completed in subsequent image processing. The existing light source coding modes mainly comprise, but are not limited to, brightness difference coding, flicker frequency coding, time sequence pulse coding, duty ratio coding and light source structure shape coding, and the coding modes can be used independently or can be combined arbitrarily according to hardware conditions. For example, some systems only distinguish the light sources by brightness differences, other systems identify the light source numbers by strobing or time sequence patterns, and still other schemes use brightness differences in combination with pulse time sequences to improve identification reliability. Because the light source codes are directly embedded into the gray value, shape or time sequence of the light spots, the light source codes do not need to rely on complex geometric model solving, and the light spots can be directly matched on the premise that the light source works normally and the light spots are complete. However, the inventor further researches and discovers that firstly, the technical scheme of the light source coding depends on the correct work of coding hardware, whether brightness difference, stroboscopic frequency, time sequence pulse or combination of multiple codes is adopted, an additional light source driving circuit and synchronous control logic are needed, so that the system structure is complex, the power consumption is increased, the volume and heating design of a head display are limited, secondly, coding characteristics are influenced by the factors of eyeball rotation, reflection angle change, camera exposure, light spot shielding, lens reflection, environmental infrared noise and the like, the stability of the light spot characteristics cannot be completely guaranteed no matter single code or combined codes are adopted, and thirdly, once a certain coding signal is destroyed, the whole failure of a light spot matching link can be possibly caused. In addition, most coding schemes need to count the on-off or intensity change of light spots in multi-frame images, so that the stability is obviously reduced when the number of the light spots is reduced or the fast eye movement and the low exposure time are reduced, and the requirements of VR scenes on instantaneity and robustness are difficult to meet. Therefore, it is difficult to maintain reliable spot matching performance in a scene where the number of spots is not fixed, noise interference i