CN-115700764-B - Control method, tracking system and non-transitory computer readable medium

Abstract

A control method includes the following steps. At least one image relating to a calibration chart is captured with a camera, wherein a first trackable device entity is attached to the camera and a second trackable device entity is attached to the calibration chart. To track a base station to track a first trackable device attached to the camera to generate a first rotation conversion matrix, to track a base station to track a second trackable device attached to the calibration chart to generate a second rotation conversion matrix. A third rotation conversion matrix is generated based on the calibration chart appearing in the at least one image. A fourth rotational transformation matrix between the camera coordinate system and the first trackable device is calculated according to the first rotational transformation matrix, the second rotational transformation matrix, and the third rotational transformation matrix, wherein the fourth rotational transformation matrix is used to track the camera. Therefore, the camera can be accurately tracked and positioned, so that the object shot by the camera is more vivid when combined with the virtual scene.

Inventors

• CHEN XINGHONG
• LI XUFENG

Assignees

• 宏达国际电子股份有限公司

Dates

Publication Date

20260505

Application Date

20220727

Priority Date

20220622

Claims (20)

1. 1. A control method, characterized in that the control method comprises: Capturing at least one image relating to a calibration graphic card with a camera, wherein a first trackable device entity is attached to the camera and a second trackable device entity is attached to the calibration graphic card; tracking the first trackable device and the second trackable device with a tracking base station to generate a first rotation conversion matrix between the first trackable device and the tracking base station and generate a second rotation conversion matrix between the second trackable device and the tracking base station; Generating a camera coordinate system of the camera and a third rotation conversion matrix between the calibration chart according to the calibration chart appearing in the at least one image, and Calculating a fourth rotation transformation matrix between the camera coordinate system and the first trackable device according to the first rotation transformation matrix, the second rotation transformation matrix, the third rotation transformation matrix, and a fifth rotation transformation matrix between the calibration graphic card and the second trackable device entity, wherein the fourth rotation transformation matrix is used for tracking the camera.
2. 2. The control method of claim 1, wherein the calibration chart includes a feature pattern and a load socket, the second trackable device entity is attached to the load socket and has a mechanism configuration relationship with respect to the feature pattern, and the fifth rotation transformation matrix between the calibration chart and the second trackable device entity is derived from the mechanism configuration relationship.
3. 3. The control method of claim 2, wherein the fourth rotation transformation matrix is calculated based on a product of the third rotation transformation matrix, the fifth rotation transformation matrix, the second rotation transformation matrix, and the first rotation transformation matrix.
4. 4. The control method of claim 1, wherein the origin of the camera coordinate system is located at an optical center of the camera, and the fourth rotation transformation matrix is used for describing a rotation relationship and a position relationship between the camera coordinate system and the first trackable device.
5. 5. The control method of claim 1, further comprising: capturing N images related to the calibration chart by the camera, wherein N is a positive integer greater than 1, and According to the calibration chart card in the N images, a camera geometric calibration is performed to generate a plurality of internal parameters and a plurality of deformation parameters.
6. 6. The control method of claim 5, wherein the plurality of internal parameters are related to coordinate system conversion between a two-dimensional pixel coordinate system corresponding to one of the N images and the camera coordinate system, the plurality of internal parameters are affected by a focal length, an optical center and a skew factor of the camera, the plurality of internal parameters are stored, and the plurality of stored internal parameters are used to adjust an image frame of the camera when the camera captures another image that does not involve the calibration card.
7. 7. The control method of claim 5, wherein the plurality of deformation parameters are associated with a plurality of non-linear lens deformations of the camera, the plurality of deformation parameters are stored, and when the camera captures another image that does not involve the calibration chart, the plurality of deformation parameters stored are used to adjust an image frame of the camera.
8. 8. The control method of claim 1, further comprising: capturing N images related to the calibration chart by the camera, wherein N is a positive integer greater than 1; generating N third rotation conversion matrixes between the camera coordinate system of the camera and the calibration chart according to the calibration chart appearing in the N images; calculating N candidate rotation transformation matrices between the camera coordinate system and the first trackable device according to the first rotation transformation matrix, the second rotation transformation matrix, the N third rotation transformation matrices, and the fifth rotation transformation matrix; Statistically analyzing the N candidate rotational transfer matrices, and And calculating the fourth rotation conversion matrix according to the analysis results of the N candidate rotation conversion matrices.
9. 9. A tracking system, the tracking system comprising: A camera for capturing at least one image related to a calibration chart; a first trackable device physically attached to the camera; A second trackable device physically attached to the calibration card; a tracking base station for tracking the first and second trackable devices to generate a first rotation conversion matrix between the first trackable device and the tracking base station and a second rotation conversion matrix between the second trackable device and the tracking base station, and A processing unit in communication with the tracking base station and the camera, wherein the processing unit is configured to: Generating a third rotation conversion matrix between a camera coordinate system of the camera and the calibration chart according to the calibration chart appearing in the at least one image; Calculating a fourth rotation transformation matrix between the camera coordinate system and the first trackable device according to the first rotation transformation matrix, the second rotation transformation matrix, the third rotation transformation matrix, and a fifth rotation transformation matrix between the calibration chart and the second trackable device entity, and Tracking the camera according to the first trackable device and the fourth rotation transformation matrix.
10. 10. The tracking system of claim 9, wherein the calibration chart includes a feature pattern and a load socket, the second trackable device entity being attached to the load socket and having a mechanical arrangement relationship with respect to the feature pattern, the fifth rotational transformation matrix between the calibration chart and the second trackable device entity being derived from the mechanical arrangement relationship.
11. 11. The tracking system of claim 10, wherein the fourth rotation transformation matrix is calculated based on a product of the third rotation transformation matrix, the fifth rotation transformation matrix, the second rotation transformation matrix, and the first rotation transformation matrix.
12. 12. The tracking system of claim 9, wherein the origin of the camera coordinate system is located at an optical center of the camera, and the fourth rotation transformation matrix is used to describe a rotation relationship and a positional relationship between the camera coordinate system and the first trackable device.
13. 13. The tracking system of claim 9, wherein the processing unit is configured to perform a geometric calibration of the camera according to the calibration chart in the at least one image to generate a plurality of internal parameters and a plurality of deformation parameters.
14. 14. The tracking system of claim 13, wherein the plurality of internal parameters are related to coordinate system conversion between a two-dimensional pixel coordinate system corresponding to the at least one image and are affected by a focal length, an optical center and a skew factor of the camera, the plurality of internal parameters are stored, and the stored plurality of internal parameters are used to adjust an image frame of the camera when the camera captures another image that does not involve the calibration card.
15. 15. The tracking system of claim 13, wherein the plurality of deformation parameters are associated with a plurality of non-linear lens deformations of the camera, the plurality of deformation parameters are stored, and the plurality of deformation parameters stored are used to adjust an image frame of the camera when the camera is capturing another image that does not involve the calibration card.
16. 16. The tracking system of claim 9, wherein the camera captures N images related to the calibration chart, wherein N is a positive integer greater than 1, According to the calibration chart card appearing in the N images, the processing unit generates N third rotation conversion matrixes between the camera coordinate system of the camera and the calibration chart card; According to the first rotation conversion matrix, the second rotation conversion matrix, the N third rotation conversion matrices and the fifth rotation conversion matrix, the processing unit calculates N candidate rotation conversion matrices between the camera coordinate system and the first trackable device; the processing unit statistically analyzing the N candidate rotation transformation matrices, and The processing unit calculates the fourth rotation conversion matrix according to the analysis results of the N candidate rotation conversion matrices.
17. 17. A non-transitory computer readable medium storing at least one program of instructions for execution by a processing unit to perform a tracking method comprising: Capturing at least one image relating to a calibration graphic card with a camera, wherein a first trackable device entity is attached to the camera and a second trackable device entity is attached to the calibration graphic card; tracking the first trackable device and the second trackable device with a tracking base station to generate a first rotation conversion matrix between the first trackable device and the tracking base station and generate a second rotation conversion matrix between the second trackable device and the tracking base station; Generating a camera coordinate system of the camera and a third rotation conversion matrix between the calibration chart according to the calibration chart appearing in the at least one image, and Calculating a fourth rotation transformation matrix between the camera coordinate system and the first trackable device according to the first rotation transformation matrix, the second rotation transformation matrix, the third rotation transformation matrix, and a fifth rotation transformation matrix between the calibration graphic card and the second trackable device entity, wherein the fourth rotation transformation matrix is used for tracking the camera.
18. 18. The non-transitory computer readable medium of claim 17, wherein the tracking method comprises: capturing N images related to the calibration chart by the camera, wherein N is a positive integer greater than 1, and According to the calibration chart card in the N images, a camera geometric calibration is performed to generate a plurality of internal parameters and a plurality of deformation parameters.
19. 19. The non-transitory computer readable medium of claim 18, wherein the plurality of internal parameters are related to coordinate system conversion between the camera coordinate system and a two-dimensional pixel coordinate system corresponding to one of the N images, the plurality of internal parameters are affected by a focal length, an optical center, and a skew factor of the camera, the plurality of internal parameters are stored, when the camera captures another image that is not related to the calibration chart, the plurality of internal parameters are stored for adjusting an image frame of the camera, the plurality of deformation parameters are related to a plurality of nonlinear lens deformations of the camera, the plurality of deformation parameters are stored, and when the camera captures another image that is not related to the calibration chart, the plurality of deformation parameters are stored for adjusting an image frame of the camera.
20. 20. The non-transitory computer readable medium of claim 17, wherein the tracking method comprises: capturing N images related to the calibration chart by the camera, wherein N is a positive integer greater than 1; generating N third rotation conversion matrixes between the camera coordinate system of the camera and the calibration chart according to the calibration chart appearing in the N images; calculating N candidate rotation transformation matrices between the camera coordinate system and the first trackable device according to the first rotation transformation matrix, the second rotation transformation matrix, the N third rotation transformation matrices, and the fifth rotation transformation matrix; Statistically analyzing the N candidate rotational transfer matrices, and And calculating the fourth rotation conversion matrix according to the analysis results of the N candidate rotation conversion matrices.

Description

Control method, tracking system and non-transitory computer readable medium Technical Field The present disclosure relates to a tracking system and a control method, and more particularly, to a tracking system capable of accurately and efficiently tracking a camera. Background Various Virtual Reality (VR), augmented Reality (Augmented Reality, AR), alternate Reality (Substitutional Reality, SR) and/or Mixed Reality (MR) devices have recently been developed to provide a user immersive experience. When a user wears a head-mounted display (HMD) device, the user's line of sight will be covered by the immersive content on the head-mounted display device. These immersive content display a virtual background and some items among the immersive scenes. In order to create vivid immersive content, one of the ways is to shoot real actors, real vehicles or real animals, and then fuse these real objects with virtual objects or virtual backgrounds in a virtual scene, in order to achieve the fusion, pose data (position data and rotation data) of the real objects must be accurately tracked. Otherwise, these real objects would be placed in the wrong position or wrong axis in the immersive scene. Disclosure of Invention An embodiment of the present disclosure discloses a control method including the following steps. The method includes capturing at least one image related to a calibration chart with a camera, wherein a first trackable device is physically attached to the camera and a second trackable device is physically attached to the calibration chart, tracking the first trackable device and the second trackable device with a tracking base station to generate a first rotation conversion matrix between the first trackable device and the tracking base station and generate a second rotation conversion matrix between the second trackable device and the tracking base station, generating a camera coordinate system of the camera and a third rotation conversion matrix between the calibration chart according to the calibration chart appearing in the at least one image, and calculating a fourth rotation conversion matrix between the camera coordinate system and the first trackable device according to the first rotation conversion matrix, the second rotation conversion matrix and the third rotation conversion matrix, wherein the fourth rotation conversion matrix is used for tracking the camera. In some embodiments, the calibration chart includes a feature pattern and a carrying socket, the second trackable device entity is attached to the carrying socket and has a mechanism configuration relation with respect to the feature pattern, a fifth rotation transformation matrix between the calibration chart and the second trackable device entity is derived from the mechanism configuration relation, and the calculation of the fourth rotation transformation matrix is further based on the fifth rotation transformation matrix. In some embodiments, the fourth rotation transformation matrix is calculated based on a product of the third rotation transformation matrix, the fifth rotation transformation matrix, the second rotation transformation matrix, and the first rotation transformation matrix. In some embodiments, the origin of the camera coordinate system is located at an optical center of the camera, and the fourth rotation transformation matrix is used to describe a rotation relationship and a positional relationship between the camera coordinate system and the first trackable device. In some embodiments, the control method further comprises capturing N images related to the calibration chart with the camera, wherein N is a positive integer greater than 1, and performing a geometric calibration of the camera to generate a plurality of internal parameters and a plurality of deformation parameters according to the calibration chart appearing in the N images. In some embodiments, a plurality of internal parameters related to coordinate system conversion between the camera coordinate system and a two-dimensional pixel coordinate system corresponding to one of the N images are affected by a focal length, an optical center and a skew coefficient of the camera, and are stored for adjusting an image frame of the camera when the camera captures another image that does not involve the calibration card. In some embodiments, the plurality of deformation parameters are related to a plurality of non-linear lens deformations of the camera, the plurality of deformation parameters are stored, and when the camera captures another image that does not involve the calibration chart, the plurality of deformation parameters are stored to adjust an image frame of the camera. In some embodiments, the control method further comprises capturing N images related to the calibration chart with the camera, wherein N is a positive integer greater than 1, generating N third rotation transformation matrices between the camera coordinate system of the camera and the calibrati