US-12621403-B2 - Methods for racking a camera bar, and related methods, systems and computer program products

US12621403B2US 12621403 B2US12621403 B2US 12621403B2US-12621403-B2

Abstract

A method of setting up non-randomly positioned markers in a field of view of a camera bar, and tracking a pose of the camera bar, including: (i) a camera bar of a video camera viewing a field of view; (ii) displaying the camera bar's view on a display; (iii) the computer system: generating a pattern of non-random marker positions, and displaying them with the camera bar's view; (iv) detecting that a marker has been placed in a position of a non-random marker position; (v) recording a marker and a respective position of the marker; (vi) repeating steps (iv) and (v) until a predetermined number of different markers and respective marker positions have been recorded; (vii) matching markers detected with the recorded markers and their respective marker positions, to obtain a pose of the camera bar; (viii) repeating step (vii), to track the pose of the camera bar.

Inventors

Gabriel BENLAHRECH
Michael Goldman
Brice Michoud

Assignees

NCAM TECHNOLOGIES LIMITED

Dates

Publication Date: 20260505
Application Date: 20220512
Priority Date: 20210512

Claims (20)

1 . A method of setting up non-randomly positioned markers in a field of view of a camera bar, and tracking a pose of the camera bar, the method including the steps of: (i) a camera bar viewing a field of view, the camera bar in fixed attachment with a video camera; (ii) displaying the camera bar's view of the field of view on a display of a computer system; (iii) the computer system generating a pattern of non-random marker positions, including positions in the field of view, and displaying the pattern of non-random marker positions in the field of view on the display of the computer system together with the camera bar's view of the field of view; (iv) the computer system detecting in the view of the camera bar, that a marker has been placed in a position of a non-random marker position, generated in step (iii); (v) the computer system recording a marker and a respective position of the marker in the field of view, in response to the computer system detecting in the view of the camera bar, that a marker has been placed in a position of a non-random marker position, in step (iv); (vi) repeating steps (iv) and (v) until a predetermined number of different markers and respective marker positions have been recorded; (vii) the computer system matching markers detected in the field of view of the camera bar with the recorded markers and their respective marker positions, to obtain a pose of the camera bar; (viii) repeating step (vii), to track the pose of the camera bar.
2 . The method of claim 1 , including the step of storing the recorded predetermined number of different markers and respective marker positions.
3 . The method of claim 1 , wherein the generated pattern of non-random marker positions in the field of view starts from an initial indicator which is detected in the field of view by the computer system.
4 . The method of claim 1 , wherein the generated pattern of non-random marker positions in the field of view is generated using a path represented by a continuously differentiable (C1) continuous path or function.
5 . The method of claim 4 , wherein the markers positions are equidistantly spaced along the path.
6 . The method of claim 1 , wherein the generated pattern of non-random marker positions in the field of view is generated using a path represented by a smooth continuous path or function.
7 . The method of claim 1 , wherein the generated pattern of non-random marker positions in the field of view is generated from a set of parameters which are passed into one or more well-defined, deterministic formulas.
8 . The method of claim 7 , wherein a formula generates a spiral pattern and the parameters include the position of an initial marker, the camera bar height above the floor and the orientation angle(s) of the camera bar.
9 . The method of claim 1 , wherein the generated pattern of non-random marker positions in the field of view is a spiral pattern.
10 . The method of claim 1 , wherein the generated pattern of non-random marker positions in the field of view is a quasicrystal pattern.
11 . The method of claim 1 , wherein the generated pattern of non-random marker positions in the field of view is on a non-periodic grid, or on a periodic grid, or on a non-periodic lattice, or on a periodic lattice.
12 . The method of claim 1 , wherein the generated pattern of non-random marker positions in the field of view is on a set of non-intersecting closed curves, e.g. on a set of concentric circles.
13 . The method of claim 1 , wherein the generated pattern of non-random marker positions in the field of view is on a set of non-intersecting open curves, e.g. on a set of concentric semicircles.
14 . The method of claim 1 , wherein in step (iii), the pattern of non-random marker positions in the field of view are overlaid on the camera bar's view of the field of view.
15 . The method of claim 1 , wherein in step (iv), the computer system displays in the display that it has detected in the view of the camera bar that a marker has been placed in a position of a non-random marker position, e.g. by changing a displayed colour of the non-random marker position, e.g. from red to green.
16 . The method of claim 1 , wherein the camera bar is a monoscopic camera bar.
17 . The method of claim 16 , wherein step (vii) includes (a) detecting markers' 2D positions in a sequence of images viewed by the camera bar using a marker extraction algorithm, where the first image and the last image of the sequence are taken from different camera bar positions, and labelling markers through the sequence using nearest neighbour matching; (b) extracting matrices using a robust extractor to extract rotation and translation, in which a scale factor is unknown, from the first frame of the sequence to the last frame of the sequence; (c) computing 3D marker positions, in which the scale factor is unknown, using the detected 2D marker positions, using camera bar calibration data; (d1) matching the computed 3D marker scaled positions to a 3D path of the markers, where the 3D path in a world coordinate system is known, including fitting the set of computed 3D marker positions to the 3D path of the markers, including deriving the scale factor; or (d2) matching the computed 3D marker scaled positions to the 3D positions of the markers, where the 3D positions of the markers in a world coordinate system are known, including fitting the set of computed 3D marker positions to the 3D positions of the markers, including deriving the scale factor; (e) modelling the matching solution as a SIM (3) of the Lie Group, to represent camera bar pose, the matching solution including a scale, and (f) extracting the camera bar pose from the matching solution, in the world coordinate system.
18 . The method of claim 17 , wherein steps (d1) or (d2) are performed by using Iterative closest point (ICP) fitting, or curve fitting or using a non-linear minimization technique such as a Levenberg-Marquardt algorithm (LMA or just LM); or using a Gauss Newton algorithm.
19 . The method of claim 17 , wherein in step (b), Random sample consensus, RANSAC, is used.
20 . The method of claim 1 , wherein the Camera bar is a stereoscopic camera bar, including two cameras mounted a fixed distance apart in a stereoscopic configuration.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the priority of International Application No. PCT/GB2022/051208 filed May 12, 2022, which claims priority to GB Application No. 2106752.5, filed May 12, 2021, the entire contents of which being fully incorporated herein by reference. BACKGROUND OF THE INVENTION 1. Field of the Invention The field of the invention relates to methods of tracking a camera bar, or of tracking a video camera including a camera bar, and to related methods, systems, computer program products, video files, and video streams. 2. Technical Background When using a system for producing a virtual scene combining live video enhanced by other imagery, such as computer generated imagery, users want to be able to set the system up quickly, because studio time is expensive. It is also desirable to reduce energy consumption of a such a system, because video processing tends to use significant amounts of energy. Users also prefer to be able to view the virtual scene combining live video enhanced by other imagery, such as computer generated imagery, in real-time. 3. Discussion of Related Art US20070248283(A1) discloses a system for producing a virtual scene combining live video enhanced by other imagery, including computer generated imagery. In one embodiment it includes a scene camera with an attached tracking camera, the tracking camera viewing a tracking marker pattern, which has a plurality of tracking markers with identifying indicia. The tracking marker pattern is positioned proximate so that when viewed by the tracking camera, the coordinate position of the scene camera can be determined in real time. The disclosure also includes filtering algorithms, that vary based on camera motion and maintain accurate positioning. It is disclosed that alternative embodiments could include a randomly distributed array of markers, and markers of various sizes and orientations. SUMMARY OF THE INVENTION According to a first aspect of the invention, there is provided a method of setting up non-randomly positioned markers in a field of view of a camera bar, and tracking a pose of the camera bar, the method including the steps of: (i) a camera bar viewing a field of view, the camera bar in fixed attachment with a video camera;(ii) displaying the camera bar's view of the field of view on a display of a computer system;(iii) the computer system generating a pattern of non-random marker positions, including positions in the field of view, and displaying the pattern of non-random marker positions in the field of view on the display of the computer system together with the camera bar's view of the field of view;(iv) the computer system detecting in the view of the camera bar, that a marker has been placed in a position of a non-random marker position, generated in step (iii);(v) the computer system recording a marker and a respective position of the marker in the field of view, in response to the computer system detecting in the view of the camera bar, that a marker has been placed in a position of a non-random marker position, in step (iv);(vi) repeating steps (iv) and (v) until a predetermined number of different markers and respective marker positions have been recorded;(vii) the computer system matching markers detected in the field of view of the camera bar with the recorded markers and their respective marker positions, to obtain a pose of the camera bar;(viii) repeating step (vii), to track the pose of the camera bar. The generated pattern of non-random marker positions may include positions outside an initial field of view of the camera bar. An advantage is that the non-randomly positioned markers in a field of view of a camera bar can be set up quickly by just one person who can place the markers while viewing the display, while the computer system detects when the person has placed the marker in a non-random marker position. An advantage is that the pose of the camera bar can be tracked using less energy, because matching the markers detected in the field of view of the camera bar with the recorded markers and their respective non-randomly positioned marker positions is computationally efficient, because the non-randomly positioned marker positions are highly distinct, which reduces the energy requirements for the computation. An advantage is that the pose of the camera bar can be tracked in real-time, because matching the markers detected in the field of view of the camera bar with the recorded markers and their respective non-randomly positioned marker positions is computationally efficient, because the non-randomly positioned marker positions are highly distinct, which reduces the time requirements for the computation. The camera bar and the computer system may be connected by a data cable, or they may be connected by a wireless data connection. The method may be one including the step of storing the recorded predetermined number of different markers and respective marker positions. The