US-12626405-B2 - Camera calibration of a telepresence system

US12626405B2US 12626405 B2US12626405 B2US 12626405B2US-12626405-B2

Abstract

A telepresence system may include a display configured to present three-dimensional images. The 3D images may be rendered from multiple images captured by multiple cameras that image an area from different viewpoints. Misalignment of any of the multiple cameras may negatively affect the rendering. Accordingly, the telepresence system may calibrate the cameras to compensate for any misalignment as part of the rending. This calibration may include capturing an image, or images, of a calibration target to determine the relative positions of the cameras. The disclosed telepresence system can perform this calibration online and using targets in fixed locations that are not easily noticeable to a user

Inventors

Guillermo Fabian Díaz Lankenau
Alejandro Jose Troccoli
Antonio Yamil Layon Halun

Assignees

GOOGLE LLC

Dates

Publication Date: 20260512
Application Date: 20230424

Claims (19)

1 . A telepresence system comprising: a local display configured to present three-dimensional images to a local user in a local viewing-area; a local set of cameras positioned relative to each other to capture images of the local viewing-area from multiple perspectives; and a local processor communicatively coupled to the local display and the local set of cameras, the local processor configured by software instructions to perform a calibration process that includes: determining a first set of relative positions of a first pair of cameras of the local set of cameras based on a target visible in a first set of images captured by the first pair of cameras; determining a second set of relative positions of a second pair of cameras of the local set of cameras based on the target visible in a second set of images captured by the second pair of cameras; determining a third set of relative positions of a third pair of cameras of the local set of cameras based on the first set of relative positions and the second set of relative positions; and generating, or updating, a calibration for the local set of cameras to support rendering of stereoscopic three-dimensional images based on the first set of relative positions, the second set of relative positions, and the third set of relative positions, the calibration corresponding to misalignments of the local set of cameras from a set of target alignments.
2 . The telepresence system according to claim 1 , the target is on an accessory that is in a fixed location in the local viewing-area, the target facing the local set of cameras and facing away from the local user so that the accessory blocks the local user from viewing the target.
3 . The telepresence system according to claim 1 , wherein: the telepresence system is configured to provide communication between the local user and a remote user; and the calibration process occurs online without interrupting the communication between the local user and the remote user.
4 . The telepresence system according to claim 1 , wherein the calibration process is triggered by a temperature corresponding to the local set of cameras.
5 . The telepresence system according to claim 1 , wherein the target is on a speaker that is fastened to a desk surface of a desk in the local viewing-area, the target facing the local set of cameras and facing away from the local user so that the speaker blocks the local user from viewing the target.
6 . The telepresence system according to claim 1 , wherein the target is on a microphone that is fastened to a desk surface of a desk in the local viewing-area, the target facing the local set of cameras and facing away from the local user so that the microphone blocks the local user from viewing the target.
7 . The telepresence system according to claim 1 , wherein the target is on an equipment-mount fastened to a desk surface of a desk in the local viewing-area, the target facing the local set of cameras and facing away from the local user so that the equipment-mount blocks the local user from viewing the target.
8 . The telepresence system according to claim 1 , wherein the target is a mirror configured to reflect a pattern presented on the local display to the local set of cameras, the mirror facing away from the local user.
9 . The telepresence system according to claim 1 , wherein the target is not viewable by the third pair of cameras of the local set of cameras.
10 . The telepresence system according to claim 1 , wherein the target is located on a wall of the local viewing-area, the wall being behind the local user and facing the local set of cameras.
11 . The telepresence system according to claim 10 , wherein the target includes a pattern that obscures a purpose of the target from the local user.
12 . The telepresence system according to claim 1 , wherein the local set of cameras are mounted in a frame surrounding the local display.
13 . The telepresence system according to claim 12 , wherein the local set of cameras include: the first pair of cameras of the local set of cameras that are positioned in a top portion of the frame above the local display; the second pair of cameras of the local set of cameras that are positioned in a left portion the frame; and the third pair of cameras of the local set of cameras that are positioned in a right portion the frame.
14 . A method for calibrating cameras of a telepresence system, the method comprising: capturing images of a set of targets using a local set of cameras mounted in a frame of a local display, each target of the set of targets at fixed locations in a local viewing-area; identifying a first set of images captured by a first pair of cameras of the local set of cameras that include a first target of the set of targets; determine a first set of relative positions of the first pair of cameras based on the first target in the first set of images; identifying a second set of images captured by a second pair of cameras of the local set of cameras that include a second target of the set of targets; determine a second set of relative positions of the second pair of cameras based on the second target in the second set of images; transitively computing a third set of relative positions of a third pair of cameras based on the first set of relative positions and the second set of relative positions; and generating, or updating, a calibration to support rendering of stereoscopic three-dimensional images based on the first set of relative positions, the second set of relative positions, and the third set of relative positions, the calibration corresponding to misalignments of the local set of cameras from a set of target alignments.
15 . The method for calibrating the cameras of the telepresence system according to claim 14 , wherein: the first target is located on a surface of an accessory that faces the local set of cameras and that faces away from a local user in the local viewing-area so that the accessory blocks the local user from viewing the first target; and the second target is located on a wall of the local viewing-area, the wall behind the local user and facing the local set of cameras.
16 . The method for calibrating the cameras of the telepresence system according to claim 15 , wherein the first target or the second target includes a pattern that obscures a purpose of the first target or the second target from the local user.
17 . The method for calibrating the cameras of the telepresence system according to claim 14 , wherein the first target or the second target includes a mirror configured to reflect a pattern displayed on the local display to the local set of cameras.
18 . A telepresence system comprising: a local viewing-area including: a local set of cameras positioned relative to each other to capture images of the local viewing-area from multiple perspectives; a target fixedly disposed on a surface of the local viewing-area; and a local processor communicatively coupled to the local set of cameras, the local processor configured by local software instructions to perform a calibration process that includes: capturing images of the target from at least two cameras of the local set of cameras; determining based on the images of the target, relative positions between the at least two cameras; and generating, or updating, a calibration file based on the relative positions; and a remote viewing-area including: a remote display configured to present three-dimensional images; and a remote processor communicatively coupled to the local processor and the remote display, the remote processor configured by remote software instructions to perform a three-dimensional image rendering process that includes: receiving images captured by the local set of cameras and the calibration file generated by the local processor; generating adjusted images based on the calibration file, the adjusted images including corrected perspectives of the local viewing-area that compensate for misalignments of the local set of cameras; and combining the adjusted images for display as the three-dimensional images on the remote display.
19 . The telepresence system according to claim 18 , wherein: the target is hidden from view of a local user in the local viewing-area and/or includes a pattern that obscures a purpose the target from the local user.

Description

FIELD OF THE DISCLOSURE The present disclosure relates to a system and method for measuring relative positions of the cameras in a telepresence system using targets installed in the environment of a user. BACKGROUND A telepresence conferencing system (i.e., telepresence system) can be used for audio/video communication between people. Some telepresence systems use a variety of techniques to enhance the realism of this communication in order to make a user feel like they are speaking in-person with another user. One technology used for this realism is the display. The display used in a telepresence system can be sized and positioned so that the user can view the person at an expected size (i.e., life-sized). Additionally, the display may be configured to display images that appear to be three-dimensional (3D). These 3D displays can require multiple images captured by a set of cameras configured to image a subject from multiple perspectives (i.e., multiple viewpoints). SUMMARY For high quality 3D video reproduction, it is necessary to know the relative positions of cameras in a camera array (i.e., set of cameras). A calibration process can be used to determine these relative positions but may require a specialized, obtrusive process that is performed infrequently (e.g., once or twice a year). Disclosed herein is a calibration process that can be performed online and that does not require any specialized equipment or user participation. Further, a user may not notice the process occurring because targets for the calibration may be positioned or designed to obscure their use from the user. In some aspects, the techniques described herein relate to a telepresence system including: a local display configured to present three-dimensional images to a local user in a local viewing-area; a local set of cameras positioned relative to each other to capture images of the local viewing-area from multiple perspectives; an accessory positioned in the local viewing-area; a target disposed on a surface of the accessory that faces the local set of cameras and that faces away from the local user so that the accessory blocks the local user from viewing the target; and a local processor communicatively coupled to the local display and the local set of cameras, the local processor configured by software instructions to perform a calibration process that includes: capturing images of the target from at least two cameras of the local set of cameras. In some aspects, the techniques described herein relate to a method for calibrating cameras of a telepresence system, the method including: capturing images of a set of targets using a local set of cameras mounted in a frame of a local display, each target of the set of targets at fixed locations in a local viewing-area; identifying a first set of images captured by at least two cameras that include a first target of the set of targets; determine a first set of relative positions of the at least two cameras based on the first target in the first set of images; identifying a second set of images captured by at least two cameras that include a second target of the set of targets; determine a second set of relative positions of the at least two cameras based on the second target in the second set of images; and generating, or updating, a calibration for the local set of cameras based on the first set of relative positions and the second set of relative positions, the calibration corresponding to misalignments of the local set of cameras from a set of target alignments. In some aspects, the techniques described herein relate to a telepresence system including: a local viewing-area including: a local set of cameras positioned relative to each other to capture images of the local viewing-area from multiple perspectives; a target fixedly disposed on a surface of the local viewing-area; and a local processor communicatively coupled to the local set of cameras, the local processor configured by local software instructions to perform a calibration process that includes: capturing images of the target from at least two cameras of the local set of cameras; determining based on the images of the target, relative positions between the at least two cameras; and generating, or updating, a calibration file based on the relative positions; and a remote viewing-area including: a remote display configured to present three-dimensional images; and a remote processor communicatively coupled to the local processor and the remote display, the remote processor configured by remote software instructions to perform a three-dimension image rendering process that includes: receiving images captured by the local set of cameras and the calibration file generated by the local processor; generating adjusted images based on the calibration file, the adjusted images including corrected perspectives of the local viewing-area that compensate for misalignments of the local set of cameras; and combining the adjusted images fo