Search

EP-4550267-B1 - SYSTEM, DEVICE, METHOD, AND COMPUTER PROGRAM PRODUCT FOR POSITION ESTIMATION

EP4550267B1EP 4550267 B1EP4550267 B1EP 4550267B1EP-4550267-B1

Inventors

  • KAWATA, KOZO
  • YASAKI, MASATOMO

Dates

Publication Date
20260513
Application Date
20241024

Claims (10)

  1. A position estimation system comprising: three imaging devices (30a, 30b, 30c); and a position estimation device (20, 50) configured to estimate a position of an object in a three-dimensional space, based on images captured by the imaging devices (30a, 30b, 30c), wherein the three imaging devices (30a, 30b, 30c) are arranged such that imaging positions of the imaging devices (30a, 30b, 30c) form a triangle, and the position estimation device (20, 50) comprises a coordinate transformation unit(25b) configured to respectively transform coordinate points of the object in image coordinates of the imaging devices (30a, 30b, 30c) to coordinate points in stereo spherical image coordinates, a determination unit (25c) configured to determine whether or not the coordinate points of two of the imaging devices (30a, 30b, 30c) in the stereo spherical image coordinates satisfy an epipolar constraint, and an estimation unit (25d, 55c) configured to estimate that the coordinate points correspond to one coordinate position of the object in a case where the coordinate points are determined by the determination unit (25c) to satisfy the epipolar constraint, wherein the position estimation device (50) estimates a skeleton of a person, as the object, in the images by each imaging device (30a, 30b, 30c), identifies joint points for joint parts in the skeleton, estimates coordinate positions of joint points satisfying the epipolar constraint, in the skeleton, estimates a distance between an article and each joint point, calculates likelihood, based on the estimated distance, and on average distances from the article to joint points and variance thereof which are prepared in advance, and associates the article with the person, based on the distance and the likelihood.
  2. The position estimation system according to claim 1, wherein in a case where the images include only one person, this one person is associated with the article, in a case where the images include two or more persons, in a case where corresponding joint points of the two or more persons have been estimated, the article is associated with the person whose joint point is closest to the article, in a case where the corresponding joint points have not been estimated, in a case where the wrist of any one of the two or more persons has been estimated as the joint point, the article is associated with the person whose joint point is closest to the article, in a case where the wrist of any one of the persons has not been estimated as the joint point, the article is associated with the person with a largest likelihood.
  3. The position estimation system according to claim 1 or 2, wherein the coordinate transformation unit (25b) transforms a coordinate point of the object in image coordinates captured by a first imaging device(30a) to a coordinate point p1 in the stereo spherical image coordinates, transforms a coordinate point of the object in image coordinates captured by a second imaging device(30b) to a coordinate point p2 in the stereo spherical image coordinates, and transforms a coordinate point of the object in image coordinates captured by a third imaging device(30c) to a coordinate point p3 in the stereo spherical image coordinates.
  4. The position estimation system according to claim 3, wherein if a coordinate point p'1 as an intersection of a yz plane and a great circle passing through the coordinate point p1 in a stereo spherical image coordinate system using an imaging position of the first imaging device (30a) as an origin C1 is equal to a coordinate point p'2 as an intersection of a yz plane and a great circle passing through the coordinate point p2 in a stereo spherical image coordinate system using an imaging position of the second imaging device (30b) as an origin C2, and an angle β1 formed by a y axis and a straight line connecting the coordinate point p'1 and the origin of the stereo spherical image coordinates is equal to an angle β2 formed by a y axis and a straight line connecting the coordinate point p'2 and the origin of the stereo spherical image coordinates, the determination unit (25c) determines that the coordinate point p'1 of the first imaging device (30a) and the coordinate point p'2 of the second imaging device (30b) in the stereo spherical image coordinates satisfy the epipolar constraint.
  5. The position estimation system according to claim 1, wherein the position estimation device (50) further comprises a skeleton estimation unit(55b) configured to estimate a skeleton of a person, as the object, imaged by each imaging device (30a, 30b, 30c), and the estimation unit(55c) estimates a coordinate position of a neck or a wrist in the skeleton estimated by the skeleton estimation unit(55b).
  6. The position estimation system according to claim 5, wherein the position estimation device (20, 50) further comprises an article-take-out detection unit configured to detect, based on the coordinate position of the wrist estimated by the estimation unit (25d, 55c) and on position information of an article acquired when the article has been taken out, whether or not the article has been taken out, and an association unit (55e) configured to associate, when the article-take-out detection unit detects that the article has been taken out, the taken out article with a person whose wrist is closest to the article.
  7. The position estimation system according to claim 1, wherein the position estimation device (50) further comprises a skeleton estimation unit(55b) configured to estimate a skeleton of a person, as the object, imaged by each imaging device (30a, 30b, 30c), the estimation unit(55c) estimates coordinate positions of a neck, an elbow, and a wrist in the skeleton estimated by the skeleton estimation unit(55b), a likelihood calculation unit configured to calculate likelihood using a probability density function, based on average distances and variance thereof from the article to the coordinate positions of the neck, the elbow, and the wrist, the probability density function giving a larger probability value with respect to a distance between each joint point and the article when the article is being taken out, and an association unit (55e) configured to associate the article with the person, based on a calculation result by the likelihood calculation unit.
  8. The position estimation device (20, 50) according to any one of claims 1 to 7 comprising: the coordinate transformation unit (25b), the determination unit (25c), and the estimation unit (25d, 55c).
  9. A computer-implemented position estimation method for estimating a position of an object in a three-dimensional space based on images captured by three imaging devices (30a, 30b, 30c) arranged such that imaging positions of the imaging devices (30a, 30b, 30c) form a triangle, the position estimation method including transforming coordinate points of the object in image coordinates of the imaging devices (30a, 30b, 30c) respectively to coordinate points in stereo spherical image coordinates; determining whether or not the coordinate points of two of the imaging devices (30a, 30b, 30c) in the stereo spherical image coordinates satisfy an epipolar constraint, and estimating that the coordinate points correspond to one coordinate position of the object in a case where the coordinate points are determined to satisfy the epipolar constraint, wherein the method further includes estimating a skeleton of a person, as the object, in the images by each imaging device (30a, 30b, 30c), identifying joint points for joint parts in the skeleton, estimating coordinate positions of joint points satisfyi ng the epipolar constraint, in the skeleton, estimating a distance between an article and each joint p oint, calculating likelihood, based on the estimated distance, and on average distances from the article to joint points and var iance thereof which are prepared in advance, and associating the article with the person, based on the distance and the likelihood.
  10. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the method of claim 9.

Description

BATECHNICAL FIELD The present disclosure relates to a system, a device, a method, and a computer program product for estimating the position of an object in a three-dimensional space. BACKGROUND ART Conventionally, in an unmanned store selling commodities, it is important to identify which customer takes which commodity. Thus, a technology that automatically identifies the commodity a customer is purchasing and achieves an automated checkout, simply by having the customer take the commodity from a showcase and proceed to a checkout counter during shopping in a store, has been known. For example, Japanese Patent No. 7225434 discloses an information processing system that causes a plurality of cameras (range sensors) provided in a store to track a customer, acquires position information on the customer, detects a taken out commodity with a weight sensor provided in a showcase, and manages the customer and the commodity in association with each other, whereby the commodity the customer is purchasing is identified and an automated checkout is achieved without an attendant or the like. US 11 501 462 B2 relates to image processing for three-dimensional object recognition and positioning based on image data representing multiple views of a scene. Shigang Li, ICPR'06, vol.3, pages 1046-1049 relate to acquiring dense three-dimensional information of environment by correlation-based real-time spherical stereo. US 2022/262069 A1 discloses systems and techniques for calibrating cameras in a real space for tracking puts and takes of items by subjects. SUMMARY However, according to the conventional art, it is difficult to accurately acquire the position information on the customer with the plurality of cameras (range sensors). Specifically, if the positions of an object captured by the respective cameras are shifted, more processing time is needed to determine whether or not such positions correspond to the same object. The present disclosure has been made in view of the problem of the conventional art. The present disclosure addresses the problem, as discussed herein, with a system, a device, a method, and a computer program product for efficiently estimating the position of an object in a three-dimensional space. The invention is set out by the appended set of claims. The embodiments and/or examples of the following description which are not covered by the claims, are provided for illustrative purpose only and are only intended to assist the reader in understanding the present invention. However, such embodiments and/or examples which are not covered by the claims do not form part of the present invention that is solely defined by the claims. The objects, features, advantages, and technical and industrial significance of this disclosure will be better understood by the following description and the accompanying drawings of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a diagram showing an outline of a position estimation system according to embodiment 1 of the present disclosure;FIG. 2 is a diagram showing an outline of position estimation by the position estimation system according to embodiment 1;FIG. 3 is a diagram showing the system configuration of the position estimation system shown in FIG. 1;FIG. 4 is a functional block diagram showing the configuration of the position estimation device shown in FIG. 1;FIG. 5 is a diagram showing an outline of a procedure for coordinate transformation;FIG. 6 illustrates the coordinate transformation;FIG. 7 illustrates a condition satisfying the epipolar constraint;FIGS. 8A to 8C illustrate calculation of α and β;FIG. 9 illustrates a case not satisfying the epipolar constraint;FIG. 10 illustrates calculation of an intersection of vectors;FIG. 11 is a flowchart showing a processing procedure of the position estimation device shown in FIG. 1;FIG. 12 is a diagram showing an outline of a position estimation system according to embodiment 2 of the present disclosure;FIG. 13 is a diagram showing the system configuration of the position estimation system shown in FIG. 12;FIG. 14 is a functional block diagram showing the configuration of a position estimation device shown in FIG. 12;FIGS. 15A and 15B illustrate distances from an article to joint points;FIG. 16 is a diagram showing one example of a probability density function used as the likelihood;FIG. 17 is a flowchart showing a processing procedure of the position estimation device shown in FIG. 12;FIG. 18 is a flowchart showing a processing procedure of a joint position identification process;FIG. 19 is a flowchart showing a processing procedure of a likelihood calculation process; andFIG. 20 is a diagram showing one example of a configuration. DESCRIPTION OF EMBODIMENTS Hereinafter, a position estimation system, a position estimation device, a position estimation method, and a computer program product for position estimation according to embodiments of the present disclosure will be described in detail with reference