US-20260127749-A1 - INFORMATION PROCESSING DEVICE, INFORMATION PROCESSING METHOD, AND PROGRAM

US20260127749A1US 20260127749 A1US20260127749 A1US 20260127749A1US-20260127749-A1

Abstract

To correctly track the same object in consideration of a change in appearance due to movement in direction, in an information processing device, a processor generates a trajectory fragment indicating a trajectory in which an object included in a frame image moves and including information indicating at least appearance features in each frame image. The processor calculates a correlation of the appearance features for an object pair formed by extracting an object from each of first and second trajectory fragments included in a fragment pair concerning the trajectory, and calculates an appearance similarity of the fragment pair based on the correlation and a similarity between appearance features of the object pair. The processor calculates a fragment pair similarity between the first and second trajectory fragments by using the appearance similarity, and combines trajectory fragment pairs based on the fragment pair similarity to calculate a trajectory for the same object.

Inventors

Shuhei Yoshida
Takashi Shibata
Makoto Terao

Assignees

NEC CORPORATION

Dates

Publication Date: 20260507
Application Date: 20251016
Priority Date: 20241106

Claims (10)

1 . An information processing device comprising: at least one memory configured to store instructions; and at least one processor configured to execute the instructions to: generate a trajectory fragment indicating at least a part of a trajectory in which an object included in a time-series frame image moves and including object information indicating time, coordinates, and an appearance feature amount in each frame image of the object; calculate a correlation of the appearance feature amounts for an object pair formed by extracting an object one by one from each of a first trajectory fragment and a second trajectory fragment included in a trajectory fragment pair that is a pair of the trajectory fragments; calculate an appearance similarity of the trajectory fragment pair based on the correlation and a similarity between appearance feature amounts of the object pair; calculate a fragment pair similarity that is a similarity between the first trajectory fragment and the second trajectory fragment by using the appearance similarity; and combine a plurality of trajectory fragment pairs based on the fragment pair similarity to calculate an object trajectory for the same object.
2 . The information processing device according to claim 1 , wherein the processor is further configured to calculate coordinate similarity indicating consistency of time and coordinate included in each of the first trajectory fragment and the second trajectory fragment, wherein the processor calculates the fragment pair similarity based on the appearance similarity and the coordinate similarity.
3 . The information processing device according to claim 1 , wherein the processor calculates the appearance similarity by weighting and adding similarity between appearance feature amounts of a plurality of object pairs using the correlation as a weight.
4 . The information processing device according to claim 3 , wherein the processor sets a weight of a maximum value of the correlation to 1 and sets a weight of a correlation other than the maximum value to 0.
5 . The information processing device according to claim 3 , wherein the processor calculates the weight by inputting a value of the correlation to a softmax function.
6 . The information processing device according to claim 1 , wherein the processor calculates an inner product or a cosine similarity of the appearance feature amounts of the object pair as the correlation.
7 . The information processing device according to claim 1 , wherein the processor calculates the correlation by inputting object information of the object pair to a neural network learned in advance.
8 . The information processing device according to claim 1 , wherein the processor calculates the object trajectory by connecting a plurality of pairs of temporally adjacent trajectory fragments in a single trajectory of the same object.
9 . An information processing method executed by a computer, the method comprising: generating a trajectory fragment indicating at least a part of a trajectory in which an object included in a time-series frame image moves and including object information indicating time, coordinates, and an appearance feature amount in each frame image of the object; calculating a correlation of the appearance feature amounts for an object pair formed by extracting an object one by one from each of a first trajectory fragment and a second trajectory fragment included in a trajectory fragment pair that is a pair of the trajectory fragments; calculating an appearance similarity of the trajectory fragment pair based on the correlation and a similarity between appearance feature amounts of the object pair; calculating a fragment pair similarity that is a similarity between the first trajectory fragment and the second trajectory fragment by using the appearance similarity; and combining a plurality of trajectory fragment pairs based on the fragment pair similarity to calculate an object trajectory for the same object.
10 . A non-transitory computer-readable recording medium storing a program causing a computer to execute processing of: generating a trajectory fragment indicating at least a part of a trajectory in which an object included in a time-series frame image moves and including object information indicating time, coordinates, and an appearance feature amount in each frame image of the object; calculating a correlation of the appearance feature amounts for an object pair formed by extracting an object one by one from each of a first trajectory fragment and a second trajectory fragment included in a trajectory fragment pair that is a pair of the trajectory fragments; calculating an appearance similarity of the trajectory fragment pair based on the correlation and a similarity between appearance feature amounts of the object pair; calculating a fragment pair similarity that is a similarity between the first trajectory fragment and the second trajectory fragment by using the appearance similarity; and combining a plurality of trajectory fragment pairs based on the fragment pair similarity to calculate an object trajectory for the same object.

Description

INCORPORATION BY REFERENCE This application is based upon and claims the benefit of priority from Japanese Patent Application 2024-194516, filed on Nov. 6, 2024, the disclosure of which is incorporated herein in its entirety by reference. TECHNICAL FIELD The present disclosure relates to tracking objects in a video. BACKGROUND ART A technique for tracking an object in a video captured by a camera or the like has been proposed. For example, Patent Document 1 A describes a method of tracking an object by linking trajectories of the object detected in a video. Patent Document 1: Japanese Patent Application Laid-Open under No. 2019-194857 SUMMARY In a method of Patent Document 1, whether to link pairs of trajectories is determined based on the similarity of the objects included in the trajectory. However, even in the case of the same object, the appearance of the object changes due to movement, a change in direction, or the like. Therefore, it is not possible to correctly determine the identity of the object only by simply comparing the similarity of the appearances. One object of the present disclosure is to provide an information processing device capable of correctly tracking the same object in consideration of a change in appearance due to movement, a change in direction, or the like of the object. According to an example aspect of the present invention, there is provided an information processing device including: at least one memory configured to store instructions; andat least one processor configured to execute the instructions to:generate a trajectory fragment indicating at least a part of a trajectory in which an object included in a time-series frame image moves and including object information indicating time, coordinates, and an appearance feature amount in each frame image of the object;calculate a correlation of the appearance feature amounts for an object pair formed by extracting an object one by one from each of a first trajectory fragment and a second trajectory fragment included in a trajectory fragment pair that is a pair of the trajectory fragments;calculate an appearance similarity of the trajectory fragment pair based on the correlation and a similarity between appearance feature amounts of the object pair;calculate a fragment pair similarity that is a similarity between the first trajectory fragment and the second trajectory fragment by using the appearance similarity; and combine a plurality of trajectory fragment pairs based on the fragment pair similarity to calculate an object trajectory for the same object. According to another example aspect of the present invention, there is provided an information processing method executed by a computer, the method including: generating a trajectory fragment indicating at least a part of a trajectory in which an object included in a time-series frame image moves and including object information indicating time, coordinates, and an appearance feature amount in each frame image of the object;calculating a correlation of the appearance feature amounts for an object pair formed by extracting an object one by one from each of a first trajectory fragment and a second trajectory fragment included in a trajectory fragment pair that is a pair of the trajectory fragments;calculating an appearance similarity of the trajectory fragment pair based on the correlation and a similarity between appearance feature amounts of the object pair;calculating a fragment pair similarity that is a similarity between the first trajectory fragment and the second trajectory fragment by using the appearance similarity; and combining a plurality of trajectory fragment pairs based on the fragment pair similarity to calculate an object trajectory for the same object. According to still another example aspect of the present invention, there is provided a non-transitory computer-readable recording medium storing a program causing a computer to execute processing of: generating a trajectory fragment indicating at least a part of a trajectory in which an object included in a time-series frame image moves and including object information indicating time, coordinates, and an appearance feature amount in each frame image of the object;calculating a correlation of the appearance feature amounts for an object pair formed by extracting an object one by one from each of a first trajectory fragment and a second trajectory fragment included in a trajectory fragment pair that is a pair of the trajectory fragments;calculating an appearance similarity of the trajectory fragment pair based on the correlation and a similarity between appearance feature amounts of the object pair;calculating a fragment pair similarity that is a similarity between the first trajectory fragment and the second trajectory fragment by using the appearance similarity; and combining a plurality of trajectory fragment pairs based on the fragment pair similarity to calculate an object trajectory for the same object. Effec