US-12620104-B2 - Information processing program, information processing method, and information processing device

Abstract

A non-transitory computer-readable recording medium has stored therein an information processing program that causes a computer to execute a process including acquiring a video analyzing the acquired video identifying, based on a result of the analyzing, a first area containing a first object included in the acquired video, a second area containing a second object included in the acquired video, and a relationship that identifies interaction between the first object and the second object based on the specified relationship, selecting a model that is relevant to any one of the first object and the second object from a plurality of models and outputting the selected model.

Inventors

• Shun KOHATA

Assignees

• FUJITSU LIMITED

Dates

Publication Date

20260505

Application Date

20230818

Priority Date

20221214

Claims (18)

1. 1 . A non-transitory computer-readable recording medium having stored therein an information processing program that causes a computer to execute a process comprising: acquiring a video; analyzing the acquired video; identifying, based on a result of the analyzing, a first area containing a first object included in the acquired video, a second area containing a second object included in the acquired video, and a relationship that identifies interaction between the first object and the second object, the relationships being a first relationship, a second relationship or a third relationship; based on the specified relationship, selecting a model that is relevant to any one of the first object and the second object from a plurality of models, the plurality of models include a first model corresponding to the first relationship, a second model corresponding to the second relationship and a third model corresponding to the third relationship, the first model is a model trained with first training data including product advertisement information, the second model is a model trained with second training data including information describing popularity of the product, and the third model is a model trained with third training data including information describing a benefit that is earned by purchasing a product; and outputting the selected model among the first model, the second model and the third model.
2. 2 . The non-transitory computer-readable recording medium according to claim 1 , wherein the process further includes: tracking, based on the result of the analyzing, the first object and the second object; based on the specified relationship, selecting a machine learning model to be applied to the second object from the first model, the second model and the third model that are machine learning models; inputting an image of the tracked second object to the selected machine learning model; specifying relevant information on the second object based on the output from the selected machine learning model; and causing a display device that is relevant to the tracked second object to output the specified relevant information relevant to the second object.
3. 3 . The non-transitory computer-readable recording medium according to claim 1 , wherein the first object is a person, the second object is a product, and the process further includes: referring to a storage in which the specified relationship and a plurality of serving-customer models that define content of serving a customer are associated; specifying a serving-customer model corresponding to the specified relationship from the first model, the second model and the third model that are serving-customer models; based on the specified serving-customer model, specifying content of serving a customer that is for the person represented by the first object and that is relevant to the object represented by the second object; and transmitting the specified content of serving a customer to a terminal device that a sales staff uses.
4. 4 . The non-transitory computer-readable recording medium according to claim 1 , wherein the first object is a person, the second object is a product, and the process further includes: referring to a storage in which the specified relationship and a plurality of trained machine learning models that output product information are associated; specifying a machine learning model corresponding to the specified relationship from the trained machine learning models that include the first model, the second model and the third model; inputting an image of the product represented by the specified second object to the specified machine learning model; specifying product information outputted from the specified machine learning model; and causing a terminal device that the person represented by the first object uses to display the specified product information.
5. 5 . The non-transitory computer-readable recording medium according to claim 4 , wherein the process further includes: tracking, based on the result of the analyzing, a position of a person in a store; based on the tracked position of the person in the store, specifying a terminal device that the person uses; and causing the specified terminal device to display the specified product information.
6. 6 . The non-transitory computer-readable recording medium according to claim 1 , wherein the process further includes: inputting the video to a given model; and specifying, based on the output from the given model, the first area, the second area, and the relationship and the given model is a model for HOID (Human Object Interaction Detection) on which machine learning is executed in order to identify first area information that presents a first class presenting a person who purchases a product and an area in which the person appears, second area information that presents a second class presenting an object containing a product and an area in which the object appears, and interaction between the first class and the second class.
7. 7 . An information processing method comprising: acquiring a video; analyzing the acquired video; identifying, based on a result of the analyzing, a first area containing a first object included in the acquired video, second area containing a second object included in the acquired video, and a relationship that identifies interaction between the first object and the second object, the relationships being a first relationship, a second relationship or a third relationship; based on the specified relationship, selecting a model that is relevant to any one of the first object and the second object from a plurality of models by using a processor, the plurality of models include a first model corresponding to the first relationship, a second model corresponding to the second relationship and a third model corresponding to the third relationship, the first model is a model trained with first training data including product advertisement information, the second model is a model trained with second training data including information describing popularity of the product, and the third model is a model trained with third training data including information describing a benefit that is earned by purchasing a product; and outputting the selected model among the first model, the second model and the third model.
8. 8 . The information processing method according to claim 7 , further including: tracking, based on the result of the analyzing, the first object and the second object; based on the specified relationship, selecting a machine learning model to be applied to the second object from the first model, the second model and the third model that are machine learning models; inputting an image of the tracked second object to the selected machine learning model; specifying relevant information on the second object based on the output from the selected machine learning model; and causing a display device that is relevant to the tracked second object to output the specified relevant information relevant to the second object.
9. 9 . The information processing method according to claim 7 , wherein the first object is a person, the second object is a product, and the method further includes: referring to a storage in which the specified relationship and a plurality of serving-customer models that define content of serving a customer are associated; specifying a serving-customer model corresponding to the specified relationship from the first model, the second model and the third model that are serving-customer models; based on the specified serving-customer model, specifying content of serving a customer that is for the person represented by the first object and that is relevant to the object represented by the second object; and transmitting the specified content of serving a customer to a terminal device that a sales staff uses.
10. 10 . The information processing method according to claim 7 , wherein the first object is a person, the second object is a product, and the method further includes: referring to a storage in which the specified relationship and a plurality of trained machine learning models that output product information are associated; specifying a machine learning model corresponding to the specified relationship from the trained machine learning models that include the first model, the second model and the third model; inputting an image of the product represented by the specified second object to the specified machine learning model; specifying product information outputted from the specified machine learning model; and causing a terminal device that the person represented by the first object uses to display the specified product information.
11. 11 . The information processing method according to claim 10 , including: tracking, based on the result of the analyzing, a position of a person in a store; based on the tracked position of the person in the store, specifying a terminal device that the person uses; and causing the specified terminal device to display the specified product information.
12. 12 . The information processing method according to claim 7 , including: inputting the video to a given model; and specifying, based on the output from the given model, the first area, the second area, and the relationship, and the given model is a model for HOID (Human Object Interaction Detection) on which machine learning is executed in order to identify first area information that presents a first class presenting a person who purchases a product and an area in which the person appears, second area information that presents a second class presenting an object containing a product and an area in which the object appears, and interaction between the first class and the second class.
13. 13 . An information processing device comprising: a memory; and a processor coupled to the memory and configured to: acquire a video; analyze the acquired video; identify, based on analyzing the acquired video, a first area containing a first object included in the acquired video, a second area containing a second object included in the acquired video, and a relationship that identifies interaction between the first object and the second object, the relationships being a first relationship, a second relationship or a third relationship; based on the specified relationship, select a model that is relevant to any one of the first object and the second object from a plurality of models, the plurality of models include a first model corresponding to the first relationship, a second model corresponding to the second relationship and a third model corresponding to the third relationship, the first model is a model trained with first training data including product advertisement information, the second model is a model trained with second training data including information describing popularity of the product, and the third model is a model trained with third training data including information describing a benefit that is earned by purchasing a product; and output the selected model.
14. 14 . The information processing device according to claim 13 , wherein the processor is further configured to: track, based on analyzing the acquired video, the first object and the second object; based on the specified relationship, select a machine learning model to be applied to the second object from the first model, the second model and the third model that are machine learning models; input an image of the tracked second object to the selected machine learning model; specify relevant information on the second object based on the output from the selected machine learning model; and cause a display device that is relevant to the tracked second object to output the specified relevant information relevant to the second object.
15. 15 . The information processing device according to claim 13 , wherein the first object is a person, the second object is a product, and the processor is further configured to: refer to a storage in which the specified relationship and a plurality of serving-customer models that define content of serving a customer are associated; specify a serving-customer model corresponding to the specified relationship from the first model, the second model and the third model that are serving-customer models; based on the specified serving-customer model, specify content of serving a customer that is for the person represented by the first object and that is relevant to the object represented by the second object; and transmit the specified content of serving a customer to a terminal device that a sales staff uses.
16. 16 . The information processing device according to claim 13 , wherein the first object is a person, the second object is a product, and the processor is further configured to: refer to a storage in which the specified relationship and a plurality of trained machine learning models that output product information are associated; specify a machine learning model corresponding to the specified relationship from the trained machine learning models that include the first model, the second model and the third model; input an image of the product represented by the specified second object to the specified machine learning model; specify product information outputted from the specified machine learning model; and cause a terminal device that the person represented by the first object uses to display the specified product information.
17. 17 . The information processing device according to claim 16 , wherein the processor is further configured to: track, based on the result of the analyzing, a position of a person in a store; based on the tracked position of the person in the store, specify a terminal device that the person uses; and cause the specified terminal device to display the specified product information.
18. 18 . The information processing device according to claim 13 , wherein the processor is further configured to: input the video to a given model; and specify, based on the output from the given model, the first area, the second area, and the relationship, and the given model is a model for HOID (Human Object Interaction Detection) on which machine learning is executed in order to identify first area information that presents a first class presenting a person who purchases a product and an area in which the person appears, second area information that presents a second class presenting an object containing a product and an area in which the object appears, and interaction between the first class and the second class.

Description

CROSS-REFERENCE TO RELATED APPLICATION (S) This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2022-199604, filed on Dec. 14, 2022, the entire contents of which are incorporated herein by reference. FIELD The embodiments discussed herein are related to an information processing program, etc. BACKGROUND Distributors take various measures in order to sell much more products. For example, information on products that are set previously is displayed on an accounting machines, or the like, and sales staffs serve customers. If a sales staff is able to serve a customer appropriately when the customer represents an interest in a product, it is possible to increase the customer's motivation to purchase the product. For example, related arts are disclosed in Patent Literature 1: Japanese Laid-open Patent Publication No. 2022-165483 SUMMARY According to an aspect of an embodiment, a non-transitory computer-readable recording medium has stored therein an information processing program that causes a computer to execute a process including acquiring a video analyzing the acquired video identifying, based on a result of the analyzing, a first area containing a first object included in the acquired video, a second area containing a second object included in the acquired video, and a relationship that identifies interaction between the first object and the second object based on the specified relationship, selecting a model that is relevant to any one of the first object and the second object from a plurality of models and outputting the selected model. The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention, as claimed. BRIEF DESCRIPTION OF DRAWINGS FIG. 1 is a diagram illustrating an example of a system according to a first embodiment; FIG. 2 is Diagram (1) for describing a process performed by an information processing device according to the first embodiment; FIG. 3 is Diagram (2) for describing the process performed by an information processing device according to the first embodiment; FIG. 4 is a diagram illustrating machine learning of HOID; FIG. 5 is a functional block diagram illustrating a configuration of the information processing device according to the first embodiment; FIG. 6 is a diagram illustrating an example of a data structure of a model table according to the first embodiment; FIG. 7 is a diagram illustrating an example of a data structure of a display device management table according to the first embodiment; FIG. 8 is a flowchart illustrating a procedure of the process performed by the information processing device according to the first embodiment; FIG. 9 is a diagram illustrating an example of a system according to a second embodiment; FIG. 10 is Diagram (1) for describing a process performed by an information processing device according to the second embodiment; FIG. 11 is Diagram (2) for describing the process performed by the information processing device according to the second embodiment; FIG. 12 is a functional block diagram illustrating a configuration of the information processing device according to the second embodiment; FIG. 13 is a diagram illustrating an example of a data structure of a model table according to the second embodiment; FIG. 14 is a flowchart illustrating a procedure of the process performed by the information processing device according to the second embodiment; and FIG. 15 is a diagram illustrating an example of a hardware configuration of a computer that implements the same functions as those of the information processing device of the embodiment. DESCRIPTION OF EMBODIMENT (S) Only simply displaying information on a product as described above often does not make it possible to increase the customer's motivation to purchase the product. Sales staffs serve customers based on know-how and advices of other sales staffs; however, because it depends largely on the skills of the sales staffs, each sales staff does not appropriately serve a customer currently. For this reason, it is needed to provide information, or the like, that assist sales staffs in serving according to the relationship between the customer and the product. For example, it is preferable that information corresponding to a relationship between a first object corresponding to a customer and a second object corresponding to a product be presented to a sales staff and the customer. Accordingly, it is an object in one aspect of an embodiment of the invention to provide an information processing program, an information processing method, and an information processing device that make it possible to provide information corresponding to a relationship between a first object correspondin