US-12620230-B2 - Non-transitory computer-readable recording medium, information processing method, and information processing apparatus for detecting fraud at accounting machine

US12620230B2US 12620230 B2US12620230 B2US 12620230B2US-12620230-B2

Abstract

An information processing apparatus acquires video image data on a person who is scanning a code of a commodity product to an accounting machine, and specifies, from the acquired video image data by analyzing the acquired video image data, a region of a hand of the person and a region of the commodity product that is being gripped in the hand of the person. The information processing apparatus tracks either a movement of the hand of the person that is gripping the commodity product, or, a movement of the gripped commodity product, and generates, based on a change in the tracked movement of the hand or a change in the tracked movement of the commodity product, an alert connected to an abnormality of a behavior of registering the commodity product to the accounting machine.

Inventors

RYO ISHIDA
Daisuke Uchida
Yasuhiro Aoki

Assignees

FUJITSU LIMITED

Dates

Publication Date: 20260505
Application Date: 20230731
Priority Date: 20221207

Claims (12)

1 . A non-transitory computer-readable recording medium having stored therein an information processing program that causes a computer to execute a process comprising: acquiring video image data on a person who is scanning a code of a commodity product to an accounting machine; specifying, from the acquired video image data by analyzing the acquired video image data, a first region that includes a hand of the person and a second region that includes the commodity product that is being gripped in the hand of the person; tracking either a movement of the hand of the person that is gripping the commodity product, or a movement of the gripped commodity product; and generating, based on a change in the tracked movement of the hand or a change in the tracked movement of the commodity product, an alert connected to an abnormality of a behavior of registering the commodity product to the accounting machine, wherein the specifying includes specifying, from the acquired video image data by inputting the acquired video image data to a machine learning model, the first region and the second region, the tracking includes tracking, when a number of commodity products that are being gripped in the hand of the person is plural, either the movement of the hand of the person that is gripping the plurality of commodity products, or, the movement of the plurality of commodity products, the generating the alert includes generating, based on the change in the tracked movement of the hand or a change in the tracked movement of the plurality of commodity products, the alert connected to an abnormality of a behavior of registering the commodity products to the accounting machine, the machine learning model outputs, in accordance with an input of the video image data, a bounding box that indicates the first region, a bounding box of the commodity product that indicates the second region, and skeleton information on the hand of the person, the specifying includes generating, by inputting the acquired video image data to the machine learning model, the bounding box that indicates the first region of the hand, the bounding box of the commodity product that indicates the second region of the commodity product, and the skeleton information on the hand, the tracking includes determining, when a number of the generated bounding boxes of the commodity product is plural, whether or not positions of the plurality of bounding boxes are overlapped, and the generating includes identifying, when it is determined that the overlapped positions are present, a pattern of the movement of the hand of the person based on the skeleton information on the hand of the person, and generating, by comparing the identified pattern of the movement of the hand with a rule that is set in advance, the alert related to the abnormality of the behavior of registering the commodity products to the accounting machine.
2 . The non-transitory computer-readable recording medium according to claim 1 , wherein the generating includes determining, when the movement of the commodity products has been tracked, whether or not a change in the movement of the plurality of commodity products included in the second region of the commodity product corresponds to a pattern that is assigned in advance, and generating, when the change in the movement of the plurality of commodity products corresponds to the pattern, the alert connected to the abnormality of the behavior of registering the commodity products to the accounting machine.
3 . The non-transitory computer-readable recording medium according to claim 1 , wherein the generating includes generating, when the movement of the hand has been tracked, and in a case where the movement of the hand corresponds to a motion of piling up the plurality of commodity products, the alert connected to the abnormality of the behavior of registering the commodity products to the accounting machine.
4 . The non-transitory computer-readable recording medium according to claim 1 , wherein the machine learning model specifies, in accordance with an input of the video image data, the first region, the second region, and a relationship between the first region and the second region, the specifying includes specifying, based on the first region, the second region, and the relationship that have been specified by inputting the acquired video image data to the machine learning model, a first motion of gripping a first commodity product in a right hand of the person and a second motion of gripping a second commodity product in a left hand, the tracking includes tracking the first motion and the second motion, and the generating includes detecting, based on the tracked first motion and the tracked second motion, that the code of the commodity product of either the first commodity product or the second commodity product has not been scanned to the accounting machine.
5 . The non-transitory computer-readable recording medium according to claim 4 , wherein, when it is determined that the code of the commodity product of each of the first commodity product and the second commodity product has not been scanned to the accounting machine based on a positional relationship between a set of the tracked first motion and the tracked second motion and an area that is set in order to scan the commodity products to the accounting machine, the generating the alert includes generating the alert indicating the abnormality of the behavior of registering the commodity products to the accounting machine.
6 . The non-transitory computer-readable recording medium according to claim 1 , wherein the generating includes generating, as the alert connected to the abnormality of the behavior of registering the commodity product to the accounting machine, an alert indicating that there is a commodity product that has not yet been registered to the accounting machine by the person, or, indicating that the commodity product that has been registered to the accounting machine by the person is abnormal.
7 . The non-transitory computer-readable recording medium according to claim 1 , wherein the process further includes notifying, when the alert connected to the abnormality of the behavior of registering the commodity product to the accounting machine is generated, a terminal used by a store clerk of identification information on the accounting machine and the generated alert in an associated manner.
8 . The non-transitory computer-readable recording medium according to claim 1 , wherein the generating includes outputting, in a case where the alert connected to the abnormality of the behavior of registering the commodity product to the accounting machine has been generated, voice or a screen that makes the person located at the accounting machine aware of a registration omission of the commodity product from the accounting machine.
9 . The non-transitory computer-readable recording medium according to claim 1 , wherein the process further includes: capturing, when the alert connected to the abnormality of the behavior of registering the commodity product to the accounting machine has been generated, the person by a camera included in the accounting machine; and storing image data on the captured person and the alert in an associated manner in a storage.
10 . The non-transitory computer-readable recording medium according to claim 1 , wherein the accounting machine is a self-service checkout terminal, the specifying includes specifying, based on the acquired video image data, the first region from the acquired video image data, the tracking includes tracking, when a number of commodity products that are being gripped in the hand of the person is plural, the movement of the hand of the person that is gripping the plurality of commodity products, and the generating includes generating, based on the change in the tracked movement of the hand, the alert connected to the abnormality of the behavior of registering the commodity products to the self-service checkout terminal.
11 . An information processing method comprising: acquiring video image data on a person who is scanning a code of a commodity product to an accounting machine; specifying, from the acquired video image data by analyzing the acquired video image data, a first region that includes a hand of the person and a second region that includes the commodity product that is being gripped in the hand of the person; tracking either a movement of the hand of the person that is gripping the commodity product, or, a movement of the gripped commodity product; and generating, based on a change in the tracked movement of the hand or a change in the tracked movement of the commodity product, an alert connected to an abnormality of a behavior of registering the commodity product to the accounting machine, using a processor, wherein the specifying includes specifying, from the acquired video image data by inputting the acquired video image data to a machine learning model, the first region and the second region, the tracking includes tracking, when a number of commodity products that are being gripped in the hand of the person is plural, either the movement of the hand of the person that is gripping the plurality of commodity products, or, the movement of the plurality of commodity products, the generating the alert includes generating, based on the change in the tracked movement of the hand or a change in the tracked movement of the plurality of commodity products, the alert connected to an abnormality of a behavior of registering the commodity products to the accounting machine, the machine learning model outputs, in accordance with an input of the video image data, a bounding box that indicates the first region, a bounding box of the commodity product that indicates the second region, and skeleton information on the hand of the person, the specifying includes generating, by inputting the acquired video image data to the machine learning model, the bounding box that indicates the first region of the hand, the bounding box of the commodity product that indicates the second region of the commodity product, and the skeleton information on the hand, the tracking includes determining, when a number of the generated bounding boxes of the commodity product is plural, whether or not positions of the plurality of bounding boxes are overlapped, and the generating includes identifying, when it is determined that the overlapped positions are present, a pattern of the movement of the hand of the person based on the skeleton information on the hand of the person, and generating, by comparing the identified pattern of the movement of the hand with a rule that is set in advance, the alert related to the abnormality of the behavior of registering the commodity products to the accounting machine.
12 . An information processing apparatus comprising: a memory; and a processor coupled to the memory and configured to: acquire video image data on a person who is scanning a code of a commodity product to an accounting machine; specify, from the acquired video image data by analyzing the acquired video image data, a first region that includes a hand of the person and a second region that includes the commodity product that is being gripped in the hand of the person; track either a movement of the hand of the person that is gripping the commodity product, or, a movement of the gripped commodity product; and generate, based on a change in the tracked movement of the hand or a change in the tracked movement of the commodity product, an alert connected to an abnormality of a behavior of registering the commodity product to the accounting machine, wherein the processor further configures to: specify, from the acquired video image data by inputting the acquired video image data to a machine learning model, the first region and the second region, track, when a number of commodity products that are being gripped in the hand of the person is plural, either the movement of the hand of the person that is gripping the plurality of commodity products, or, the movement of the plurality of commodity products, generate, based on the change in the tracked movement of the hand or a change in the tracked movement of the plurality of commodity products, the alert connected to an abnormality of a behavior of registering the commodity products to the accounting machine, wherein the machine learning model outputs, in accordance with an input of the video image data, a bounding box that indicates the first region, a bounding box of the commodity product that indicates the second region, and skeleton information on the hand of the person, wherein the processor further configures to: generate, by inputting the acquired video image data to the machine learning model, the bounding box that indicates the first region of the hand, the bounding box of the commodity product that indicates the second region of the commodity product, and the skeleton information on the hand, determine, when a number of the generated bounding boxes of the commodity product is plural, whether or not positions of the plurality of bounding boxes are overlapped, identify, when it is determined that the overlapped positions are present, a pattern of the movement of the hand of the person based on the skeleton information on the hand of the person, and generate, by comparing the identified pattern of the movement of the hand with a rule that is set in advance, the alert related to the abnormality of the behavior of registering the commodity products to the accounting machine.

Description

CROSS-REFERENCE TO RELATED APPLICATION This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2022-195932, filed on Dec. 7, 2022, the entire contents of which are incorporated herein by reference. FIELD The embodiments discussed herein are related to an information processing program, an information processing method, and an information processing apparatus. BACKGROUND In stores, such as supermarkets and convenience stores, self-service checkout registers are becoming widely used. A self-service checkout register is a point of sale (POS) checkout register system in which a user who purchases commodity products perform a series of processes between a process of reading bar code assigned to each of the commodity products and a process of calculating a payment amount. For example, by installing the self-service checkout register, it is possible to improve a labor shortage caused by a decrease in population and suppress labor costs. Patent Document 1: Japanese Laid-open Patent Publication No. 2020-53019 SUMMARY According to an aspect of an embodiment, a non-transitory computer-readable recording medium stores therein an information processing program that causes a computer to execute a process. The process includes acquiring video image data on a person who is scanning a code of a commodity product to an accounting machine, specifying, from the acquired video image data by analyzing the acquired video image data, a region of a hand of the person and a region of the commodity product that is being gripped in the hand of the person, tracking either a movement of the hand of the person that is gripping the commodity product, or, a movement of the gripped commodity product, and generating, based on a change in the tracked movement of the hand or a change in the tracked movement of the commodity product, an alert connected to an abnormality of a behavior of registering the commodity product to the accounting machine. The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention. BRIEF DESCRIPTION OF DRAWINGS However, in the technology described above, it is difficult to detect a fraud. For example, at an accounting machine, such as a self-service checkout register, an inevitable error, an intentional fraud, or the like is made by a user, thus resulting in incurring an unpaid amount or the like. An example of the inevitable error includes a scan omission of a commodity product as a result of a user forgetting to scan the commodity product and moving the commodity product from a basket to a checkout bag, and the like. In addition, examples of the intentional fraud include a bar code hiding occurring when a user pretends to scan a commodity product by hiding only a bar code by the user's fingers, a read error occurring when a user erroneously reads a bar code assigned to a can instead of a bar code assigned to a beer case as a result of different bar codes being assigned to each of the beer case containing, for example, a set of six cans and the individual cans, and the like. In addition, it is conceivable to detect a fraud by automatically counting the number of commodity products by installing a weight sensor in each of the self-service checkout registers; however, a cost is excessively high, and this method is unrealistic for, in particular, large-scale stores and stores having operations across the country. In addition, at the self-service checkout register, there is another aspect in which it is difficult to detect a fraudulent act because a scan of a commodity product code or calculation of a payment amount is entrusted by a user himself or herself. For example, even if image recognition Artificial Intelligence (AI) is used in an aspect of detecting the fraudulent act described above, a huge amount of training data is needed for training of the image recognition AI. However, in stores, such as supermarkets and convenience stores, many types of commodity products are present, and, in addition, a life cycle of each of the commodity products is short, so that a replacement of each of commodity products frequently occurs. It is difficult to tune the image recognition AI in accordance with the life cycle of these types of commodity products, or it is difficult to perform training of new image recognition AI. FIG. 1 is a diagram illustrating an example of the overall configuration of a self-service checkout register system according to a first embodiment; FIG. 2 is a diagram illustrating a detection example of an abnormal behavior according to the first embodiment; FIG. 3 is a functional block diagram illustrating a functional configuration of an information processing apparatus according to the fir