US-12620255-B2 - Image processing apparatus, image processing method, and non-transitory storage medium

US12620255B2US 12620255 B2US12620255 B2US 12620255B2US-12620255-B2

Abstract

To accurately determine a work content at an image capturing time point by an image analysis, the present invention provides an image processing apparatus 10 including an acquisition unit 11 that acquires an image, a detection unit 12 that detects a hand of a person and a work target object from the image, and a determination unit 13 that determines, based on a relative position relation within the image between the detected hand of the person and the detected work target object, a work content at a time point when the image is captured.

Inventors

Tetsuo Inoshita

Assignees

NEC CORPORATION

Dates

Publication Date: 20260505
Application Date: 20230628
Priority Date: 20220705

Claims (19)

1 . An image processing apparatus comprising: at least one memory configured to store one or more instructions; and at least one processor configured to execute the one or more instructions to: acquire an image; detect a hand of a person and a work target object from the image; and determine, based on a relative position relation within the image between the detected hand of the person and the detected work target object, a work content at a time point when the image is captured, wherein the work target object is a physical object.
2 . The image processing apparatus according to claim 1 , wherein the processor is further configured to execute the one or more instructions to: acquire a plurality of the time-series images; detect the hand of the person and the work target object from each of a plurality of the time-series images; determine the work content at a time point when each of a plurality of the time-series images is captured; and generate evaluation information indicating an evaluation content of work, based on a determination result of the work content at a time point when each of a plurality of the time-series images is captured.
3 . The image processing apparatus according to claim 2 , wherein the evaluation information indicates at least one of a total time in which work is performed, and a total time in which work is not performed.
4 . The image processing apparatus according to claim 2 , wherein the evaluation information indicates a total time in which each of a plurality of pieces of work is performed.
5 . The image processing apparatus according to claim 1 , wherein the processor is further configured to execute the one or more instructions to determine, in the determination of the work content, whether work is performed.
6 . The image processing apparatus according to claim 5 , wherein, the processor is further configured to execute the one or more instructions to determine that work is performed, when the detected hand of the person overlaps the detected work target object.
7 . The image processing apparatus according to claim 1 , wherein the processor is further configured to execute the one or more instructions to determine, in the determination of the work content, which piece of work is performed among a plurality of pieces of work.
8 . The image processing apparatus according to claim 7 , wherein the processor is further configured to execute the one or more instructions to determine which piece of work is performed among a plurality of pieces of work, based on at least one of: a relative position relation within the image between the detected hand of the person and the detected work target object, a kind of the detected work target object, an orientation of the detected work target object, a shape of the detected work target object, color information of the detected work target object, a position of the detected work target object within the image, a position of the detected hand of the person within the image, a shape of the detected hand of the person, and a kind of a tool held by the detected hand of the person.
9 . The image processing apparatus according to claim 1 , wherein the processor is further configured to execute the one or more instructions to: detect the work target object from the image, based on a previously registered reference image of the work target object; and newly register, as the reference image, the image satisfying a predetermined condition.
10 . The image processing apparatus according to claim 9 , wherein the predetermined condition includes a condition that a degree of similarly to a comparison target image computed based on at least one of a shape, a size, or color information of the work target object is less than a criterion value.
11 . The image processing apparatus according to claim 9 , wherein the predetermined condition is being captured at a predetermined timing.
12 . The image processing apparatus according to claim 11 , wherein the predetermined timing includes at least one of: a timing when a position of the work target object within the image changes, a timing when a predetermined event is detected within sensing data of a pressure sensor installed on a table on which a work target object is put, a timing when a predetermined sound is detected within sound data recorded during work, a timing when a predetermined keyword is detected within sound data recorded during work, and a timing when a predetermined action by a worker is detected within the image.
13 . The image processing apparatus according to claim 9 , wherein the processor is further configured to execute the one or more instructions to newly register, as the reference image, an image in which the work target object does not overlap the hand of the person, or an image in which a work target object overlaps the hand of the person, but an extent of the overlap is smaller than a criterion, among the images satisfying the predetermined condition.
14 . The image processing apparatus according to claim 1 , wherein the processor is further configured to execute the one or more instructions to: acquire a plurality of the time-series images; detect the hand of the person from each of a plurality of the time-series images, and also detect the work target object, based on a previously registered reference image of the work target object; in the detection of the work target object: detect the work target object from the image of a processing target by utilizing some of a plurality of the reference images, images; and determine, based on the reference image utilized when the work target object is detected from the image before the image of a processing target in a time-series order, the reference image to be utilized when the work target object is detected from the image of the processing target.
15 . An image processing method comprising, by one or more computers: acquiring an image; detecting a hand of a person and a work target object from the image; and determining, based on a relative position relation within the image between the detected hand of the person and the detected work target object, a work content at a time point when the image is captured, wherein the work target object is a physical object.
16 . A non-transitory storage medium storing a program causing a computer to: acquire an image; detect a hand of a person and a work target object from the image; and determine, based on a relative position relation within the image between the detected hand of the person and the detected work target object, a work content at a time point when the image is captured, wherein the work target object is a physical object.
17 . The image processing apparatus according to claim 1 , wherein the work target object is the physical object acted upon by at least one work operation performed by the detected hand.
18 . The image processing method according to claim 15 , wherein the work target object is the physical object acted upon by at least one work operation performed by the detected hand.
19 . The non-transitory storage medium according to claim 16 , wherein the work target object is the physical object acted upon by at least one work operation performed by the detected hand.

Description

This application is based upon and claims the benefit of priority from Japanese patent application No. 2022-108465, filed on Jul. 5, 2022, the disclosure of which is incorporated herein in its entirety by reference. TECHNICAL FIELD The present invention relates to an image processing apparatus, an image processing method, and a program. BACKGROUND ART Techniques being related to the present invention are disclosed in Patent Documents 1 to 3. Patent Document 1 (Japanese Patent Application Publication No. 2020-184250) discloses a technique for detecting a position and an orientation of a worker, determining, based on a detection result, which worktable the worker works on, and measuring, based on a determination result, a work time at each worktable. Patent Document 2 (Japanese Patent Application Publication No. 2020-135767) discloses a technique for detecting a position of a worker and a position of a target object, determining that work is being performed when the positions of the worker and the target object are associated with each other, and measuring a work time. Patent Document 3 (Japanese Patent Application Publication No. 2020-177533) discloses a technique for measuring a work time, based on an image capturing a hand of a worker. The technique preliminarily registers, as a background image, a state where work is not performed and there is no motion on a worktable, determines, to be a working state, a state where there is a difference from the background image, and determines, to be a non-working state, a state where there is no difference from the background image. Then, the technique measures a work time, based on a result of the determination. DISCLOSURE OF THE INVENTION A technique for determining a work content at an image capturing time point by an image analysis is desired. For example, measurement of a work time of each piece of work or the like can be performed by utilizing the technique. However, it is not easy to accurately determine a work content at an image capturing time point by an image analysis. A technique disclosed in Patent Document 1 measures a work time at each worktable, based on a position and an orientation of a worker. However, a situation where, even when a worker is directed toward a certain worktable near the worktable, work is not performed actually can occur. In determination based on a position and an orientation of a worker, a work content of the worker cannot be accurately determined. A technique disclosed in Patent Document 2 determines that work is being performed when a position of a worker and a position of a target object are associated with each other, and measures a work time. However, a situation where, even when a worker and a target object are present in the same work space, work is not performed actually can occur. In determination based on a position relation between a worker and a target object, a work content of the worker cannot be accurately determined. A technique disclosed in Patent Document 3 determines, to be a working state, a state where there is a difference from a background image, and determines, to be a non-working state, a state where there is no difference from the background image. In this case, for example, a state where only a hand of a worker put on a worktable is captured, a state where only an object put on a worktable is captured, or the like is also determined to be a working state. In determination based on a difference from a background image, a work content of the worker cannot be accurately determined. In view of the problem described above, one example of an object of the present invention is to provide an image processing apparatus, an image processing method, and a program that solve a subject of accurately determining a work content at an image capturing time point by an image analysis. One aspect of the present invention provides an image processing apparatus including: an acquisition unit that acquires an image;a detection unit that detects a hand of a person and a work target object from the image; anda determination unit that determines, based on a relative position relation within the image between the detected hand of the person and the detected work target object, a work content at a time point when the image is captured. One aspect of the present invention provides an image processing method including, by one or more computers:acquiring an image;detecting a hand of a person and a work target object from the image; anddetermining, based on a relative position relation within the image between the detected hand of the person and the detected work target object, a work content at a time point when the image is captured. One aspect of the present invention provides a program causing a computer to function as: an acquisition unit that acquires an image;a detection unit that detects a hand of a person and a work target object from the image; anda determination unit that determines, based on a relative position relation wi