CN-122029580-A - Action analysis device, program, and action analysis method

Abstract

The operation analysis device (100) is provided with a joint position information acquisition unit (103) which acquires joint position information indicating the positions of M joints (M is an integer of 2 or more) of an operator included in an image, an operation determination unit (105) which uses one of one or more learning models for estimating an operation being performed by a person based on at least the positions of the joints of the person, performs estimation of the operation being performed by the operator based on the positions of the M joints, and determines the operation being performed by the operator based on the estimation result, and an object selection unit (106) which selects N joints (N is an integer satisfying 1≤N < M) from the M joints when the reliability of the estimation is less than a predetermined threshold, and uses the positions of the N joints as an object of use, and the operation determination unit (105) uses one of the one or more learning models when the reliability of the estimation is less than the predetermined threshold, and performs the operation being performed again based on the object of use.

Inventors

• Sayaka Sayaka
• KUSANO KATSUO
• Shimizu Naogo
• KODAIRA TAKAYUKI

Assignees

• 三菱电机株式会社

Dates

Publication Date

20260512

Application Date

20231012

Claims (17)

1. 1. An operation analysis device is characterized by comprising: A joint position information acquisition unit that acquires joint position information indicating positions of M joints (M is an integer of 2 or more) of an operator included in an image; a job determination unit that uses one of one or more learned models for estimating a job being performed by a person based at least on the positions of joints of the person, estimates a job being performed by the worker based on the positions of the M joints, and determines a job being performed by the worker based on the result of the estimation, and A target selection unit that selects N joints (N is an integer satisfying 1≤N < M) from among the M joints when the estimated reliability is less than a predetermined threshold, and uses the positions of the N joints as targets of use, When the reliability of the estimation is less than a predetermined threshold, the job determination unit uses one of the one or more learned models to estimate again the job being performed by the worker based on the usage object.
2. 2. The motion analysis apparatus according to claim 1, wherein, The use object selecting unit selects a predetermined joint as the N joints.
3. 3. The motion analysis apparatus according to claim 1, wherein, The motion analysis device further includes an object position information acquisition unit that acquires object position information indicating a position of an object included in the image, The usage object selecting unit selects, as the N joints, a joint that corresponds to the object in advance, and uses the positions of the N joints and the position of the object as the usage object.
4. 4. The motion analysis apparatus according to claim 1, wherein, The motion analysis device further comprises: An object position information acquisition unit for acquiring object position information indicating the position of an object included in the image, and A distance calculating unit that calculates distances between the position of the object and the positions of the M joints, The object selection unit selects, as the N joints, a joint whose distance is smaller than a predetermined threshold.
5. 5. The motion analysis apparatus according to claim 4, wherein, The usage object selecting unit also uses the position of the object as the usage object.
6. 6. The motion analysis apparatus according to claim 1, wherein, The motion analysis device further comprises: An object position information acquisition unit for acquiring object position information indicating the position of an object included in the image, and A distance calculating unit that calculates distances between the position of the object and the positions of the M joints, The usage object selecting unit may select, as the usage object, positions of the N joints selected from among the joints having a distance smaller than a predetermined threshold value, and may select, as the N joints, a predetermined joint when the N joints cannot be selected from among the joints having a distance smaller than the predetermined threshold value.
7. 7. The motion analysis apparatus according to claim 6, wherein, When the N joints can be selected from the joints having the distances smaller than a predetermined threshold, the use object selecting unit may set the position of the object as the use object.
8. 8. The motion analysis apparatus according to claim 6 or 7, wherein, The operation analysis device further includes a job defining section that, when the N joints can be selected from among the joints having the distances smaller than a predetermined threshold, determines one or more jobs corresponding to the object as candidate jobs by referring to correspondence information that associates each of the plurality of jobs with at least any one of the plurality of objects, When the N joints can be selected from the joints whose distances are smaller than a predetermined threshold, the job specifying unit specifies a job that is inferred from the use object and matches the candidate job as a job of the operator.
9. 9. An operation analysis device is characterized by comprising: A joint position information acquisition unit that acquires joint position information indicating positions of M joints (M is an integer of 2 or more) of an operator included in an image; an object position information acquisition unit that acquires object position information indicating a position of an object included in the image; a distance calculating unit that calculates distances between the position of the object and the positions of the M joints; A utilization object selecting unit for selecting N joints from the joints with the distance smaller than a predetermined threshold value (N is an integer satisfying 1≤N < M), and taking the positions of the N joints as utilization objects, and And a work specification unit that uses a learning model for estimating a work being performed by the person from a position of a joint of the person and a position of an object, and that performs estimation of the work being performed by the worker based on the object to be used, and that specifies the work being performed by the worker based on a result of the estimation.
10. 10. An operation analysis device is characterized by comprising: A joint position information acquisition unit that acquires joint position information indicating positions of M joints (M is an integer of 2 or more) of an operator included in an image; an object position information acquisition unit that acquires object position information indicating a position of an object included in the image; a distance calculating unit that calculates distances between the position of the object and the positions of the M joints; A use object selection unit that, when N joints can be selected from among the joints having a distance smaller than a predetermined threshold (N is an integer satisfying 1≤N < M), uses the positions of the N joints selected from among the joints having a distance smaller than the predetermined threshold as a use object, when N joints cannot be selected from among the joints having a distance smaller than the predetermined threshold, selects the N joints from among the predetermined joints, uses the positions of the N joints as the use object, and A work specifying unit that uses one of one or more learned models for estimating a work being performed by a person based at least on a position of a joint of the person, performs estimation of the work being performed by the worker based on the use object, and specifies the work being performed by the worker based on a result of the estimation.
11. 11. The motion analysis apparatus according to claim 10, wherein, The operation analysis device further includes a job defining section that, when the N joints can be selected from among the joints having the distances smaller than a predetermined threshold, determines one or more jobs corresponding to the object as candidate jobs by referring to correspondence information that associates each of the plurality of jobs with at least any one of the plurality of objects, When the N joints can be selected from the joints whose distances are smaller than a predetermined threshold, the job specifying unit specifies a job that is inferred from the use object and matches the candidate job as a job of the operator.
12. 12. A program for causing a computer to function as a joint position information acquisition unit, a job specification unit, and a target selection unit, The joint position information acquisition unit acquires joint position information indicating positions of M joints (M is an integer of 2 or more) of an operator included in an image, The job determination unit uses one of one or more learned models for estimating a job being performed by a person based at least on the positions of the joints of the person, estimates a job being performed by the worker based on the positions of the M joints, determines a job being performed by the worker based on the results of the estimation, The usage object selecting unit selects N joints (N is an integer satisfying 1≤N < M) from the M joints when the estimated reliability is smaller than a predetermined threshold, uses the positions of the N joints as usage objects, The job determination unit may use one of the one or more learned models to again estimate, based on the usage object, a job being performed by the operator when the estimated reliability is less than a predetermined threshold.
13. 13. A program for causing a computer to function as a joint position information acquisition unit, an object position information acquisition unit, a distance calculation unit, a use object selection unit, and a work specification unit, The joint position information acquisition unit acquires joint position information indicating positions of M joints (M is an integer of 2 or more) of an operator included in an image, The object position information acquisition unit acquires object position information indicating a position of an object included in the image, The distance calculating section calculates respective distances between the position of the object and the positions of the M joints, The object selection unit selects N joints (N is an integer satisfying 1≤N < M) from among the joints having the distance smaller than a predetermined threshold, sets the positions of the N joints as objects to be used, The work specification unit uses a learning model for estimating a work being performed by a person based on a position of a joint of the person and a position of an object, performs estimation of the work being performed by the worker based on the use object, and specifies the work being performed by the worker based on a result of the estimation.
14. 14. A program for causing a computer to function as a joint position information acquisition unit, an object position information acquisition unit, a distance calculation unit, a use object selection unit, and a work specification unit, The joint position information acquisition unit acquires joint position information indicating positions of M joints (M is an integer of 2 or more) of an operator included in an image, The object position information acquisition unit acquires object position information indicating a position of an object included in the image, The distance calculating section calculates respective distances between the position of the object and the positions of the M joints, The object selection unit selects N joints from among the joints having a distance smaller than a predetermined threshold (N is an integer satisfying 1≤N < M), wherein the N joints are selected from among the joints having a distance smaller than the predetermined threshold, and the N joints are selected from among the joints having a distance smaller than the predetermined threshold, The job specifying unit uses one of one or more learned models for estimating a job being performed by a person based at least on the position of the joint of the person, performs estimation of the job being performed by the worker based on the use object, and specifies the job being performed by the worker based on the result of the estimation.
15. 15. A motion analysis method is characterized in that, Acquiring joint position information indicating positions of M joints (M is an integer of 2 or more) of an operator included in an image, Using one of the one or more learned models for inferring a task being performed by a person based at least on the positions of the joints of the person, inferring a task being performed by the worker based on the positions of the M joints, determining a task being performed by the worker based on the results of the inferring, When the reliability of the estimation is less than a predetermined threshold, N joints are selected from the M joints (N is an integer satisfying 1≤N < M), the positions of the N joints are used as the utilization objects, And if the reliability of the estimation is less than a predetermined threshold, estimating again the work being performed by the operator from the use object using one of the one or more learned models.
16. 16. A motion analysis method is characterized in that, Acquiring joint position information indicating positions of M joints (M is an integer of 2 or more) of an operator included in an image, Object position information indicating the position of an object included in the image is acquired, Calculating respective distances between the position of the object and the positions of the M joints, Selecting N joints from the joints with the distance smaller than a preset threshold value (N is an integer which satisfies 1≤N < M), taking the positions of the N joints as the utilization objects, Using a learning model for estimating an ongoing operation of a person from a position of a joint of the person and a position of an object, estimating an ongoing operation of the worker from the utilization object, and determining an ongoing operation of the worker from a result of the estimating.
17. 17. A motion analysis method is characterized in that, Acquiring joint position information indicating positions of M joints (M is an integer of 2 or more) of an operator included in an image, Object position information indicating the position of an object included in the image is acquired, Calculating respective distances between the position of the object and the positions of the M joints, When N joints can be selected from the joints having a distance smaller than a predetermined threshold value (N is an integer satisfying 1≤N < M), the positions of the N joints selected from the joints having a distance smaller than the predetermined threshold value are used as the object of use, When the N joints cannot be selected from the joints with the distances smaller than the preset threshold value, the N joints are selected from the preset joints, the positions of the N joints are taken as the utilization objects, Using one of one or more learned models for inferring a task being performed by a person based at least on the position of the person's joints, inferring a task being performed by the worker based on the utilization object, and determining a task being performed by the worker based on the result of the inference.

Description

Action analysis device, program, and action analysis method Technical Field The present disclosure relates to an operation analysis device, a program, and an operation analysis method. Background In a production site of a manufacturing industry, measurement of a working time is performed for the purpose of improving a work by a manual work or the like. The measurement of the work time can be performed with less burden by automatically specifying the work performed by the worker using image recognition. The gesture analysis apparatus described in patent document 1 uses a learned gesture model, region model, background model, and sequence model to determine a job from a job image. Prior art literature Patent literature Patent document 1 Japanese patent application laid-open No. 2020-201772 Disclosure of Invention Problems to be solved by the invention However, in the conventional technique, even in the same operation, the difference in joint position information indicating the positions of the joints of the operator becomes large according to the worker's preference, and the posture data of the operator becomes inaccurate. Therefore, if the operator mapped on the image used for the model learning is different from the operator who determines the job, the conventional technique may not accurately determine the job order. Accordingly, one or more aspects of the present disclosure are directed to accurately determining an operation even if the operation of an operator has a habit. Means for solving the problems The motion analysis device according to claim 1 of the present disclosure is characterized by comprising a joint position information acquisition unit that acquires joint position information indicating positions of M joints (M is an integer of 2 or more) of an operator included in an image, a task determination unit that uses one of one or more learned models for estimating an ongoing task of the person based at least on the positions of the joints of the person, performs estimation of the ongoing task of the operator based on the positions of the M joints, and determines the ongoing task of the operator based on the result of the estimation, and a utilization object selection unit that selects N joints (N is an integer satisfying 1≤N < M) from the M joints when the reliability of the estimation is less than a predetermined threshold, wherein the task determination unit uses one of the one or more learned models as a utilization object, and performs the operation of the ongoing task again based on the estimation object of the operator when the reliability of the estimation is less than the predetermined threshold. The motion analysis device according to claim 2 of the present disclosure is characterized by comprising a joint position information acquisition unit that acquires joint position information indicating positions of M joints (M is an integer of 2 or more) of an operator included in an image, an object position information acquisition unit that acquires object position information indicating positions of an object included in the image, a distance calculation unit that calculates distances between the positions of the object and the positions of the M joints, an object selection unit that selects N joints (N is an integer satisfying 1N < M) from among the joints having the distances smaller than a predetermined threshold, and uses the positions of the N joints as a use object, and a task determination unit that uses a learning model for estimating an operation being performed by the operator from the positions of the joints of the operator and the positions of the object, performs estimation of the operation being performed by the operator from the use object, and determines the operation being performed by the operator from the result of the estimation. The motion analysis device according to claim 3 of the present disclosure is characterized by comprising a joint position information acquisition unit that acquires joint position information indicating positions of M joints (M is an integer of 2 or more) of an operator included in an image, an object position information acquisition unit that acquires object position information indicating positions of an object included in the image, a distance calculation unit that calculates distances between the positions of the object and the positions of the M joints, an object selection unit that uses, when N joints (N is an integer satisfying 1N < M) can be selected from among the joints having the distances smaller than a predetermined threshold, the positions of the N joints selected from among the joints having the distances smaller than a predetermined threshold as a use object, and when the N joints cannot be selected from among the joints having the distances smaller than a predetermined threshold, the N joints are selected from among the predetermined joints, and the N joints are used as the use object, and a determination unit that uses, based o