JP-2026075990-A - Image processing device, image processing method, and program

Abstract

[Problem] To provide an image processing device that can perform hand masking with high precision. [Solution] The image processing device includes a hand tracking unit 102 that performs hand tracking by analyzing an image, an estimation unit 103 that estimates the shape or posture of the hand from the tracking results of the hand tracking unit 102, and image processing units 104 to 106 that perform image processing based on the shape or posture of the hand estimated by the estimation unit 103. [Selection Diagram] Figure 1

Inventors

• 小川 将人

Assignees

• キヤノン株式会社

Dates

Publication Date

20260511

Application Date

20241023

Claims (6)

1. A hand tracking unit that performs hand tracking by analyzing images, An estimation unit that estimates the shape or posture of the hand from the tracking results of the hand tracking unit, An image processing apparatus characterized by having an image processing unit that performs image processing based on the shape or posture of the hand estimated by the estimation unit.
2. The aforementioned image processing unit, A color registration unit extracts skin tones from the aforementioned image and generates a color registration table, A hand masking unit that performs hand masking using the aforementioned color registration table, The image processing apparatus according to claim 1, further comprising: a switching unit that switches the color registration table used by the hand masking unit according to the hand shape or posture estimated by the estimation unit.
3. The image processing unit includes a merge color registration table generation unit that generates a merge color registration table from the color registration table, The switching unit switches to the merge color registration table if the estimation unit is unable to estimate the shape or posture of the hand. The image processing apparatus according to claim 2, characterized in that the hand masking unit performs hand masking using the merge color registration table.
4. The image processing apparatus according to claim 2 or 3, characterized in that the image processing unit has a color registration table prediction and generation unit that predicts and automatically generates a color registration table corresponding to a hand shape or posture different from that which has been registered, from the color registration table.
5. The hand tracking step involves performing hand tracking by analyzing images, An estimation step for estimating the shape or posture of the hand from the tracking results of the hand tracking step, An image processing method characterized by comprising: an image processing step of performing image processing based on the shape or posture of the hand estimated in the estimation step.
6. A program for causing a computer to execute each step of the image processing method described in claim 5.

Description

This disclosure relates to an image processing apparatus, an image processing method, and a program. Head-mounted displays (HMDs) that allow users to experience mixed reality are being sold. A technique called hand masking is used to represent the spatial relationship between hands and computer graphics. Hand masking is a technique that involves pre-registering hand colors, recognizing the area of the hand as the hand area, and masking the area of the CG corresponding to the hand area when the hand is in front of the CG background image, thereby making the hand visible to the HMD user. Japanese Patent Publication No. 2005-228140 This is a block diagram illustrating the first embodiment.This figure shows an example of a bone for hand tracking.This figure shows an example of a color registration table based on hand shape and posture.This is a flowchart illustrating the first embodiment.This is a block diagram illustrating the second embodiment.This figure shows an example of a merge color registration table.This is a flowchart illustrating the second embodiment.This is a block diagram illustrating a third embodiment.This figure shows an example of predicting a color registration table.This is a flowchart of the color registration table prediction generation unit. Preferred embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. Figure 1 is a block diagram relating to an embodiment of this disclosure. (First embodiment) Hereinafter, an image processing apparatus according to a first embodiment of the present disclosure will be described with reference to Figure 1. The image processing apparatus is composed of the components described below. The functions of each component are configured by one or more CPUs (central processing units) that function as the control unit of the image processing apparatus executing programs. The components of the image processing apparatus may be composed of integrated circuits or the like, as long as they perform similar functions. The imaging unit 101 acquires real-world video footage to be used for hand masking. The hand tracking unit 102 performs hand tracking. Hand tracking is a technology that detects and tracks the position of hands and fingers in the video. The detected information such as the position of the hands and fingers (tracking results) is visualized using a method called bones. Figure 2 shows an example of bones. The joints of the hands and fingers are represented by spheres 201, and the shape of the hand is reproduced by connecting them with lines 202. Hand tracking can be done by analyzing images taken with a regular camera, infrared camera, or TOF sensor to detect the position of the hands and fingers, or by wearing gloves equipped with sensors to detect movement. In this embodiment, the method of analyzing images taken with a camera is used. In addition to acquiring and using images taken with a camera, images from a wider-angle camera or TOF sensor may also be acquired and used by the imaging unit 101. The hand shape and posture estimation unit 103 estimates the hand shape or posture from the shape of the bones detected by the hand tracking unit 102. Here, it is estimated whether the hand is open or closed, whether the palm is facing forward or backward, etc. This embodiment is characterized by having an image processing unit (corresponding to 104-106 in Figure 1) that performs image processing based on this estimation result. The color registration table switching unit 104 switches to a color registration table corresponding to the hand shape and posture estimated by the hand shape and posture estimation unit 103. Figure 3 is an example of a color registration table corresponding to hand shape and posture. Hands 300, 310, 320, 330, and 340 represent examples of hand shapes and postures, with hand 300 being open with the palm facing forward, hand 310 being palm facing backward, hand 320 being closed with the palm facing forward, hand 330 being palm facing backward, and hand 340 being a pointing shape. Graphs 301, 311, 321, 331, and 341 represent the a*b* plane of the color registration table. This embodiment describes the case where the color registration table is represented in the L*a*b* color space. It may also be represented in RGB space, YUV space, etc. The a*b* plane represents the L*a*b* color space as viewed from directly above. The skin tone of the hand is in the first quadrant of the a*b* plane, corresponding to regions 302, 312, 322, 332, and 342, and it can be seen that the regions differ slightly depending on the shape and posture of the hand. For example, in the case of hand 320 (closed with palm facing forward), region 322 is wide, and in the case of hand 330 (closed with palm facing away), region 332 is narrow. This is because hand 320 includes various skin tones such as the palm, back of the hand, and shadow. Graphs 303, 313, 323, 333, and 343 represent the L*a* plane of the color registrat