Search

CN-115702438-B - Emotion determination device, emotion determination method, and program product

CN115702438BCN 115702438 BCN115702438 BCN 115702438BCN-115702438-B

Abstract

The emotion determination device includes 1 st inference means for inferring an emotion of a user based on a change in an expression of the user detected from a facial image of the user, 2 nd inference means for inferring the emotion of the user based on a change in a temperature of the user detected from the user in a noncontact manner, and determination means for determining the emotion of the user based on an inference result of the 1 st inference means and an inference result of the 2 nd inference means.

Inventors

  • SHI YUNAI
  • OHTA, TAKASHI
  • SHUN SAKAI
  • KODA TAKASHI

Assignees

  • 欧姆龙株式会社

Dates

Publication Date
20260505
Application Date
20210519
Priority Date
20200624

Claims (9)

  1. 1. An emotion determining apparatus comprising: a1 st estimating unit that estimates an emotion of a user based on a change in expression of the user detected from a facial image of the user; A2 nd inference unit inferring emotion of the user based on a change in temperature of the user detected from the user in a noncontact manner; a determination unit that determines the emotion of the user based on the estimation result of the 1 st estimation unit and the estimation result of the 2 nd estimation unit, and A changing unit configured to detect a timing at which a variance value in a period until a predetermined time, which is a score indicating a likelihood that the user holds the inferred emotion, is smaller than a predetermined threshold value, change an inference result of the 1 st inference unit and an inference result of the 2 nd inference unit so as to trace back from a1 st timing at which the variance value is smaller than the predetermined threshold value, replace an inference result to a timing immediately after a2 nd timing at which the variance value is smaller than the predetermined threshold value with the inference result of the 1 st timing, The determination unit determines the emotion of the user based on the estimation result after the change by the change unit.
  2. 2. The emotion determination device of claim 1, wherein, The 1 st inference unit and the 2 nd inference unit calculate, for each emotion of a plurality of emotions, a likelihood that the user holds the emotion, i.e., a score, The determination unit determines the emotion of the user based on the score of each emotion calculated by the 1 st inference unit and the score of each emotion calculated by the 2 nd inference unit.
  3. 3. The emotion determination device of claim 2, wherein, In the case where the emotion of the maximum score calculated by the 1 st inference unit coincides with the emotion of the maximum score calculated by the 2 nd inference unit, the determination unit determines the emotion as the emotion of the user, In the case where the emotion of the maximum score calculated by the 1 st inference unit does not coincide with the emotion of the maximum score calculated by the 2 nd inference unit, the determination unit determines the emotion of the user based on the score of each emotion calculated by the 1 st inference unit and the score of each emotion calculated by the 2 nd inference unit.
  4. 4. The emotion determination device as claimed in claim 3, characterized in that, In the case where the emotion of the maximum score calculated by the 1 st inference unit does not coincide with the emotion of the maximum score calculated by the 2 nd inference unit, the judgment unit judges the emotion of which the calculated score is higher among the emotions as the emotion of the user.
  5. 5. The emotion determination device as claimed in claim 3, characterized in that, In the case where the emotion of the maximum score calculated by the 1 st inference unit does not coincide with the emotion of the maximum score calculated by the 2 nd inference unit, the determination unit determines, as the emotion of the user, an emotion of which the value obtained by adding the score calculated by the 1 st inference unit and the score calculated by the 2 nd inference unit is large.
  6. 6. The emotion determination device as claimed in claim 3, characterized in that, When the emotion of the maximum score calculated by the 1 st estimating means does not coincide with the emotion of the maximum score calculated by the 2 nd estimating means, the determining means determines, as the emotion of the user, an emotion of the emotions whose amount of change from the previous score is large.
  7. 7. The emotion determination device as claimed in claim 3, characterized in that, In the case where the emotion of the maximum score calculated by the 1 st inference unit does not coincide with the emotion of the maximum score calculated by the 2 nd inference unit, the determination unit determines an emotion of which the variance value of the score between the plurality of emotions is large as the emotion of the user.
  8. 8. A mood determination method, comprising: a1 st estimating step of estimating an emotion of a user based on a change in expression of the user detected from a facial image of the user; A step 2 of deducing the emotion of the user based on the change in the temperature of the user detected from the user in a noncontact manner; A determination step of determining the emotion of the user based on the result of inference in the 1 st inference step and the result of inference in the 2 nd inference step, and A changing step of detecting a timing at which a variance value in a period until a predetermined time, which is a likelihood that the user holds the inferred emotion, is smaller than a predetermined threshold, and changing the inference result of the 1 st inference step and the inference result of the 2 nd inference step so as to trace back from the 1 st timing at which the variance value is smaller than the predetermined threshold, and replacing the inference result until a timing immediately after the 2 nd timing at which the variance value is smaller than the predetermined threshold with the inference result of the 1 st timing, In the determining step, the emotion of the user is determined based on the estimation result after the changing step.
  9. 9. A program product for causing a computer to execute the steps of the emotion determination method of claim 8.

Description

Emotion determination device, emotion determination method, and program product Technical Field The present invention relates to a technique for determining a user's emotion such as normal, anger, sadness, and happiness. Background For appropriate control and the like, a technique for determining a user's holding emotion has been proposed. If the emotion can be accurately determined, for example, the action (speech or the like) of the communication robot can be controlled to perform appropriate communication with the user. Conventional techniques for determining emotion are disclosed in patent documents 1 to 3, for example. Patent documents 1 to 3 disclose techniques for determining emotion based on various pieces of biological information such as brain waves and heart rhythms measured by wearable devices. Prior art literature Patent literature Patent document 1 Japanese patent application laid-open No. 2014-178970 Patent document 2 Japanese patent application laid-open No. 2017-144222 Patent document 3 Japanese patent application laid-open No. 2012-059107 Disclosure of Invention Problems to be solved by the invention However, in the conventional techniques disclosed in patent documents 1 to 3, the usability is poor because the user must wear a wearable device. In addition, since a lot of biological information is used, the processing load of emotion determination is large, and it takes time for emotion determination. The present invention has been made in view of the above-described circumstances, and an object thereof is to provide a technique capable of accurately determining emotion with a simple configuration having high usability. Means for solving the problems In order to achieve the above object, the present invention adopts the following configuration. The first side of the present invention provides an emotion determination device, comprising a1 st inference unit that infers an emotion of a user based on a change in an expression of the user detected from a facial image of the user, a2 nd inference unit that infers the emotion of the user based on a change in a temperature of the user detected from the user in a noncontact manner, and a determination unit that determines the emotion of the user based on an inference result of the 1 st inference unit and an inference result of the 2 nd inference unit. Here, the face image is an image obtained by capturing a face with an image sensor (camera), a portion of the image obtained by capturing a face with the image sensor, or the like. The temperature is detected by a thermal sensor or the like in a noncontact manner. According to the above configuration, emotion can be determined with high accuracy by a simple configuration based on the result of emotion estimation using a face image and the result of emotion estimation using temperature without using much biological information. In addition, since both the face image and the temperature (detection value) are obtained in a noncontact manner, the user does not have to wear a wearable device, and high usability can be achieved. Further, since there are individual differences in the amount of change in expression, the correspondence between emotion and expression, and the like, it is difficult to accurately determine emotion only from a facial image, and the temperature of a user varies greatly depending on the environment (season, place (indoor/outdoor)), clothing, and it is also difficult to accurately determine emotion only from temperature. According to the above configuration, by combining the result of emotion determination (emotion estimation) using the face image and the result of emotion determination using the temperature, the emotion can be estimated with higher accuracy than the emotion determination of one side. The 1 st inference unit and the 2 nd inference unit may calculate, for each of a plurality of emotions, a score that is a likelihood that the user holds the emotion, and the determination unit may determine the emotion of the user based on the score of each emotion calculated by the 1 st inference unit and the score of each emotion calculated by the 2 nd inference unit. Here, in emotion estimation using a face image and emotion estimation using a temperature, scores are normalized on the same basis, and the ranges of scores are shared. For example, in emotion estimation using a face image and emotion estimation using a temperature, the maximum value of the score shares a predetermined value such as 100. The plurality of emotions are normal, anger, sadness, happiness, etc. For example, the determining unit may determine the emotion as the emotion of the user when the emotion of the maximum score calculated by the 1 st estimating unit matches the emotion of the maximum score calculated by the 2 nd estimating unit, and may determine the emotion of the user based on the score of each emotion calculated by the 1 st estimating unit and the score of each emotion calculated by the 2 nd estimati