Search

CN-115457618-B - Yawning detection method and device and electronic equipment

CN115457618BCN 115457618 BCN115457618 BCN 115457618BCN-115457618-B

Abstract

The invention provides a yawning detection method, a yawning detection device and electronic equipment, wherein the method comprises the steps of determining an image to be processed and a mouth area image in the image to be processed; detecting a mouth region image to obtain a detection result of the mouth region image, determining a mouth height-width ratio according to at least one mouth key point in the detection result, determining a shooting angle of an image to be processed, inquiring a ratio threshold value list according to the shooting angle to obtain a ratio threshold value corresponding to the shooting angle, and determining a yawning detection result of the image to be processed according to the mouth height-width ratio and the ratio threshold value. The method can acquire the mouth key points of the mouth area image, determine the mouth height-width ratio, determine the shooting angle of the image to be processed, determine the yawing detection result of the image to be processed according to the ratio threshold value of the shooting angle and the mouth height-width ratio, reduce the misjudgment rate and improve the yawing detection accuracy.

Inventors

  • WEI TAO

Assignees

  • 北京罗克维尔斯科技有限公司

Dates

Publication Date
20260508
Application Date
20220329

Claims (13)

  1. 1. The yawning detection method is characterized by comprising the following steps of: determining an image to be processed and a mouth area image in the image to be processed; detecting the mouth region image to obtain a detection result of the mouth region image, wherein the detection result comprises at least one mouth key point in the mouth region image; Determining a mouth height-width ratio according to the at least one mouth key point; Determining a shooting angle of the image to be processed; inquiring a ratio threshold list according to the shooting angle to obtain a ratio threshold corresponding to the shooting angle; determining a yawning detection result of the image to be processed according to the mouth height-width ratio and the ratio threshold; Before querying a ratio threshold list according to the shooting angle to obtain a ratio threshold corresponding to the shooting angle, the method further comprises: Acquiring more than a preset number of sample mouth region images in a mouth opening state, shooting angles corresponding to the sample mouth region images and sample mouth height-width ratio corresponding to the sample mouth region images; for each shooting angle, acquiring at least one sample mouth region image corresponding to the shooting angle and a sample mouth height-width ratio corresponding to the sample mouth region image; adding and averaging at least one sample mouth width-to-height ratio to obtain a processed mouth width-to-height ratio; and determining a ratio threshold corresponding to the shooting angle according to the processed mouth height-width ratio.
  2. 2. The method of claim 1, wherein the determining the image to be processed, and the mouth region image in the image to be processed, comprises: Determining an image to be processed; performing face detection on the image to be processed to obtain a face area image in the image to be processed; detecting the face key points of the face area image to obtain at least one face key point in the face area image; And determining a mouth region image in the image to be processed according to the mouth key points in the at least one face key point and the image to be processed.
  3. 3. The method according to claim 2, wherein the determining a mouth region image in the image to be processed from the mouth keypoints of the at least one face keypoint and the image to be processed comprises: determining position information of a mouth region in the image to be processed according to the mouth key points in the at least one face key point; And cutting the image to be processed according to the position information to obtain the mouth region image.
  4. 4. The method of claim 1, wherein said determining a mouth aspect ratio from said at least one mouth keypoint comprises: determining a mouth width according to a mouth corner key point in the at least one mouth key point; determining a mouth height according to a lip tip key point in the at least one mouth key point; and determining the ratio of the width of the mouth to the height of the mouth as the ratio of the width of the mouth.
  5. 5. The method of claim 1, wherein the determining the photographing angle of the image to be processed comprises: And inputting the at least one mouth key point into a preset shooting angle prediction model to obtain the shooting angle output by the shooting angle prediction model.
  6. 6. The method according to claim 1, wherein the determining the yawing detection result of the image to be processed according to the mouth width-to-height ratio and the ratio threshold value includes: when the ratio of the mouth height to the width is larger than or equal to the ratio threshold, determining that the yawning detection result is that the yawning behavior exists on the face in the image to be processed; and when the mouth width-to-height ratio is smaller than the ratio threshold, determining that the yawning detection result is that the yawning action does not exist on the face in the image to be processed.
  7. 7. The method of claim 1, wherein the detecting the mouth region image further includes determining a mouth status, and the determining the yawing detection result of the image to be processed according to the mouth aspect ratio and the ratio threshold includes: When the ratio of the height to the width of the mouth is larger than or equal to the ratio threshold value and the mouth state is a mouth opening state, determining that the yawning detection result is that the yawning action exists on the face in the image to be processed; And when the mouth height-width ratio is smaller than the ratio threshold value or the mouth state is a non-mouth opening state, determining that the yawning detection result is that the yawning behavior of the face in the image to be processed does not exist.
  8. 8. A yawning detection apparatus, comprising: The first determining module is used for determining an image to be processed and a mouth area image in the image to be processed; The first acquisition module is used for detecting the mouth region image and acquiring a detection result of the mouth region image, wherein the detection result comprises at least one mouth key point in the mouth region image; The second determining module is used for determining the mouth height-width ratio according to the at least one mouth key point; The third determining module is used for determining the shooting angle of the image to be processed; the second acquisition module is used for inquiring a ratio threshold value list according to the shooting angle so as to acquire a ratio threshold value corresponding to the shooting angle; a fourth determining module, configured to determine a yawning detection result of the image to be processed according to the mouth width-to-height ratio and the ratio threshold; the device also comprises a third acquisition module, a fourth acquisition module, a processing module and a fifth determination module; the third acquisition module is used for acquiring more than a preset number of sample mouth area images in a mouth opening state, shooting angles corresponding to the sample mouth area images and sample mouth height-width ratio corresponding to the sample mouth area images; the fourth obtaining module is configured to obtain, for each shooting angle, at least one sample mouth region image corresponding to the shooting angle, and a sample mouth height-width ratio corresponding to the sample mouth region image; The processing module is used for carrying out addition and averaging processing on at least one sample mouth width-to-height ratio to obtain a processed mouth width-to-height ratio; And the fifth determining module is used for determining a ratio threshold corresponding to the shooting angle according to the processed mouth width-to-height ratio.
  9. 9. The apparatus of claim 8, wherein the first determination module comprises a first determination unit, a first acquisition unit, a second acquisition unit, and a second determination unit; The first determining unit is used for determining an image to be processed; The first acquisition unit is used for carrying out face detection on the image to be processed so as to acquire a face area image in the image to be processed; The second obtaining unit is configured to perform face key point detection on the face area image, so as to obtain at least one face key point in the face area image; the second determining unit is configured to determine a mouth region image in the image to be processed according to the mouth keypoints in the at least one face keypoint and the image to be processed.
  10. 10. The apparatus of claim 8, wherein the second determining means is specifically configured to, Determining a mouth width according to a mouth corner key point in the at least one mouth key point; determining a mouth height according to a lip tip key point in the at least one mouth key point; and determining the ratio of the width of the mouth to the height of the mouth as the ratio of the width of the mouth.
  11. 11. The apparatus of claim 8, wherein the fourth determination module is configured to, When the ratio of the mouth height to the width is larger than or equal to the ratio threshold, determining that the yawning detection result is that the yawning behavior exists on the face in the image to be processed; and when the mouth width-to-height ratio is smaller than the ratio threshold, determining that the yawning detection result is that the yawning action does not exist on the face in the image to be processed.
  12. 12. The apparatus of claim 8, wherein the detection result of the mouth region image further comprises a mouth state, the fourth determining module is specifically configured to, When the ratio of the height to the width of the mouth is larger than or equal to the ratio threshold value and the mouth state is a mouth opening state, determining that the yawning detection result is that the yawning action exists on the face in the image to be processed; And when the mouth height-width ratio is smaller than the ratio threshold value or the mouth state is a non-mouth opening state, determining that the yawning detection result is that the yawning behavior of the face in the image to be processed does not exist.
  13. 13. An electronic device, comprising: a processor and a memory; wherein the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory for implementing the yawning detection method according to any of claims 1-7.

Description

Yawning detection method and device and electronic equipment Technical Field The invention relates to the technical field of deep learning, in particular to a yawning detection method and device and electronic equipment. Background At present, the yawning detection scheme is that a face mouth area is positioned through face detection and a face key point detection algorithm, key point analysis is carried out on a detection result of the mouth area according to a face key point model, the opening and closing degree of the mouth is determined based on the key points obtained through the analysis, and whether the mouth belongs to the mouth opening and closing state or not is judged according to the opening and closing degree and opening and closing degree threshold values. In the scheme, the threshold value of the opening and closing degree is fixed, so that the detection accuracy under partial angles is poor. Disclosure of Invention The present invention aims to solve the technical problems in the related art to some extent. Therefore, a first object of the present invention is to provide a yawning detection method, which includes determining a mouth width-to-height ratio by obtaining a mouth key point of a mouth region image, determining a shooting angle of an image to be processed, determining a yawning detection result of the image to be processed according to a ratio threshold of the shooting angle and the mouth width-to-height ratio, reducing a misjudgment rate, and improving accuracy of yawning detection. A second object of the present invention is to provide a yawning detection device. A third object of the present invention is to propose an electronic device. A fourth object of the present invention is to propose a non-transitory computer readable storage medium. A fifth object of the invention is to propose a computer programme product. In order to achieve the above purpose, an embodiment of the first aspect of the present invention provides a yawning detection method, which includes the steps of determining an image to be processed and a mouth area image in the image to be processed, detecting the mouth area image to obtain a detection result of the mouth area image, wherein the detection result includes at least one mouth key point in the mouth area image, determining a mouth height-width ratio according to the at least one mouth key point, determining a shooting angle of the image to be processed, querying a ratio threshold list according to the shooting angle to obtain a ratio threshold corresponding to the shooting angle, and determining a yawning detection result of the image to be processed according to the mouth height-width ratio and the ratio threshold. According to the yawning detection method, an image to be processed and a mouth area image in the image to be processed are firstly determined, the mouth area image is detected, and a detection result of the mouth area image is obtained, wherein the detection result comprises at least one mouth key point in the mouth area image, a mouth height-width ratio is determined according to the at least one mouth key point, a shooting angle of the image to be processed is determined, a ratio threshold value list is queried according to the shooting angle to obtain a ratio threshold value corresponding to the shooting angle, and the yawning detection result of the image to be processed is determined according to the mouth height-width ratio and the ratio threshold value. The method comprises the steps of obtaining mouth key points of a mouth area image, determining a mouth height-width ratio, determining a shooting angle of an image to be processed, determining a yawing detection result of the image to be processed according to a ratio threshold value of the shooting angle and the mouth height-width ratio, reducing the misjudgment rate and improving the yawing detection accuracy. In addition, the yawning detection method provided by the embodiment of the first aspect of the present invention may further have the following additional technical features: According to one embodiment of the invention, the method for determining the image to be processed and the mouth region image in the image to be processed comprises the steps of determining the image to be processed, performing face detection on the image to be processed to obtain the face region image in the image to be processed, performing face key point detection on the face region image to obtain at least one face key point in the face region image, and determining the mouth region image in the image to be processed according to the mouth key point in the at least one face key point and the image to be processed. According to one embodiment of the invention, the determining of the mouth region image in the image to be processed according to the mouth key points in the at least one face key point and the image to be processed comprises determining position information of the mouth region in the imag