Search

EP-4740184-A1 - METHOD, DEVICE AND SYSTEM FOR DETECTING FACIAL MODIFICATION

EP4740184A1EP 4740184 A1EP4740184 A1EP 4740184A1EP-4740184-A1

Abstract

The invention relates to a method for detecting facial modification, the method comprising the following steps carried out by means of a computer (1): receiving at least one pair of images with a colour image of a face and a heat image, captured simultaneously, of the face; adapting the colour image and the heat image so that the face in the colour image is congruent with the face in the heat image; generating a combined image by combining the adapted colour image and the adapted heat image; determining from the combined image, for one or more classes indicating a type of modification, a probability value in each case that specifies a probability that a modification is present for the corresponding class; and classifying, on the basis of the probability values, whether facial modification is present.

Inventors

  • PETERS, FLORIAN
  • SEDLMEYR, Thomas
  • SCHULTHEISS, CHRISTOPH

Assignees

  • Bundesdruckerei GmbH

Dates

Publication Date
20260513
Application Date
20240620

Claims (13)

  1. 1. A method for detecting facial manipulation, the method comprising the following steps carried out by means of a computer (1): - Receiving at least one image pair comprising a colour image of a face and a simultaneously recorded thermal image of the face; - Adjusting the color image and the thermal image so that the face in the color image is congruent with the face in the thermal image; - Creating a combined image by combining the adjusted color image and the adjusted thermal image; - determining on the combined image for one or more classes indicating a type of manipulation a probability value which indicates a probability that manipulation is present for the corresponding class; and - Classify based on probability values whether facial manipulation has occurred.
  2. 2. The method of claim 1, wherein adjusting the color image and the thermal image comprises the steps of: - determining characteristic facial features of the face in the color image; - scaling dimensions of the color image so that a distance between two of the characteristic facial features in the color image has a predetermined value; - perspective transforming the thermal image using a coordinate transformation so that a perspective of the thermal image and a perspective of the color image match; and - Scaling dimensions of the thermal image to the dimensions of the scaled color image such that the face in the scaled thermal image is congruent with the face in the scaled color image.
  3. 3. The method of claim 2, further comprising: - Cropping the scaled color image based on the characteristic facial features so that the face is arranged at a predetermined position and with a predetermined size in the color image; - Cropping the scaled thermal image analogously to cropping the color image so that the cropped thermal image and the cropped color image have the same section.
  4. 4. Method according to one of claims 2 or 3, wherein the characteristic facial features in the color image are determined by means of artificial intelligence.
  5. 5. Method according to one of the preceding claims, further comprising: - Normalizing, by the computer, the probability value for each of the one or more classes to a normalized probability value; and - where classifying whether facial manipulation has occurred includes: - Summing the normalized probability values for several classes, - Classifying that facial manipulation is present if the sum of the normalized probability values is greater than a first threshold, and - Classify that no facial manipulation is present if the sum of the normalized probability values is less than the first threshold or is less than a second threshold that is lower than the first threshold.
  6. 6. Method according to one of the preceding claims, wherein the determination of a probability value for one or more classes indicating a type of manipulation is carried out by means of a convolutional neural network.
  7. 7. Method according to one of the preceding claims, further comprising: - determining, by the computer, on the combined image an orientation of the Frankfort horizontal of a head comprising the face relative to a horizontal and an orientation of the frontal plane of the head relative to a vertical; - Rejecting the normalized image if an angle between the Frankfort horizontal of the head and the horizontal exceeds a third threshold and/or if an angle between the frontal plane of the head and the vertical exceeds a fourth threshold.
  8. 8. A method according to any one of the preceding claims, wherein the method further comprises: Recording a color image by means of a first camera (2) which is designed to record light with wavelengths in the visible range; and - Recording a thermal image simultaneously with the recording of the color image by means of a second camera (3) which is designed to record wavelengths in the infrared range; - generating, by the computer (1), at least one image pair by assigning the color image and the simultaneously recorded thermal image.
  9. 9. The method of claim 8, further comprising: - taking a color image calibration image with the first camera (2) of a test object having a plurality of light points; - taking a thermal imaging calibration image with the second camera (3) of the test object with the plurality of light points; - Calculating, by the computer (1), the coordinate transformation between a perspective of the first camera (2) and a perspective of the second camera (3) by comparing the positions of the plurality of light points in the color image calibration image and the thermal image calibration image.
  10. 10. Device, in particular a computer (1), for detecting facial manipulation, wherein the device is arranged and designed to carry out the method according to one of claims 1-7.
  11. 11. Computer program product comprising instructions which, when executed by a data processing device, in particular the device according to claim 10, cause the device to carry out the method steps according to at least one of claims 1 to 7.
  12. 12. A system for detecting facial manipulation, the system comprising: - a device according to claim 10; - a first camera (2) in connection with the device, wherein the first camera (2) is designed to record light with wavelengths in the visible range and to generate the color image; - a second camera (3) in connection with the device, wherein the second camera (3) is designed to record wavelengths in the infrared range and to generate the thermal image; - wherein the device is further configured and designed to control the first camera (2) and the second camera (3) such that the first camera (2) color image of the face and the second camera (3) takes the thermal image of the face at the same time as the color image.
  13. 13. The system of claim 12, wherein the system comprises means configured and designed to enable a person to independently capture the color image and the thermal image.

Description

Method, device and system for detecting facial manipulation The invention relates to a method for detecting facial manipulation, a device for detecting facial manipulation, a system for detecting facial manipulation and a computer program product which comprises instructions which cause a correspondingly designed device to carry out the method for detecting facial manipulation. A person's face is crucial to their identity. A visual representation of the face of a person to be identified is often taken using a color image for passport photos or ID photos. A color image can be, for example, an image in the RGB color space, the CMYK color space or a monochrome image. These images should be authentic, i.e. reflect the person's real face, in order to allow reliable conclusions to be drawn about the person's identity, for example in personalized ID cards. In addition, facial recognition methods are known in which an image of a person's face is recorded using a camera in the visible wavelength range. The image recorded in this way provides a representation of the person's face. A person can be clearly identified by analyzing a geometric arrangement of characteristic facial features such as the eyes, nose or mouth and their position, relative distance and location in the area of the face, i.e. an area on the frontal head of the person that is specific to each person. A corresponding image of the person's face can, for example, be taken on site by an employee of a security service provider, in which the employee takes an on-site image of the face of the person to be identified. Recently, the trend has been to take a picture of a person's face without the involvement of another person, such as an employee of a service provider. A so-called self-enrollment system can be provided for this purpose. in which a person to be identified takes a picture of the face independently without the presence of another person. A possible identification process or facial recognition process can then be carried out automatically by software based on the picture taken in this way with the face of the person to be identified. Since the image of the face of the person to be identified is now taken without the presence of an authorized person, there is a possibility that the person to be identified could manipulate the system. In the worst case, the person to be identified tries to deceive the system by presenting the system with a different identity than their own by presenting the system with a different or at least partially falsified face. Such attempts at deception may, for example, consist of the person to be identified bringing an image of another person at least partially into the field of view of the camera, putting on a mask, holding a printed or digital image with the image of another or even non-real person into the field of view of the camera, or concealing certain facial features, for example with sunglasses, an eye patch or something similar. Through such deception attempts, the image captured by the camera shows a different or at least partially distorted face of the person to be identified. This creates the risk that the image taken of the face of the person to be identified will show a different or falsified face of the person, thus creating a false identity for the person to be identified. The document US 2021/110018 A1 describes systems and methods for using machine learning for image-based counterfeit detection and in this case a method in which an input data set is retrieved that contains multiple images of a biometrically authenticated subject taken by multiple cameras of a camera system. The method also includes entering the input data set into a trained machine learning module and processing the input data set using the machine learning module to obtain a forgery detection result for the biometric authentication subject from the machine learning module. The method also includes outputting the forgery detection result for the biometric authentication subject. The document Chen, X. et al., IR and visible light face recognition, Computer Vision and Image Understanding, 2005, Vol. 99, No. 3, pp. 332-358, presents the results of several large-scale studies on face recognition using visible light and infrared imaging in the context of principal component analysis. The document DE 10 2019 135 107 A1 relates to a device and a method for determining a biometric feature of a person's face. The document US 7,421,097 B2 describes facial identification using three-dimensional modeling. Summary Against this background, one object of the invention is to detect facial manipulations when taking a picture. The object is achieved by a method for detecting facial manipulation, which comprises the following steps carried out by a computer: receiving at least one image pair with a color image of a face and a simultaneously recorded thermal image of the face; adjusting the color image and the thermal image so that the face in the color