US-12620172-B2 - Method and image-processing device for detecting a reflection of an identified object in an image frame

US12620172B2US 12620172 B2US12620172 B2US 12620172B2US-12620172-B2

Abstract

An image-processing device generates a three-dimensional model of a background scene of the image frame based on three-dimensional information about the background scene. The image-processing device defines a three-dimensional bounding box of the object in the three-dimensional model. The image-processing device defines a centre coordinate in the three-dimensional model and a colour value of surface elements of the three-dimensional bounding box. The image-processing device determines a three-dimensional coordinate of a surface in the three-dimensional model which reflects light from a surface element into the camera, by tracing rays from the centre coordinate and based on a normal of the surface. The image-processing device further identifies a first pixel in the image frame corresponding to the three-dimensional coordinate and detects the reflection of the object.

Inventors

Ludvig HASSBRING
Mikael MURSTAM

Assignees

AXIS AB

Dates

Publication Date: 20260505
Application Date: 20240322
Priority Date: 20230330

Claims (13)

1 . A method, performed by an image-processing device, for detecting a reflection of an identified object in an image frame captured by a camera, the method comprising: generating a three-dimensional model of a background scene of the image frame based on obtained three-dimensional information about the background scene; defining the identified object in the image frame based on image information in the image frame; defining a three-dimensional bounding box of the defined object in the three-dimensional model of the background scene; defining surface elements of the three-dimensional bounding box, a respective surface element being defined by a centre coordinate in the three-dimensional model of the background scene and a colour value; determining a three-dimensional coordinate of a surface in the three-dimensional model of the background scene which reflects light from a surface element of the three-dimensional bounding box of the object into the camera, wherein the determining is performed by tracing rays from the centre coordinate of the surface element of the three-dimensional bounding box of the object and based on a normal of the surface in the three-dimensional model of the background scene at said three-dimensional coordinate; identifying a first pixel in the image frame which corresponds to the determined three-dimensional coordinate of the surface; detecting the reflection of the object in the image frame when a mixing of a first colour value of the identified first pixel and the colour value of the surface element of the object and a ground truth colour value of the identified first pixel fulfils a mixing criteria; and in response to detecting the reflection of the object applying a mask to a pixel area of the image frame, the pixel area comprising the identified first pixel.
2 . The method according to claim 1 , wherein the ground truth colour value is obtained from one or more background image frames or from one or more neighbouring pixels to the identified first pixel in the image frame.
3 . The method according to claim 1 , further comprising: obtaining a second image frame of a video sequence comprising the image frame, the second image frame comprising the identified object; defining the identified object in the second image frame based on image information in the second image frame; determining a respective second centre coordinate of the surface element of the three-dimensional bounding box of the object in the three-dimensional model of the background scene based on an obtained second position of the object in the three-dimensional model; determining a second three-dimensional coordinate of the surface or of a second surface in the three-dimensional model of the background scene, which reflect light from the surface element of the three-dimensional bounding box of the object into the camera and differs from the determined three-dimensional coordinate of the surface, wherein the determining is performed by tracing rays from the second centre coordinate of the surface element and based on a second normal of the surface or of the second surface in the three-dimensional model of the background scene at said second three-dimensional coordinate; identifying a second pixel in the second image frame which corresponds to the determined second three-dimensional coordinate of the surface or of the second surface; obtaining a second colour value of the second pixel; and confirming detection of the reflection of the object when a mixing of the first colour value and the second colour value fulfils a second mixing criteria; or rejecting detection of the reflection of the object when the mixing of the first colour value and the second colour value does not fulfil the second mixing criteria, a colour value of the first pixel in the second image frame equals the first colour value.
4 . The method according to claim 3 , further comprising: in response to confirming detection of the reflection of the object, applying a mask to a second pixel area of the second image frame, the second pixel area comprising the identified second pixel.
5 . The method according to claim 3 , further comprising: in response to rejecting detection of the reflection of the object, determining to not apply a mask to the second pixel area comprising the identified second pixel.
6 . The method according to claim 1 , wherein the masked pixel area comprises all pixels corresponding to a projection of the surface element of the three-dimensional bounding box onto the surface or the second surface in the three-dimensional model of the background scene, which projection is reflected into the camera.
7 . The method according to claim 1 , wherein the three-dimensional information about the background scene comprises three-dimensional coordinates of surfaces in the background scene, a respective normal vector of the surfaces, three-dimensional coordinates of the camera and an orientation of the camera.
8 . The method according to claim 1 , wherein the colour value is a hue value or a combination of Cb and Cr values in the YUV format.
9 . The method according to claim 1 , wherein extrapolating the object is performed by extrapolating bounding pixels of the object from the captured image frame.
10 . The method according to claim 1 , wherein a respective surface element of the three-dimensional bounding box corresponds to multiple pixels from the captured image frame and wherein the colour value of the respective surface element is calculated as an average of colour values of the corresponding multiple pixels.
11 . An image-processing device configured to perform the method according to claim 1 .
12 . The image-processing device according to claim 11 , wherein the image-processing device is a video camera, such as a surveillance camera, or a video-server.
13 . A non-transitory computer readable medium that has a computer program, stored therein comprising computer readable code units which when executed on an image-processing device causes the image-processing device to perform the method according to claim 1 .

Description

CROSS-REFERENCE TO RELATED APPLICATION This application claims priority to European Patent Application No. 23165670.3 filed on Mar. 30, 2023, the entire disclosure of which is incorporated by reference herein. TECHNICAL FIELD The embodiments herein relate to a method and an image-processing device for detecting a reflection of an identified object in an image frame. A corresponding computer program and a computer program carrier are also disclosed. BACKGROUND Surveillance of the public using imaging, in particular video imaging, is common in many areas around the world. Areas that may need monitoring are for example banks, stores, and other areas where security is needed, such as schools and government facilities. Other areas that may need monitoring are processing, manufacturing and logistics applications where video surveillance is primarily used to monitor processes. However, there may be requirements to not be able to identify persons from the video surveillance. The requirement to not be able to identify persons may be in conflict with the requirement of being able to determine what is happening in the video. For example, it may be of interest to perform people counting or queue monitoring on anonymous image data. In practice, there is a trade-off between meeting these two requirements: non-identifiable video and extracting large amounts of data for different purposes such as people counting. Several image-processing techniques have been described to avoid identifying persons while still being able to recognize activities. For example, edge detection/representation, edge enhancement, silhouetting objects, and different sorts of “colour blurring”, such as colour variation or dilation are such examples of manipulations. Privacy masking is another image-processing technique used in video surveillance to protect personal privacy by concealing parts of an image from view with a masked area. Image processing refers to any processing that is applied to an image. The processing can include application of various effects, masks, filters or the like, to the image. In this manner, the image can e.g. be sharpened, converted to grey scale, or altered in some way. The image has typically been captured by a video camera, a still image camera or the like. As mentioned above one way to avoid identification of persons is by masking moving people and objects in images in real time. Masking in live and recorded video can be done by comparing a live camera view with a set background scene and applying dynamic masking to areas of change-essentially moving people and objects. Color masking, which may also be referred to solid color masking or monochrome masking where an object is masked by an overlaid solid mask of a certain color, provides privacy protection while enabling you to see movements. Mosaic masking, also referred to as pixelation, pixelated privacy masking or transparent pixelated masking shows moving objects in lower resolution and allows you to better distinguish forms by seeing an object's colors. Masking live and recorded video is suitable for remote video monitoring or recording in areas where surveillance is otherwise problematic due to privacy rules and regulations. It's ideal for processing, manufacturing and logistics applications when video surveillance is primarily used to monitor processes. Other potential applications are in retail, education and at government facilities. Although masking techniques evolve there is still room for improvement. One problem that may arise is that reflective surfaces in a scene may reflect what is behind a privacy mask into a camera capturing the scene. This problem is especially tricky for dynamic masks, which are expected to move with the object. Document CN 108 090 947 A discloses a ray tracing optimization method for 3D scenes. SUMMARY An object of embodiments herein may thus be to obviate some of the problems mentioned above, or at least reduce the impact of them. Specifically, an object of embodiments herein may be to identify pixels in an image that represent surfaces in the scene that reflect an object into a camera such that it is possible to apply image processing to the pixels representing those reflective surfaces. The reflections may, for example, be masked as well to provide improved anonymization. Embodiments herein solve the above problems by creating a three-dimensional representation of a scene captured by an image frame from a camera, including an identified masked object, and tracing optical rays in the three-dimensional representation of the scene from the identified object to the camera via reflecting surfaces in the scene. However, only those reflecting surfaces that are sufficiently similar to the actual object itself will be detected as reflecting the object. Specifically, only those reflecting surfaces that produce a reflection which colour value matches a colour value of the object being reflected will be detected. Comparison of the colour valu