Search

EP-4738242-A1 - METHOD AND APPARATUS FOR PERFORMING VIDEO ENHANCEMENT

EP4738242A1EP 4738242 A1EP4738242 A1EP 4738242A1EP-4738242-A1

Abstract

Provided is a method and device for performing video enhancement. The method includes: identifying a first image from among a plurality of images included in a first video, wherein the first image includes an anomaly; obtaining guide information corresponding to the anomaly from among a plurality of types of guide information; obtaining a second image to replace the first image based on the guide information; and obtaining a second video in which the anomaly has been removed by using the second image.

Inventors

  • RIM, Junho
  • SHIN, JAEWOOK
  • YEO, Haedong

Assignees

  • Samsung Electronics Co., Ltd.

Dates

Publication Date
20260506
Application Date
20240805

Claims (15)

  1. A method of performing video enhancement, the method comprising: identifying a first image from among a plurality of images included in a first video, wherein the first image comprises an anomaly (S210); obtaining guide information corresponding to the anomaly from among a plurality of types of guide information (S220); obtaining a second image to replace the first image based on the guide information (S230); and obtaining a second video in which the anomaly has been removed by using the second image (S240).
  2. The method of claim 1, wherein the obtaining guide information(S220) comprises: generating, based on a type of the anomaly, first guide information from among a guide image, a guide mask, and a guide feature (S910, S920); identifying whether the first guide information is valid (S930); and based on a result of identifying whether the first guide information is valid, identifying the first guide information as the guide information or identifying second guide information that is different from the first guide information as the guide information (S950).
  3. The method of claim 1 or 2, wherein the identifying as the guide information (S950) further comprises: based on identifying that the first guide information is valid, identifying the first guide information as the guide information; and based on identifying that the first guide information is invalid, changing a guide information generating method, generating the second guide information, and identifying the second guide information as the guide information.
  4. The method of any one of claims 1 to 3, wherein the identifying as the guide information (S950) further comprises: based on identifying that the first guide information is valid, identifying whether additional guide information is necessary (S940); based on identifying that the additional guide information is not necessary, identifying the first guide information as the guide information; and based on identifying that the additional guide information is necessary, changing a guide information generating method, generating the second guide information, and identifying the first guide information and the second guide information as the guide information.
  5. The method of any one of claims 1 to 4, wherein, based on the anomaly comprising a blur effect caused by steam, the obtaining guide information (S220) comprises: obtaining a first guide image, from among the plurality of images, based on a comparison of the first image and one or more images, from among the plurality of images, in which the blur effect caused by steam is not detected, wherein the comparison of the first image and the one or more images is based on a comparison of a shape or an outline of an object included in the first image and in the one or more images; identifying whether the first guide image is valid based on a degree of similarity between the first guide image and the first image; and based on a result of the identifying whether the first guide image is valid, identifying the first guide image as the guide information, or identifying a guide mask configured to perform segmentation on the object within the first image, a guide feature comprising a color map of the object, and a second guide image without the object as the guide information.
  6. The method of any one of claims 1 to 5, wherein, based on the anomaly comprising a color change caused by a lighting change, the obtaining guide information (S220) comprises: generating a first guide feature based on color frequency information extracted from the plurality of images; identifying whether the first guide feature is within a certain range; and based on a result of the identifying whether the first guide feature is within the certain range, identifying, as the guide information, a second guide feature based on a color range specified by a user or the first guide feature.
  7. The method of any one of claims 1 to 6, wherein, based on the anomaly comprising a shadow area, the obtaining guide information (S220) comprises: generating a first guide mask corresponding to the shadow area; identifying whether the first guide mask is valid based on a range and a position of the first guide mask; and based on a result of the identifying whether the first guide mask is valid, identifying the first guide mask as the guide information, or identifying a second guide mask generated by changing a mask generation model as the guide information.
  8. The method of any one of claims 1 to 7, wherein, based on the anomaly comprising a reflection effect, the obtaining guide information (S220) comprises: obtaining a first guide image, from among the plurality of images, based on a comparison of the first image and one or more images, from among the plurality of images, in which the reflection effect has not been detected, wherein the comparison is based on a degree of similarity between the first image and the one or more images; generating a first guide mask corresponding to an area, in the first image, that includes the reflection effect; identifying whether the first guide mask is valid based on identifying whether the first guide mask corresponds to a portion, of the first image, in which reflection does not occur; and based on a result of identifying whether the first guide mask is valid, identifying the first guide mask and the first guide image as the guide information, or identifying a second guide mask generated by changing a mask generation model and the first guide image as the guide information.
  9. The method of any one of claims 1 to 8, wherein the identifying the first image (S210) comprises: extracting feature information from each of the plurality of images (S310); identifying a presence of the anomaly in an image of the plurality of images based on the feature information (S320); and identifying the image that includes the anomaly as the first image (S330).
  10. The method of any one of claims 1 to 9, wherein the identifying the presence of the anomaly (S320) comprises performing a certain operation to detect the anomaly based on the feature information or using an anomaly detection model using the feature information as an input.
  11. A device (100) for performing video enhancement, the device comprising: at least one memory (110) storing one or more instructions; and at least one processor (120) configured to execute the one or more instructions, wherein the one or more instructions, when executed by the at least one processor (120), are configured to cause the device (100) to: identify a first image, from among a plurality of images included in a first video, wherein the first image comprises an anomaly, obtain guide information corresponding to the anomaly from among a plurality of types of guide information, obtain a second image to replace the first image based on the guide information, and obtain a second video in which the anomaly has been removed by using the second image.
  12. The device (100) of claim 11, wherein the one or more instructions, when executed by the at least one processor (120), are further configured to cause the device (100) to: generate, based on a type of the anomaly, first guide information from among a guide image, a guide mask, and a guide feature, identify whether the first guide information is valid, and identify, based on a result of identifying whether the first guide information is valid, either the first guide information or second guide information that is different from the first guide information as the guide information.
  13. The device (100) of claim 11 or 12, wherein the one or more instructions, when executed by the at least one processor (120), are further configured to cause the device (100) to: based on identifying that the first guide information is valid, identify the first guide information as the guide information, and based on identifying that the first guide information is invalid, change a guide information generating method, generate the second guide information, and identify the second guide information as the guide information.
  14. The device (100) of any one of claims 11 to 13, wherein the one or more instructions, when executed by the at least one processor (120), are further configured to cause the device (100) to: extract feature information from each of the plurality of images; identify a presence of the anomaly in an image of the plurality of images based on the feature information; and identify the image that includes the anomaly as the first image.
  15. A computer readable medium having instructions stored therein, which when executed by at least one processor cause the at least one processor to execute a method of performing video enhancement, the method comprising: identifying a first image from among a plurality of images included in a first video, wherein the first image comprises an anomaly; obtaining guide information corresponding to the anomaly from among a plurality of types of guide information; obtaining a second image to replace the first image based on the guide information; and obtaining a second video in which the anomaly has been removed by using the second image.

Description

TECHNICAL FIELD The disclosure relates to a method and device for performing video enhancement. BACKGROUND ART Electronic devices, having generative artificial intelligence (Al) installed therein, may generate content requested by a user. Generative Al may generate similar content by using existing content. Generative AI may learn patterns of content and generate new content as an interference result. Recently, various types of electronic devices that utilize neural network models to perform vision tasks have been developed. Furthermore, electronic devices, having generative artificial AI installed therein, may reproduce images according to conditions input by a user, or may generate videos from text or sentences input by a user. DISCLOSURE OF INVENTION SOLUTION TO PROBLEM According to an embodiment of the disclosure, a method of performing video enhancement is provided. The method of performing video enhancement includes identifying a first image from among a plurality of images included in a first video, wherein the first image includes an anomaly. In addition, the method of performing video enhancement includes obtaining guide information corresponding to the anomaly from among a plurality of types of guide information. In addition, the method of performing video enhancement includes obtaining a second image to replace the first image based on the guide information. In addition, the method of performing video enhancement includes obtaining a second video in which the anomaly has been removed by using the second image. According to an embodiment of the disclosure, a device for performing video enhancement is provided. The device for performing video enhancement includes at least one memory storing one or more instructions; and at least one processor configured to execute the one or more instructions. The one or more instructions, when executed by the at least one processor, are configured to cause the device to identify a first image, from among a plurality of images included in a first video, wherein the first image includes an anomaly. In addition, the one or more instructions, when executed by the at least one processor, are configured to cause the device to obtain guide information corresponding to the anomaly from among a plurality of types of guide information. In addition, the one or more instructions, when executed by the at least one processor, are configured to cause the device to obtain a second image to replace the first image based on the guide information. In addition, the one or more instructions, when executed by the at least one processor, are configured to cause the device to obtain a second video in which the anomaly has been removed by using the second image. According to an embodiment of the disclosure, a computer readable medium having instructions stored therein, which when executed by at least one processor cause the at least one processor to execute a method of performing video enhancement, is provided. The method includes identifying a first image from among a plurality of images included in a first video, wherein the first image includes an anomaly. In addition, the method includes obtaining guide information corresponding to the anomaly from among a plurality of types of guide information. In addition, the method includes obtaining a second image to replace the first image based on the guide information. In addition, the method includes obtaining a second video in which the anomaly has been removed by using the second image. BRIEF DESCRIPTION OF DRAWINGS The above and other aspects and features of certain embodiments of the present disclosure will be more apparent from the following description taken in conjunction with the accompanying drawings, in which: FIG. 1 illustrates an image in which an anomaly has been detected and an image from which an anomaly has been removed;FIG. 2 is a flowchart for describing a method of performing video enhancement, according to an embodiment of the disclosure;FIG. 3 is a flowchart for describing a process of determining a first image in which an anomaly has been detected in a first video including a plurality of images, according to an embodiment of the disclosure;FIG. 4 is a diagram for describing a method of detecting a blur effect caused by steam, according to an embodiment of the disclosure;FIG. 5 is a diagram for describing a method of detecting a color change caused by a lighting change, according to an embodiment of the disclosure;FIG. 6 is a diagram for describing a method of detecting a shadow required to be removed, according to an embodiment of the disclosure;FIG. 7 is a diagram for describing a method of detecting a reflection effect, according to an embodiment of the disclosure;FIG. 8 is a diagram for describing an anomaly detection model according to an embodiment of the disclosure;FIG. 9 is a flowchart for describing a process of obtaining guide information corresponding to an anomaly from among a plurality of types of guide inf