Search

EP-3973375-B1 - ELECTRONIC DEVICE AND METHOD FOR CONTROLLING THE ELECTRONIC DEVICE

EP3973375B1EP 3973375 B1EP3973375 B1EP 3973375B1EP-3973375-B1

Inventors

  • KIM, JUNGWOOK
  • KIM, Jaedeok
  • SONG, Juyong
  • RIM, Junho
  • SHIN, JAEWOOK
  • LIM, Aeran

Dates

Publication Date
20260506
Application Date
20201008

Claims (15)

  1. An electronic device (100) comprising: a camera (110); a non-volatile memory (120) storing at least one instruction, a plurality of object recognition models and operation mode data (20) including information about a number of levels of a hierarchical structure corresponding to each of a plurality of operation modes of the electronic device (100), a type of object recognition model that may be placed at each level of the hierarchical structure corresponding to each operation mode, and an association relationship between the plurality of object recognition models; a volatile memory (130); and at least one processor (140), connected to the non-volatile memory (120), the volatile memory (130), and the camera (110), and configured to control the electronic device (100), wherein the at least one processor (140), by executing the at least one instruction, is configured to: determine an operation mode of the electronic device (100), identify, through the operation mode data (20) stored in the non-volatile memory (120), information about the number of levels of a hierarchical structure corresponding to the determined operation mode, the type of object recognition model that may be placed at each level of the hierarchical structure and the association relationship between the object recognition models, construct a hierarchical object recognition model having the hierarchical structure corresponding to the determined operation mode through the identified information by connecting the object recognition model arranged for each level to a branch using information on an association relation between the identified object recognition models, wherein the constructed hierarchical object recognition model is layered up to the same level as the number of levels of the hierarchical structure corresponding to the determined operation mode, based on the determined operation mode of the electronic device (100), load, to the volatile memory (130), the hierarchical object recognition model, obtain information on an object by inputting an object image obtained through the camera (110) to the hierarchical object recognition model, and determine an operation to be performed by the electronic device (100) based on the information on the object.
  2. The electronic device (100) of claim 1, wherein the at least one processor (140) is further configured to: based on determining the operation mode as a first operation mode, load, to the volatile memory (130), a first level object recognition model for identifying whether the object is an avoidance target, among the plurality of object recognition models, at a first level in the hierarchical object recognition model, and identify whether the object is the avoidance target based on feature data of the object obtained by inputting the object image to the first level object recognition model.
  3. The electronic device (100) of claim 2, wherein the at least one processor (140) is further configured to: based on determining the operation mode as a second operation mode, load, to the volatile memory (130), the first level object recognition model and at least two second level object recognition models for identifying a type of the object, among the plurality of object recognition models, the at least two second level object recognition models being at a second level lower than the first level in the hierarchical object recognition model, and identify at least one second level object recognition model to which the feature data of the object is to be input from the first level object recognition model, among the at least two second level object recognition models, based on whether the object is the avoidance target.
  4. The electronic device (100) of claim 3, wherein the at least one processor (140) is further configured to identify the type of the object by inputting the feature data of the object obtained through the first level object recognition model to the at least one second level object recognition model.
  5. The electronic device (100) of claim 4, wherein the at least one processor (140) is further configured to, based on the identified type of the object, control the electronic device (100) to variously perform an operation corresponding to the second operation mode.
  6. The electronic device (100) of claim 4, wherein the at least one processor (140) is further configured to: based on determining the operation mode as a third operation mode, load, to the volatile memory (130), the first level object recognition model, the at least one second level object recognition model, and at least one third level object recognition model, the at least one third level object recognition model being loaded at a third level lower than the second level in the hierarchical object recognition model and being capable of recognizing a face of a human, among the plurality of object recognition models.
  7. The electronic device (100) of claim 6, wherein the at least one processor (140) is further configured to, based on the identifying the type of the object as the human through the at least one second level object recognition model, identify whether a human face included in the object image is a pre-registered human face by inputting the feature data of the human to the at least one third level object recognition model.
  8. The electronic device (100) of claim 1, wherein the at least one processor (140) is further configured to: based on an image of a new object being obtained through the camera (110) as the object image, determine the operation mode as a learning mode, obtain feature data of the new object, determine an object recognition model corresponding to the new object, among the plurality of object recognition models, based on the feature data of the new object and information on the plurality of object recognition models, and control to train the determined object recognition model and another object recognition model, among the plurality of object recognition models, based on the feature data of the new object, the another object recognition model being connectable as an upper level of the determined object recognition model in the hierarchical object recognition model.
  9. The electronic device (100) of claim 1, further comprising: a display (160), wherein the at least one processor (140) is further configured to: based on determining the operation mode as a learning mode, control the display (160) to display a user interface (UI) indicating the plurality of object recognition models, based on an object recognition model corresponding to a new object being selected through the UI while an image of the new object is obtained through the camera (110) as the object image, obtain feature data of the new object by inputting the image of the new object to the selected object recognition model, based on a name of a class corresponding to the new object being input by a user, identify the feature data of the new object as data for the class corresponding to the new object for which the name is input, and control to train the selected object recognition model and another object recognition model, among the plurality of object recognition models, based on the data of the class corresponding to the new object, the another object recognition model being connectable as an upper level of the selected object recognition model in the hierarchical object recognition model.
  10. A method of controlling an electronic device (100) including a volatile memory (130) and a non-volatile memory (120) storing a plurality of object recognition models and operation mode data (20) including information about a number of levels of a hierarchical structure corresponding to each of a plurality of operation modes of the electronic device (100), a type of object recognition model that may be placed at each level of the hierarchical structure corresponding to each operation mode, and an association relationship between the plurality of obj ect recognition models, the method comprising: determining an operation mode of the electronic device (100), identifying, through the operation mode data (20) stored in the non-volatile memory (120), information about the number of levels of a hierarchical structure corresponding to the determined operation mode, the type of object recognition model that may be placed at each level of the hierarchical structure and the association relationship between the object recognition models, constructing a hierarchical object recognition model having the hierarchical structure corresponding to the determined operation mode through the identified information by connecting the object recognition model arranged for each level to a branch using information on an association relation between the identified object recognition models, wherein the constructed hierarchical object recognition model is layered up to the same level as the number of levels of the hierarchical structure corresponding to the determined operation mode, based on an operation mode of the electronic device (100), loading, to the volatile memory (130), the hierarchical object recognition model; obtaining information on an object by inputting an object image obtained through a camera (110) to the hierarchical object recognition model; and determining an operation of the electronic device (100) based on the information on the object.
  11. The method of claim 10, wherein the loading further comprises: based on determining the operation mode as a first operation mode, loading, to the volatile memory (130), a first level object recognition model for identifying whether the object is an avoidance target, among the plurality of object recognition models, at a first level in the hierarchical object recognition model; and identifying whether the object is the avoidance target based on feature data of the object obtained by inputting the object image to the first level object recognition model.
  12. The method of claim 11, wherein the loading further comprises: based on determining the operation mode as a second operation mode, loading, to the volatile memory (130), the first level object recognition model and at least two second level object recognition models for identifying a type of the object, among the plurality of object recognition models, the at least two second level object recognition models being at a second level lower than the first level in the hierarchical object recognition model; and identifying at least one second level object recognition model to which the feature data of the object is to be input from the first level object recognition model, among the at least two second level object recognition models, based on whether the object is the avoidance target.
  13. The method of claim 12, further comprising: identifying the type of the object by inputting the feature data of the object obtained through the first level object recognition model to the at least one second level object recognition model.
  14. The method of claim 13, wherein the determining the operation of the electronic device (100) further comprises: based on the identified type of the object, controlling the electronic device (100) to variously perform an operation corresponding to the second operation mode.
  15. The method of claim 14, further comprising: based on the identifying the type of the object as the human through the at least one second level object recognition model, identifying whether a human face included in the object image is a pre-registered human face by inputting the feature data of the human to the at least one third level object recognition model.

Description

Technical Field The disclosure relates to an electronic device and a method for controlling the electronic device and, more particularly, to an electronic device capable of operating an object recognition model having a hierarchical structure that may be configured in various manners according to an operation mode of an electronic device. Background Art As a function of an electronic device utilizing an object recognition technology is advanced, the number of objects to be recognized by an electronic device is increasing. Accordingly, various object recognition technologies using artificial intelligence (AI) models have been studied and developed. In the related art, the technology of recognizing an object through an artificial intelligence model is used and as the number of objects to be recognized increases, a size of a memory and a consumption amount of calculation continuously increase. In the related art, when the object recognition is performed, an entire artificial intelligence model, not only a specific model in accordance with a certain circumstance, is loaded on a memory, there is a disadvantage that the memory is used inefficiently. As a result, there is a limitation in the object recognition capability of a user device which might have limitations in memory, computational capability, and communication capability by only improving the functionality of one artificial intelligence model itself. Document CN 110013197 A discloses an object recognition method of a sweeping robot with high intelligence and strong self-learning ability. According to the method, multiple object decision trees including feature information and processing operation information are established according to conventional object information to form a random forest. After a picture is obtained by using the camera of the sweeping robot and a feature related to an object in the picture is extracted, the extracted object feature is compared with all of the established object decision trees in the random forest. Based on a similarity comparison, the object is classified and added to the object decision tree with the highest similarity. Based on this decision tree, the sweeping robot performs a corresponding operation. Document US 2019/102667 Al discloses an autonomous personal companion utilizing a method of object identification that relies on a hierarchy of object classifiers for categorizing one or more objects in a scene and document US 2007/179918 Al discloses an identification system capable of locating objects in images by searching for local features of an object. Technical Solution The invention is set out in the appended set of claims. Provided are an electronic device for selecting and combining at least one recognition model having a hierarchical structure corresponding to an operation mode determined in an electronic device to be loaded into a volatile memory, and determining an operation to be performed through a selected and combined recognition model having the hierarchical structure loaded in the memory, and a control method thereof. Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the presented embodiments. In accordance with an aspect of the disclosure, an electronic device includes a camera, a non-volatile memory storing at least one instruction and a plurality of object recognition models, a volatile memory, and a processor, connected to the non-volatile memory, the volatile memory, and the camera, configured to control the electronic device, and the processor, by executing the at least one instruction, is configured to, based on an operation mode of the electronic device being determined, load, to the volatile memory, at least one object recognition model with a hierarchical structure corresponding to the determined operation mode, among the plurality of object recognition models, obtain information on the object by inputting an object image obtained through the camera to the loaded object recognition model, and determine an operation of the electronic device based on the information on the object. The at least one processor may be further configured to: based on determining the operation mode as a first operation mode, load, to the volatile memory, a first level object recognition model for identifying whether the object is an avoidance target, among the plurality of object recognition models, at a first level in the hierarchical object recognition model among the plurality of levels, and identify whether the object is the avoidance target based on feature data of the object obtained by inputting the object image to the first level object recognition model. The at least one processor may be further configured to: based on determining the operation mode as a second operation mode, load, to the volatile memory, the first level object recognition model and at least two second level object recognitio