KR-102963422-B1 - ELECTRONIC DEVICE AND CONTROLLING METHOD OF ELECTRONIC DEVICE

KR102963422B1KR 102963422 B1KR102963422 B1KR 102963422B1KR-102963422-B1

Abstract

An electronic device capable of providing text information corresponding to a user's voice through a user interface and a method for controlling the same are provided. Specifically, the electronic device according to the present disclosure analyzes an image containing at least one object when the image containing at least one object is acquired to identify at least one object included in the image, performs voice recognition for the user's voice when the user's voice is received to acquire text information corresponding to the user's voice, identifies an object among at least one object included in the image that corresponds to the user's voice, and displays a memo UI (User Interface) containing text information on an area on a display corresponding to the object identified as corresponding to the user's voice.

Inventors

신민규
김상윤
이도균
한창우
유종욱
이재원

Assignees

삼성전자주식회사

Dates

Publication Date: 20260513
Application Date: 20191002
Priority Date: 20181018

Claims (20)

In electronic devices, display; mike; Memory for storing at least one instruction; and A processor that executes at least one of the above instructions; comprising, The above processor is, When an image containing at least one object is obtained, the image is analyzed to identify at least one object included in the image, and When a user's voice is received through the microphone, voice recognition is performed on the user's voice to obtain text information corresponding to the user's voice, and Based on the above text information, a memo UI (User Interface) including the above text information is obtained, and Based on the text information above, identify whether one of the at least one object included in the image corresponds to the text information, and When the one object corresponding to the above text information is identified, the area corresponding to the identified object is identified on the display, and An electronic device that controls the display to display the memo UI containing the text information on the identified area.
In Article 1, The above processor is, An electronic device that identifies an object corresponding to the information about at least one object included in the image among at least one object included in the image as the object corresponding to the user voice, if the acquired text information includes information about at least one object included in the image.
In Article 1, The above processor is, An electronic device that identifies the area on the display where the touch interaction is received as the area when the one object corresponding to the text information is not identified and the user voice is received.
In Paragraph 3, The above processor is, Control the display to display the image on the display, and An electronic device that identifies an object displayed on an area on the display where the touch interaction is received as the object corresponding to the user's voice.
In Paragraph 3, The above processor is, An electronic device that performs voice recognition on a user voice received through the microphone while the touch interaction is maintained on the display, and obtains text information corresponding to the user voice.
In Paragraph 3, The above processor is, Based on the coordinate values of the user's touch interaction received on the display, at least one of the size and shape of the memo UI is determined, and An electronic device that controls the display to display the memo UI based on at least one of the determined size and shape on the area corresponding to the object identified as corresponding to the user voice.
In Article 1, An electronic device in which an area corresponding to an object identified as corresponding to the above user voice includes an area where the object identified as corresponding to the above user voice is displayed on the display and an area within a preset distance from the area where the object identified as corresponding to the above user voice is displayed on the display.
In Article 1, The above processor is, If there are two or more objects identified as corresponding to the user voice, the display is controlled to display a selection UI containing information about the two or more objects on the display, and An electronic device that, upon receiving user input selecting one of the two or more objects through the selection UI, identifies the selected object as the object corresponding to the user's voice.
In Article 1, The above processor If the above-mentioned acquired text information includes a pronoun, the text information is obtained in which the pronoun is changed to the name of an object identified as corresponding to the user voice, and An electronic device that controls the display to display a memo UI containing the modified text information on the area corresponding to the object identified as corresponding to the user voice.
In Article 1, A communication unit including a circuit; further comprising, The above processor is, An electronic device that controls a communication unit to transmit the acquired text information to a user terminal corresponding to the user's name when the acquired text information includes the name of a user previously stored.
In a method for controlling an electronic device including a display, When an image containing at least one object is obtained, a step of analyzing the image to identify at least one object included in the image; When a user voice is received, a step of performing voice recognition on the user voice to obtain text information corresponding to the user voice; Based on the above text information, a step of obtaining a memo UI (User Interface) including the above text information; A step of identifying whether one of the at least one object included in the image corresponds to the text information based on the text information; When the one object corresponding to the text information is identified, a step of identifying an area corresponding to the identified object on the display; and A method for controlling an electronic device comprising the step of displaying the memo UI containing the text information on the identified area.
In Article 11, The step of identifying an object corresponding to the above user voice is, A method for controlling an electronic device comprising the step of identifying, if the acquired text information includes information about at least one object included in the image, an object among the at least one object included in the image that corresponds to the information about at least one object as the object corresponding to the user voice.
In Article 11, The step of identifying an object corresponding to the above user voice is, A method for controlling an electronic device comprising the step of identifying the area on the display where the touch interaction was received as the area, when the one object corresponding to the text information is not identified and the user voice is received.
In Article 13, The step of identifying an object corresponding to the above user voice is, A step of displaying the image on the above display; and A method for controlling an electronic device comprising: a step of identifying an object displayed on an area on the display where the touch interaction is received as the object corresponding to the user's voice.
In Article 13, The step of obtaining text information corresponding to the user voice above is, A method for controlling an electronic device comprising: a step of performing voice recognition on the received user voice while the touch interaction is maintained on the display among the received user voices to obtain the text information corresponding to the user voice.
In Article 13, The step of displaying the above memo UI is, A step of determining at least one of the size and shape of the memo UI based on the coordinate values of the user's touch interaction received on the display; and A method for controlling an electronic device comprising: a step of displaying the memo UI based on at least one of the determined size and shape on the area corresponding to the object identified as corresponding to the user voice.
In Article 11, A control method for an electronic device comprising an area corresponding to an object identified as corresponding to the above user voice, the area where the object identified as corresponding to the above user voice is displayed on the display, and an area within a predetermined distance from the area where the object identified as corresponding to the above user voice is displayed on the display.
In Article 11, The step of identifying an object corresponding to the above user voice is, If there are two or more objects identified as corresponding to the user voice, a step of displaying a selection UI on the display that includes information about the two or more objects; and A method for controlling an electronic device comprising: a step of identifying the selected object as the object corresponding to the user's voice when a user input selecting one of the two or more objects is received through the selection UI.
In Article 11, The step of displaying the above memo UI is, If the above-mentioned acquired text information includes a pronoun, the step of acquiring text information in which the pronoun is changed to the name of an object identified as corresponding to the user voice; and A method for controlling an electronic device comprising: a step of displaying a memo UI containing the modified text information on the area corresponding to the object identified as corresponding to the user voice.
In Article 11, A method for controlling an electronic device further comprising the step of transmitting the acquired text information to a user terminal corresponding to the user's name if the acquired text information includes a user's name previously stored therein.

Description

Electronic Device and Controlling Method of Electronic Device The present disclosure relates to an electronic device and a method for controlling an electronic device, and specifically, to an electronic device capable of providing text information corresponding to a user's voice through a user interface and a method for controlling the same. Recently, as voice recognition technology has been applied to various types of electronic devices, users can generate text information by speaking their voice without directly inputting text information and display the generated text information on a display. However, according to the prior art, there is a problem in that a user of an electronic device must endure the inconvenience of performing multiple steps, such as the step of executing an application to generate text information, the step of inputting a trigger input for voice recognition, the step of speaking a user voice corresponding to the text information, the step of specifying the location where the text information will be displayed, and the step of specifying the size and format of the text information to be displayed, in order to generate text information on the electronic device using voice recognition and display it on an area on the display desired by the user. Therefore, there is a growing need for an intuitive and simple technology capable of performing a series of processes to acquire text information corresponding to a user's voice and display the acquired text information in an area on the display desired by the user. FIG. 1 is a conceptual diagram for briefly explaining the control process of an electronic device according to the present disclosure, FIG. 2 is a flowchart illustrating a method for controlling an electronic device according to one embodiment of the present disclosure, FIG. 3a is a block diagram briefly illustrating the configuration of an electronic device according to the present disclosure, FIG. 3b is a drawing for explaining an embodiment according to the present disclosure based on the premise of the coupling relationship between the hardware configuration of an electronic device (100) as shown in FIG. 3a and a software module according to the present disclosure. FIG. 4 is a drawing for explaining the object recognition process according to the present disclosure in more detail, FIG. 5 is a drawing for explaining the speech recognition process according to the present disclosure in more detail, FIG. 6 is a diagram illustrating an embodiment in which an electronic device identifies an object corresponding to a user's voice based on text information corresponding to the user's voice. FIGS. 7a to 7c are drawings for explaining an embodiment in which an electronic device identifies an object corresponding to a user's voice based on a user's touch interaction. FIGS. 8A and 8B are drawings for explaining embodiments related to displaying different sizes and shapes of a memo UI based on user touch interaction. FIGS. 9a and 9b are drawings for illustrating various types related to a memo UI according to the present disclosure, FIG. 10a is a drawing for integrally explaining an embodiment related to the process of identifying an object corresponding to a user voice and the case where there are two or more objects corresponding to the user voice according to the present disclosure. FIG. 10b is a drawing for explaining an embodiment of displaying a memo UI on an area where a user's touch interaction is received on a display. FIGS. 11a and 11b are drawings for explaining in detail an embodiment related to providing a selection UI when there are two or more objects identified as corresponding to a user voice, FIG. 12 is a drawing for explaining an embodiment related to the case where pronouns are included in text information corresponding to user voice, FIG. 13a is a drawing for explaining an embodiment related to a case where information about a user name is included in text information corresponding to a user voice, FIG. 13b is a drawing for explaining an embodiment related to a case where text information corresponding to a user's voice includes information about the expiration date of an object. FIG. 13c is a drawing for explaining an embodiment related to a case where text information corresponding to a user's voice includes information about the user's intention. FIG. 14 is a drawing for explaining an embodiment related to displaying a memo UI on a home screen provided through an electronic device, FIGS. 15a and 15b are drawings for illustrating embodiments related to cases where the position of an object corresponding to a user voice is moved or disappears. FIG. 16 is a drawing for briefly explaining the structure of a refrigerator related to the present disclosure, where the electronic device according to the present disclosure is a refrigerator. FIG. 17 is a block diagram showing in detail the configuration of an electronic device according to the present disclosure, FIGS. 18a and