KR-20260066575-A - ELECTRONIC DEVICE FOR GENERATING SOUND AND OPERATING METHOD THEREOF

Abstract

The present disclosure provides a method for an HMD device. The method of the HMD device may include an operation of acquiring structural information about a space where a user of the HMD device is located based on image data acquired through a camera of the HMD device, an operation of acquiring material information about at least one object located in the space based on the image data using a learned first model, and an operation of generating sound based on the structural information, material information, and speaker placement setting information. The material information may include sound characteristic information corresponding to the material of the object.

Inventors

• 황호철
• 강두석
• 손동일
• 양성관
• 천재봉

Assignees

• 삼성전자주식회사

Dates

Publication Date

20260512

Application Date

20241122

Priority Date

20241104

Claims (20)

1. In a head-mounted display (HMD) device, camera; At least one processor including a processing circuit; and It includes memory comprising at least one storage medium for storing instructions, and When the above instructions are executed individually or collectively by the at least one processor, the HMD device: Based on image data acquired through the camera, structural information regarding the space where the user of the HMD device is located is acquired, and Using a trained first model, material information for at least one object located in the space is obtained based on the image data, and It causes sound to be generated based on the above structural information, the above material information, and virtual speaker placement setting information, and An HMD device in which the above material information includes sound characteristic information corresponding to the material of the object.
2. In paragraph 1, When the above instructions are executed individually or collectively by the at least one processor, the HMD device: Using a trained second model, the user's predicted location information is obtained based on the user's location data, and The operation of generating the above sound is: An HMD device comprising an operation to generate sound for a user at a predicted location based on the predicted location information, using the above structural information, the above material information, and the above virtual speaker placement setting information.
3. In paragraph 1 or 2, The first model is trained to receive image data for at least one object as input data and to output material classification information for at least one object as output data, wherein the material classification information includes information indicating the type of material of the object. An HMD device wherein the operation of acquiring the above material information includes the operation of acquiring sound characteristic information corresponding to the material of an object based on the above material classification information.
4. In paragraph 3, An HMD device in which the operation of acquiring the above sound characteristic information is performed using a preset material sound characteristic mapping table, and the material sound characteristic mapping table is set using a machine learning model.
5. In paragraph 1 or 2, The above-described first model is an HMD device trained to receive the above-described image data as input data and output the above-described sound characteristic information corresponding to the material of at least one object as output data.
6. In any one of paragraphs 1 through 5, The above at least one object includes at least one of a wall, ceiling, column, or floor located in the space, and An HMD device in which the above sound characteristic information includes information regarding the reflection coefficient or sound absorption coefficient of sound corresponding to the material of the object.
7. In any one of paragraphs 1 through 6, When the above instructions are executed individually or collectively by the at least one processor, the HMD device: Based on the above image data, it causes to generate or update a 3D or 2D map of the above space, and The above map is an HMD device that provides structural information about the space and location information of the user.
8. In any one of paragraphs 1 through 7, When the above instructions are executed individually or collectively by the at least one processor, the HMD device: Identifying whether the current location or predicted location of the user belongs to a second space different from a first space in which at least one virtual speaker is placed based on the virtual speaker placement setting information, and An HMD device that causes the virtual speaker placement setting information to be changed so that the at least one virtual speaker is placed in the second space based on the identification that the user's current location or predicted location belongs to the second space.
9. In any one of paragraphs 1 through 8, The above HMD device includes a plurality of speakers, and When the above instructions are executed individually or collectively by the at least one processor, the HMD device: An HMD device that causes the sound to be output through the plurality of speakers.
10. In any one of paragraphs 1 through 9, The above HMD device includes a communication circuit, and When the above instructions are executed individually or collectively by the at least one processor, the HMD device: An HMD device that causes the sound data to be transmitted to a sound device including a plurality of speakers through the communication circuit.
11. In a method for an HMD (head mounted display) device, An operation to acquire structural information about the space where the user of the HMD device is located, based on image data acquired through the camera of the HMD device; An operation to acquire material information for at least one object located in the space based on the image data using the learned first model; and It includes an operation to generate sound based on the above structural information, the above material information, and virtual speaker placement setting information, and A method in which the above material information includes sound characteristic information corresponding to the material of the object.
12. In paragraph 11, the above method is: The operation of obtaining predicted location information of the user based on the user's location data using a second learned model is further included. The operation of generating the above sound is: A method comprising the operation of generating sound for a user at a predicted location based on the predicted location information, using the above structural information, the above material information, and virtual speaker placement setting information.
13. In Article 11 or Article 12, The first model is trained to receive image data for at least one object as input data and to output material classification information for at least one object as output data, wherein the material classification information includes information indicating the type of material of the object. A method in which the operation of acquiring the above material information includes the operation of acquiring sound characteristic information corresponding to the material of an object based on the above material classification information.
14. In Paragraph 13, A method in which the operation of acquiring the above sound characteristic information is performed using a pre-set material sound characteristic mapping table, and the material sound characteristic mapping table is set using a machine learning model.
15. In Article 11 or Article 12, A method wherein the first model is trained to receive the image data as input data and output sound boundary characteristic information for at least one object as output data.
16. In any one of paragraphs 11 through 15, The above at least one object includes at least one of a wall, ceiling, column, or floor located in the space, and A method in which the above sound characteristic information includes information regarding the reflection coefficient or sound absorption coefficient of sound corresponding to the material of the object.
17. In any one of paragraphs 11 to 16, the method is: Based on the image data above, it further includes the operation of generating or updating a 3D or 2D map of the space, and A method in which the above map provides structural information about the space and location information of the user.
18. In any one of paragraphs 11 through 17, the method is: An operation to identify whether the current location or predicted location of the user belongs to a second space different from a first space in which at least one virtual speaker is placed based on the virtual speaker placement setting information; and A method comprising the operation of changing virtual speaker placement setting information so that at least one virtual speaker is placed in the second space based on identifying that the user's current location or predicted location belongs to the second space.
19. In any one of paragraphs 11 through 18, The above HMD device includes a plurality of speakers, and the method is: A method comprising the operation of outputting the sound through the plurality of speakers.
20. In any one of paragraphs 11 through 19, The above HMD device includes a communication circuit, and the method is: A method comprising the operation of transmitting sound data to a sound device including a plurality of speakers through the above communication circuit.

Description

Electronic device for generating sound and method of operating the same The present disclosure relates to an electronic device for generating sound and a method of operation. Virtual reality (VR) and augmented reality (AR) systems utilize various technologies to enable users to experience immersion. In particular, it is important to provide visual elements through a head-mounted display (HMD) and apply sound effects within the space to make users experience being in a real space. Surround sound technology typically outputs sound through a fixed array of speakers or provides a sense of spatiality through simple left-right stereo sound. However, for users wearing an HMD, their position and orientation change in real time; accordingly, the direction and distance of the sound source must also be dynamically adjusted to match the user's location. This allows users to perceive the location of the sound source more accurately, thereby further enhancing immersion and realism. Therefore, technology is required to provide optimized surround sound by considering the real-time location of the user wearing the HMD. To enhance the user experience, this technology enables customized sound output that reflects the relative position and direction of the user and the sound source. The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. No claim or determination is made as to whether any of the foregoing may be applied as prior art related to the present disclosure. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure. FIG. 2 is a drawing showing the configuration of a wearable electronic device according to one embodiment of the present disclosure. FIGS. 3a to 3c are drawings showing the front and rear views of a wearable electronic device according to one embodiment of the present disclosure. FIGS. 4a and 4b are drawings illustrating an operation in which an HMD device, according to one embodiment of the present disclosure, generates sound to be provided to a user wearing the HMD device using a first sound generation method. FIG. 5 is a drawing illustrating the configuration of an HMD device according to one embodiment of the present disclosure. FIG. 6 is a flowchart illustrating the operation of an HMD device generating sound according to one embodiment of the present disclosure. FIG. 7 is a flowchart illustrating the operation of an HMD device outputting sound according to one embodiment of the present disclosure. FIG. 8 is a flowchart illustrating the operation of an HMD device outputting sound according to one embodiment of the present disclosure. FIG. 9 is a drawing illustrating a first model for material recognition of an object according to one embodiment of the present disclosure. FIG. 10 is a drawing illustrating a second model for predicting a user's location according to one embodiment of the present disclosure. FIG. 11 is a diagram illustrating an operation in which an HMD device performs spatial structure mapping based on image data according to one embodiment of the present disclosure. FIG. 12 is a diagram illustrating an operation in which an HMD device, according to one embodiment of the present disclosure, generates sound to be provided to a user wearing the HMD device using a second sound generation method. FIG. 13 is a diagram illustrating the operation of an HMD device changing a virtual speaker placement setting according to one embodiment of the present disclosure. Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity. FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure. Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor