CN-122029797-A - Apparatus and method for user discomfort management for AR/VR applications in a smart home environment

CN122029797ACN 122029797 ACN122029797 ACN 122029797ACN-122029797-A

Abstract

A method of a device supporting Augmented Reality (AR) or Virtual Reality (VR) includes detecting an occurrence of at least one perceived discomfort of a user when the user uses the device supporting AR or VR, determining, by an operating state of a source Internet of things (IoT) device, the source IoT device that triggers the occurrence of the at least one perceived discomfort of the user, providing, by a user interface of the device, at least one suggestion to the user, and adjusting the operating state of the source IoT device based on a response of the user to the provided at least one suggestion.

Inventors

Ashish Burghart
Kushram Giovaria
Sahil Gautam
Oman Chitan Fremwara

Assignees

三星电子株式会社

Dates

Publication Date: 20260512
Application Date: 20240429
Priority Date: 20230929

Claims (14)

1. A method of a device (100) supporting an augmented reality AR or virtual reality VR, the method comprising: detecting (401) the occurrence of at least one perceived discomfort of a user when the user uses the AR or VR enabled device; Determining (403) a source IoT device by an operational state of the source IoT device, the source IoT device triggering an occurrence of the at least one perceived discomfort of the user; providing (405) at least one suggestion to the user via a user interface of the device, and The operating state of the source IoT device is adjusted (407) based on the user's response to the provided at least one suggestion.
2. The method of claim 1, wherein detecting the occurrence of the at least one sensory discomfort of the user comprises: detecting an occurrence of the at least one perceived discomfort of the user using a machine learning ML model based on at least one of the historical discomfort data of the user, VR data associated with the device, and VR activity performance data of the user.
3. The method of claim 2, further comprising: Detecting a sensory discomfort event associated with the user for a period of time when the user interrupts AR/VR activity, manually operates one of the one or more IoT devices, and resumes the AR/VR activity, and Historical discomfort data for the user is generated based on the detected sensory discomfort events within the time period.
4. A method according to claim 3, wherein, for each of the sensory discomfort events, the generated historical discomfort data of the user comprises: The type of sensory discomfort of the user, Information corresponding to the sensor that detected the sensory discomfort, The facial expression of the user when the sensory discomfort occurs, An IoT device operated by the user when the sensory discomfort occurs, and At least one operating characteristic of the IoT device that is modified by the user.
5. The method of claim 2, further comprising: Monitoring VR screen activity of the user while the user is using the device, and Generating VR activity performance data of the user based on the monitored VR screen activity of the user, wherein the generated VR activity performance data of the user includes at least one of content information during the VR screen activity of the user and information of the movement of the user.
6. The method of claim 2, further comprising receiving the VR data from one or more sensors of the device, Wherein the VR data includes at least one of information associated with at least one of a heart rate of the user, a breathing pattern of the user, skin conductance of the user, smell around the user, a light level of a glasses screen of the device, a facial expression of the user, a blink rate of the user, eye movements of the user, or pupil dilation of the user.
7. The method of claim 1, further comprising: Monitoring an operational status of each of the one or more IoT devices, and Generates IoT device context data for respective IoT devices of the one or more IoT devices based on the monitored operational state, Wherein the IoT device context data includes the monitored operational states of respective IoT devices of the one or more IoT devices and operational characteristics of each of the one or more IoT devices configured to modify the monitored operational states.
8. The method of claim 7, wherein determining the source IoT device comprises: Determining a correlation between the one or more IoT devices, the operating feature, and the at least one perceived discomfort based on the IoT device context data and the detected occurrence of the at least one perceived discomfort, and The method further includes determining the source IoT device and the operating feature of the source IoT device based on the determined correlation, the operating feature of the source IoT device configured to modify the operating state of the source IoT device.
9. The method of claim 8, wherein providing the at least one suggestion to the user comprises: generating an AR object for controlling the operating characteristics of the source IoT device, and The generated AR object is provided to the user as the user interface in the device.
10. The method of claim 1, wherein adjusting the operational state of the source IoT device comprises: detecting a user gesture as the response of the user to the at least one suggestion provided; Mapping the detected user gesture to an operation control command of the source IoT device, and The operating state of the source IoT device is adjusted based on a mapping of the detected user gesture to the operating control command of the source IoT device.
11. An apparatus (100) supporting an augmented reality AR or/and virtual reality VR, the apparatus comprising: A memory (114) for storing instructions; At least one processor (108) configured to, when executing the instructions, cause the apparatus to perform operations comprising: detecting the occurrence of at least one perceived discomfort of a user when the user uses the AR or VR enabled device, Determining a source internet of things, ioT, device by an operational state of the source IoT device, the source IoT device triggering the occurrence of the at least one sensory discomfort of the user, Providing at least one suggestion to the user via a user interface of the device, and The operating state of the source IoT device is adjusted based on the user's response to the provided at least one suggestion.
12. The apparatus of claim 11, wherein the operations further comprise at least one operation of the method of one of claims 2 to 10.
13. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor (108) of a device (100) supporting an augmented reality AR or a virtual reality VR, cause the device to perform operations comprising: Detecting occurrence of at least one perceived discomfort of a user when the user uses the AR or VR enabled device; Determining, by an operational state of a source internet of things IoT device, the source IoT device that triggers the occurrence of the at least one sensory discomfort of the user; Providing at least one suggestion to the user via a user interface of the device, and The operating state of the source IoT device is adjusted based on the user's response to the provided at least one suggestion.
14. The non-transitory computer-readable storage medium of claim 13, wherein the operations further comprise at least one operation of the method of one of claims 2 to 10.

Description

Apparatus and method for user discomfort management for AR/VR applications in a smart home environment Technical Field The present disclosure relates generally to the field of wearable display devices, and more particularly to an apparatus and method for user discomfort management for Augmented Reality (AR)/Virtual Reality (VR) applications in, for example, a smart home environment. Background AR/VR devices have experienced significant evolution over the years, resulting in increased demand. Technological advances in areas such as display technology, computer processing power, and motion tracking have prompted an increase in demand for AR/VR devices. These advances have resulted in a more immersive and realistic experience. When using an AR/VR device, there are a number of key aspects that the user needs to consider. One of the most important aspects when a user uses an AR/VR device is user comfort, which plays a vital role in enhancing the overall experience. When a user uses an AR/VR device, a number of factors may affect the comfort of the user. One of these factors is the physical environment of the user. The physical environment of the user encompasses the surrounding environment in which the content is presented to the user. The physical environment of the user may also include other devices or objects that may affect the user experience. Unexpected events related to other devices may cause discomfort to the user when the user is immersed in the AR/VR device. For example, the physical environment of the user may be a smart home environment having a plurality of internet of things (IoT) devices. The operating conditions of IoT devices in the plurality of IoT devices may cause user discomfort. The plurality of IoT devices may include ovens, music systems, air conditioners, hybrid juicers, and other IoT-enabled devices. In a non-limiting example, during a VR session, strong odors from ovens may cause user discomfort, affecting the user experience corresponding to VR activity. In another non-limiting example, loud and bass music from a music system may cause haptic discomfort to a user and may potentially impact the user's experience when the user is engaged in an online meeting during a VR session. In another non-limiting example, when a user exercises during a VR session, an air conditioner operating at a higher temperature in an economy mode may cause the user to feel skin discomfort due to perspiration. In such a scenario, the user needs to stop exercising in order to manually change the air conditioning fan speed setting, which destroys the user's cadence and can potentially be annoying to the user. Accordingly, there is a need for an improved method and apparatus that overcomes all of the limitations and problems discussed above associated with existing AR/VR devices in a user's physical environment. Disclosure of Invention Technical proposal This summary is provided to introduce a selection of aspects of the present disclosure in a simplified form that are further described below in the detailed description of the present disclosure. This summary is not intended to identify key or essential aspects of the disclosure, nor is it intended to be used to identify the scope of the disclosure. According to an aspect of the disclosure, a method of a device supporting Augmented Reality (AR) or Virtual Reality (VR) is provided. The method may include detecting the occurrence of at least one perceived discomfort of a user when the user uses the AR or VR enabled device. The method may include determining, by an operational state of a source internet of things (IoT) device, the source IoT device that triggers an occurrence of the at least one perceived discomfort of the user. The method may include providing at least one suggestion to the user through a user interface of the device. The method may include adjusting the operational state of the source IoT device based on the user's response to the provided at least one suggestion. According to an aspect of the disclosure, an apparatus supporting Augmented Reality (AR) or Virtual Reality (VR) is provided. The apparatus may include a memory storing instructions and at least one processor configured to, when executing the instructions, cause the apparatus to perform operations. The operations may include detecting an occurrence of at least one perceived discomfort of a user when the user uses the AR or VR enabled device. The operations may include determining, by an operational state of a source internet of things (IoT) device, the source IoT device that triggers an occurrence of the at least one perceived discomfort of the user. The operations may include providing at least one suggestion to the user through a user interface of the device. The operations may include adjusting the operational state of the source IoT device based on the user's response to the at least one suggestion provided. According to one aspect of the present disclosure, a non-transitory comput