US-12626578-B2 - Hybrid method and system for multi scenario drowsiness detection and method for data processing using real-time and historical data on wearable devices

US12626578B2US 12626578 B2US12626578 B2US 12626578B2US-12626578-B2

Abstract

A system and method for drowsiness detection by a wearable device (smartwatch or smart ring), which constantly records and processes physiological and behavioral signals combined with historical user data. The system and method actively monitor the user's state, and using signal processing and machine-learning module, detect when the user is entering in a drowsy state. Based on the drowsiness detection module identifying a non-drowsiness an output of a drowsiness detection is directly fed back for future re-evaluation, and based on the drowsiness detection identifying a drowsiness state as being detected, an alert is provided to the user.

Inventors

RENATO STOFFALETTE JOÃO
Paula Gabrielly RODRIGUES
Frank Alexis CANAHUIRE CABELLO
Paulo Augusto Alvez Luz Viana
Klairton Lima Brito
Lizeth Stefania Benavides Cabrera
Sofia Gelio Da Silva
RUAN ROBERT BISPO DOS SANTOS
Gustavo Pessoa Caixeta Pinto Da Luz
Jaime Armando Delgado Vargas
Everton Zaccaria Nadalin

Assignees

Samsung Eletrônica da Amazônia Ltda.

Dates

Publication Date: 20260512
Application Date: 20240409
Priority Date: 20240207

Claims (19)

1 . A hybrid system for multi scenario drowsiness detection using real-time and historical data on a wearable device, the hybrid system comprising: a data acquisition module to acquire physiological and behavioral data from embedded sensors of the wearable device; a data processing module to process data by performing pre-processing and feature extraction; a drowsiness detection module to identify whether a user is drowsy based on the physiological and behavioral data acquired from the embedded sensors of the wearable device in combination with the historical data of the user; and an alert generation module to alert the user, wherein based on the drowsiness detection module identifying a non-drowsiness state, an output of the drowsiness detection module is directly fed back in the data acquisition module as an input for future re-evaluation drowsiness of the user, and wherein based on the drowsiness detection module identifying a drowsiness state as being detected, the alert generation module alerts the user.
2 . The hybrid system as in claim 1 , wherein the data acquisition module comprises a health/fitness database with the historical data and real time sensors data collected from an accelerometer, a PPG, a gyroscope, a thermometer and a galvanic skin response (GSR) sensor.
3 . The hybrid system as in claim 1 , wherein any type of sensor that captures a PPG signal of an individual user is enabled to be employed in PPG module.
4 . The hybrid system as in claim 1 , wherein the data processing module comprises a pre-processing module and a feature extraction module.
5 . The hybrid system as in claim 1 , wherein the hybrid system uses historical health and behavioral data including information regarding one or more health indicators, as stress, sleep quality, medicines, blood glucose, and muscular inflammation.
6 . The hybrid system as in claim 1 , wherein the hybrid system uses a stress index that is measurable by the user and that provides a value in a range of 0 to 100, which is collected from a data source.
7 . The hybrid system as in claim 1 , wherein a muscle wear index is defined as: ∑ w i ∈ W ( α × T ⁡ ( w i ) ) + ( β × I ⁡ ( w i ) ) + ( δ × D ⁢ ( w i ) ) wherein elements α, β and δ in the muscle wear index represent weights associated with a type, an intensity, and a duration of a workout, respectively.
8 . The hybrid system as in claim 1 , wherein a metric that indicates whether sleep sessions during a day x reached a minimum level defined as: Q ⁡ ( S x ) = ∑ s ∈ S x ⁢ time ⁢ ( s ) × score ⁢ ( s ) M is used by the hybrid system.
9 . The hybrid system as in claim 1 , wherein the hybrid system uses a feature that consider user historical sleep quality defined as: ∑ i = 0 k Q ⁢ ( s i ) × 1 2 i wherein k is calibrated for each user, and assuming a day 0 refers to a present day and day j, such that j>0, refers to the j past days.
10 . The hybrid system as in claim 1 , wherein the hybrid system uses a feature that considers user historical sleep quality which is alternatively defined as: ∑ i = 0 k Q ⁢ ( s i ) × w i wherein w i is a weight associated with an i-th last day.
11 . The hybrid system as in claim 1 , wherein the hybrid system uses a data source enabled to collect age, sex, weight, and height as profile data.
12 . The hybrid system as in claim 1 , wherein the hybrid system uses a classification model classifies a signal as being drowsiness onset or not drowsiness onset.
13 . The hybrid system as in claim 1 , wherein the drowsiness detection module includes a drowsiness scoring model, which is designated to receive data extracted from the embedded sensors in real time, profile data as inputs, and a drowsiness threshold model, which outputs a personalized threshold for detecting the drowsiness state.
14 . The hybrid system as in claim 1 , wherein the alert to the user by the alert generation module comprises one of a visual notification on a screen, an audible notification or a vibration signal.
15 . The hybrid system as in claim 1 , wherein the alert generation module includes a wearable alert generation, a smartphone alert trigger, a smartphone alert generation and a drowsiness record, and the system is triggered when the drowsiness state is detected in the drowsiness detection module.
16 . A hybrid method for multi scenario drowsiness detection using real-time and historical data on a wearable device, the hybrid method comprising: acquiring physiological and behavioral data from embedded sensors of the wearable device; processing data by pre-processing and feature extraction; and identifying whether a user is drowsy based on algorithm of a drowsiness detection module which uses the physiological and behavioral data acquired from the embedded sensors of the wearable device in combination with the historical data of the user, wherein based on a non-drowsiness state being identified at the drowsiness detection module, an output is directly fed back in a data acquisition module as an input for future re-evaluation drowsiness of the user, and wherein based on a drowsiness state being identified as being detected, alerting the user.
17 . The hybrid method as in claim 16 , wherein, the alerting of the user provides an alert notification, and after the alert notification is received, drowsiness record including information regarding health/fitness and sensor data that generated the drowsiness state is fed back in the data acquisition module for future re-evaluation.
18 . The hybrid method as in claim 16 , wherein the hybrid method uses user sleep quality history information, which is collected from a data source.
19 . The hybrid method as in claim 16 , wherein the processing data further comprises: filtering a PPG signal to remove motion artifacts using an adaptive Least Mean Squares (LMS) technique, using accelerometers as reference signals, and filtering band pass between 0.1 Hz and 5 Hz; normalizing PPG, accelerometer, gyroscope and thermometer signals to predefined ranges; and performing pre-processing and feature extraction including health/fitness filtering, performing outlier detection and removal for multi scenario drowsiness detection using real-time and historical data on a wearable device that adjusts future re-evaluation of drowsiness of a user based on current drowsiness detection.

Description

CROSS-REFERENCE TO RELATED APPLICATION(S) This application is based on and claims priority under 35 U.S.C. § 119 to Brazilian Patent Application No. BR 10 2024 002538 5, filed on Feb. 7, 2024, in the Brazilian Intellectual Property Office, the disclosure of which is incorporated by reference herein in its entirety. TECHNICAL FIELD The present invention relates to systems and methods for drowsiness detection by a wearable device (smartwatch or smart ring), which constantly records and processes physiological and behavioral signals combined with historical user data. More specifically, the systems and methods of the present invention actively monitors the user's state, and by means of signal processing and machine-learning modeling, detects when the user is entering in a drowsy state. The present invention can serve as a monitoring and alerting system for long journey drivers, thus collaborating to reduce traffic accidents. It can also serve as an alert system for heavy machinery operators and alert the operator prior to entering in a life risky situation. It can serve as an alert system for healthcare professionals and those with long working hours, as a system to alert professionals about their current level of attention and reduce the risk of health-related incidents. BACKGROUND OF THE INVENTION Drowsiness (or sleepiness) can be defined as the propensity of falling asleep and is characterized by a low arousal level and propensity to doze off, which is generally related to the feeling of lethargy, tiredness or sleepiness. The drowsy state can be caused by different factors such as sleep deprivation, medical conditions, and sleep disorders. Even with an adequate sleep time, the drowsiness state can occur, and it may be dangerous in several daily activities, such as working or driving. When drowsiness occurs at inappropriate times, particularly during activities that require complete alertness, it becomes a problem that can cause serious consequences not only for the person who is in a drowsy state, but also for other people around. For instance, a driver who loses control of the vehicle and may end up hitting pedestrians, or a machinery operator who does not notice a person approaching during operation and causes an accident. Drowsiness is considered the main cause of thousands of accidents on highways. According to the International Association of Oil & Gas Producers (IOGP), sleepiness contributes to approximately 1 out of 5 fatal and serious road accidents. The IOGP also estimates that a drowsy driver is three times more likely to be involved in a road crash. Furthermore, drowsiness directly interferes with cognitive function, which includes memory, attention and decision making, reducing productivity and performance in tasks that require mental focus. In addition to the problems that drowsiness can cause during daily activities, in some cases it can be a symptom of underlying medical conditions that may require medical attention, such as sleep apnea, diabetes, depression, etc. In the last decade, wearable devices have become very popular and countless applications in the healthcare field have been designed. One of the reasons for the popularity of such devices is the possibility to measure and analyze biological signals captured using non-invasive sensors such as accelerometers, thermometers, and light pulses. The data collected by such sensors can be used to detect and prevent serious incidents that may occur due to drowsiness. According to the prior art, there are four major and mostly used systems to detect drowsiness: Vehicle-based systems: based on sensors placed on various components of the vehicle, including steering wheel, pedal, etc.Behavioral-based systems: based on body movements, accelerometers, facial expressions, and head movements, for example frequent yawning, head tilt, eye blinking, etc.Physiological-based systems: based on physiological signals, for example heart rate, skin impedance, body temperature, etc.Hybrid systems: based on the combination of other systems, for example Physiological and Behavioral, Behavioral and Vehicle-based, or Physiological and Vehicle-based. Drowsiness detection based on wearable devices presents numerous challenges. Usually, the information regarding the facial expressions (i.e., yawning and eyes movements), vehicle sensors, or that require an image processing are not available, and thus making it harder to detect the drowsy state. However, adopting a hybrid approach, which combines non-intrusive physiological, behavioral, and historical information tends to provide good results and shows a better accuracy in comparison to other approaches. Furthermore, drowsiness detection using wearable devices on the wrist brings a significant advantage from an economic standpoint when considered for an industrial purpose, since there is no need to install any expensive device/sensor in the vehicle or at the workstation. It is important to note that the hybrid approach a