Search

US-12622607-B2 - Human sleep posture extraction from millimeter-wave wireless systems

US12622607B2US 12622607 B2US12622607 B2US 12622607B2US-12622607-B2

Abstract

Methodology and corresponding apparatus pertain to human sleep posture monitoring, using a wireless signal-based monitoring system leveraging millimeter-wave technology. A software-only human sleep posture monitoring solution based on millimeter-wave (mmWave) wireless-based solutions enables fine-grained posture monitoring under no light without being privacy-invasive. In zero visibility, body joint information and changes can be extracted directly from mmWave imaging using improved capabilities for extracting human sleep posture data from millimeter-wave wireless systems. A single-person sleep posture monitoring system leverages signal processing and deep learning models to enable fine-grained monitoring continuously and non-intrusively with commodity (i.e., generally available) mmWave devices. The system directly predicts joint locations from reflected mmWave signals by learning the hidden association between them from thousands of data samples. Learning is accomplished through a customized Deep Convolutional Neural Network (DCNN), that predicts the 3D locations of several key body joints from the reflected signals captured by multiple mmWave antennas.

Inventors

  • Sanjib Sur
  • AAKRITI ADHIKARI

Assignees

  • UNIVERSITY OF SOUTH CAROLINA

Dates

Publication Date
20260512
Application Date
20240208

Claims (14)

  1. 1 . Method for automatically classifying sleep postures of a human subject from millimeter-wave (mmWave) wireless signals reflecting from the human subject, comprising: training a Deep Convolutional Neural Network (DCNN) learning model, based on inputs of ground truth sleep postures of a plurality of human subjects and generated input and output pairs of mmWave reflected signals from the plurality of human subjects and respectively corresponding with the ground truth inputs, to learn the association between millimeter-wave (mmWave) wireless signals reflected from a human subject and joint locations of a human subject; operating the trained DCNN learning model to process further input data thereto, to determine and output identification of diverse sleep postures during the rest state of the human subject; transmitting millimeter-wave (mmWave) wireless signals configured for interacting with a human subject; receiving millimeter-wave (mmWave) wireless signals reflecting from the human subject; operating the DCNN based on the received signal reflections for directly predicting joint locations of the human subject; and operating the DCNN for classifying the sleep posture of the human subject into one of a plurality of diverse sleep postures during the rest state of the human subject, based on predicted joint locations of the human subject.
  2. 2 . Methodology according to claim 1 , wherein predicting comprises predicting the 3D locations of a plurality of key body joints based on the received signal reflections, and based on evaluating a hierarchal relationship of arrangements for the body joints of a given human subject.
  3. 3 . Method according to claim 1 , further comprising: using at least one existing 5G wireless device in a home setting for transmitting millimeter-wave (mmWave) wireless signals to the human subject, to provide a single-person sleep posture monitoring system with mmWave-based posture monitoring under no light without being privacy-invasive.
  4. 4 . Method according to claim 1 , further comprising refining the training of the DCNN for a specific human subject user based on the ground truth height of the specific human subject user.
  5. 5 . Method according to claim 1 , further comprising monitoring a human subject using an observation arrangement in which a human subject is reclined on a bed, and at least one mmWave transmitter and receiving antenna is positioned in a range from 2 to 5 meters away from the human subject, with the antenna having a beamwidth which covers the whole bed area of the bed on which the human subject is reclined.
  6. 6 . Method according to claim 1 , wherein the DCNN comprises an architecture of a plurality of stacked neural networks having at least two convolution layers each, all using ReLU activation in each layer, followed by a flatten layer.
  7. 7 . Method according to claim 6 , wherein the DCNN comprises further architecture of a pair of paths in parallel based on the output of the flatten layer, with one path comprising a joint regressor to output 3D joint locations and the other path comprising a height classifier to output height labels.
  8. 8 . One or more tangible, non-transitory computer-readable media that collectively store instructions that, when executed, cause a computing device including one or more processors to perform operations, the operations comprising automatically identifying diverse sleep postures during the rest state of a human subject from millimeter-wave (mmWave) wireless signals reflecting from the human subject, by: training a Deep Convolutional Neural Network (DCNN) learning model, based on inputs of ground truth sleep postures of a plurality of human subjects and generated input and output pairs of mmWave reflected signals from the plurality of human subjects and respectively corresponding with the ground truth inputs, to learn the association between millimeter-wave (mmWave) wireless signals reflected from a human subject and joint locations of a human subject; operating the trained DCNN learning model to process further input data thereto, to determine and output identification of diverse sleep postures during the rest state of the human subject; transmitting millimeter-wave (mmWave) wireless signals configured for interacting with a human subject; receiving millimeter-wave (mmWave) wireless signals reflecting from the human subject; operating the DCNN based on the received signal reflections for directly predicting joint locations of the human subject; and operating the DCNN for classifying the sleep posture of the human subject into one of a plurality of diverse sleep postures during the rest state of the human subject, based on predicted joint locations of the human subject.
  9. 9 . The one or more tangible, non-transitory computer-readable media according to claim 8 , wherein predicting comprises predicting the 3D locations of a plurality of key body joints based on the received signal reflections, and based on evaluating a hierarchal relationship of arrangements for the body joints of a given human subject.
  10. 10 . The one or more tangible, non-transitory computer-readable media according to claim 8 , further comprising operations of: using at least one existing 5G wireless device in a home setting for transmitting millimeter-wave (mmWave) wireless signals to the human subject, to provide a single-person sleep posture monitoring system with mmWave-based posture monitoring under no light without being privacy-invasive.
  11. 11 . The one or more tangible, non-transitory computer-readable media according to claim 8 , further comprising operations of refining the training of the DCNN for a specific human subject user based on the ground truth height of the specific human subject user.
  12. 12 . The one or more tangible, non-transitory computer-readable media according to claim 8 , further comprising operations of monitoring a human subject using an observation arrangement in which a human subject is reclined on a bed, and at least one mmWave transmitter and receiving antenna is positioned in a range from 2 to 5 meters away from the human subject, with the antenna having a beamwidth which covers the whole bed area of the bed on which the human subject is reclined.
  13. 13 . The one or more tangible, non-transitory computer-readable media according to claim 8 , wherein the DCNN comprises an architecture of a plurality of stacked neural networks having at least two convolution layers each, all using ReLU activation in each layer, followed by a flatten layer.
  14. 14 . The one or more tangible, non-transitory computer-readable media according to claim 13 , wherein the DCNN comprises further architecture of a pair of paths in parallel based on the output of the flatten layer, with one path comprising a joint regressor to output 3D joint locations and the other path comprising a height classifier to output height labels.

Description

PRIORITY CLAIM The present application claims the benefit of priority of U.S. Provisional Patent Application No. 63/497,218, titled Human Sleep Posture Extraction From Millimeter-Wave Wireless Systems, filed Apr. 20, 2023, and which is fully incorporated herein by reference for all purposes. STATEMENT REGARDING SPONSORED RESEARCH OR DEVELOPMENT This invention was made with government support under Grant Number 2144505, awarded by the NSF. The government has certain rights in the invention. BACKGROUND OF THE PRESENTLY DISCLOSED SUBJECT MATTER 1. Importance, Challenges, and Prior Art Humans spend approximately one-third of their life sleeping. High-quality sleep is of vital importance for the short term proper functioning of the human body and for long-term good health [1]. A key metric to monitoring sleep is the spatial and temporal understanding of sleep postures through the night, as the postures directly influence sleep behavior and critical parameters [2]. Each of us sleeps in one of the broad categories of posture, such as supine, lateral, fetal, etc., and exhibits wide variations of them throughout the night [3]. The effect of different sleep postures has been studied widely to identify their relationship to different health conditions [4-6]. Specific sleep postures could be fatal depending on the pre-existing medical conditions. For example, supine posture is linked with exacerbating obstructive sleep apnea by creating unfavorable airway geometry, causing a reduction in lung volume and limiting the movement of airway dilator muscles, which could be life-threatening [7]. Infrequent turns due to impairment in control of the motor activity of Parkinson's patients lead to parasomnia and restless leg syndrome [8]. Infrequent changes in sleep posture are also the primary cause of pressure ulcers (i.e., bedsores) in post-surgical and elderly patients. Additionally, physicians recommend different sleep postures for different medical conditions: It is recommended to sleep on side posture to reduce snoring, or left side to prevent heartburn, or supine posture to lower back or shoulder pain, or fetal posture during pregnancy, or some specific posture variations during post-surgery recovery [9; 10]. These examples highlight the importance of a sleep posture monitoring system that can provide real spatio-temporal observations, which could help with corrections and prevent fatal accidents. Existing at-home approaches for sleep posture monitoring that use wearables, pressure mattresses, or vision cameras are either cumbersome, costlier, or highly privacy-invasive [11-14]. Further, their performance is hindered by dark bedroom conditions and occlusion. Millimeter-wave (mmWave) wireless-based solutions can overcome these challenges by enabling fine-grained posture monitoring under no light without being privacy-invasive. MmWave signals can penetrate certain obstacles, work under zero visibility, and have higher-resolution than Wi-Fi. So, mmWave imaging can facilitate “seeing” the body posture under dark conditions and under the blanket. Besides, mmWave transceivers are poised to soon become ubiquitous in all 5G-and-beyond devices, such as access points, enabling the opportunity for bringing privacy non-invasive sleep posture monitoring system to the masses at-home. However, there exist two fundamental challenges in mmWave imaging. First, mmWave signals could be absorbed by many body parts or specularly reflect from them in different directions away from the device, causing most signals to never reach back to the receiver. So, the output human shape could have a lot of missing parts from which it is difficult to infer joint locations. Second, mmWave devices have extremely low-resolution compared to vision-based systems; so, many high-frequency components, such as the contour and limbs, will be eliminated from the generated images [15]. Moreover, the reflected signals carry additional information about the bed and surrounding objects close to the body, making it harder to separate the human shape. So, it is challenging to extract body joint information and changes directly from traditional mmWave imaging during sleep. SUMMARY OF THE PRESENTLY DISCLOSED SUBJECT MATTER The presently disclosed systems and corresponding and/or associated methodologies generally relate to improved mmWave imaging technology, and more particularly to improved capabilities for extracting human sleep posture data from millimeter-wave wireless systems. Presently disclosed subject matter (also in some instances presently referenced as “MiSleep”) is for some embodiments a wireless signal based sleep posture monitoring system. MiSleep leverages the built-in millimeter-wave technology on ubiquitous 5G wireless devices and provides a software-only sleep posture monitoring solution, so it does not require any extra hardware as do existing pressure mattress technologies. To overcome above-referenced challenges for some settings, the presently disclosed