US-12622604-B2 - Systems and methods for monitoring movements

US12622604B2US 12622604 B2US12622604 B2US 12622604B2US-12622604-B2

Abstract

Disclosed are systems and methods for classifying body movements, including gestures and activities of daily living. This may include preprocessing data output from an accelerometer and gyroscope into features that are input into a movement classifier. In some examples, the disclosed technology may utilize user confirmed labels of movements to improve the accuracy of the movement classifiers. This may include providing a notification to a user when a movement classifier determines a particular movement is detected that requests confirmation of the movement label.

Inventors

Savan R. Devani

Assignees

BIOTRILLION, INC.

Dates

Publication Date: 20260512
Application Date: 20200529

Claims (20)

1 . A method comprising: receiving a set of sensor data comprising data output from an accelerometer and a gyroscope; preprocessing the set of sensor data to output a set of features; processing the set of features with a cyclical individualized movement classifier for a user to output a movement classification, wherein the movement classification comprises one or more of eating, eating with hands, eating with fork and knife, eating with spoon, drinking, lying down, cyclical exercise motions, washing hands, brushing teeth, toilet usage, repeated contact with different anatomic regions, or other cyclical motions that can quantify the time, frequency, and duration of activities of daily living; providing a notification to the user to confirm the movement classification on an interface; receiving user input confirming the movement classification; in response to the user input, generating a plurality of rolling, overlapping time windows over the sensor data within a preset interval before and after a time of the user input, and concurrently processing the plurality of rolling, overlapping time windows in parallel with the cyclical individualized movement classifier; determining, based at least in part on agreement of at least a threshold percentage of the plurality of rolling, overlapping time windows, a start time A and an end time B of the confirmed movement: defining AB windows as those rolling, overlapping time windows of the plurality of rolling, overlapping time windows whose ranges fall between the start time A and the end time B; computing cycle features from the AB windows; updating, on a device comprising the cyclical individualized movement classifier, the cyclical individualized movement classifier for the user by training the cyclical individualized movement classifier using the confirmed movement classification and the cycle features from the AB windows, to generate an updated cyclical individualized movement classifier for the user; compiling statistics related to the movement classification; establishing a baseline of movements based on the statistics, wherein the baseline is specific to the user; and predicting risks of health conditions based on changes of the movements relative to the established baseline of the user.
2 . The method of claim 1 , wherein the notification is a pop-up message on a touch screen interface.
3 . The method of claim 2 , wherein the user input is an interaction with the touch screen interface comprising a confirmed label of the movement classification.
4 . The method of claim 1 , wherein the set of features comprises: acceleration along an x axis, acceleration along a y axis, acceleration along a z axis, angular velocity along axis x, angular velocity along the y axis, and angular velocity along the z axis, and duration.
5 . The method of claim 1 , wherein the preprocessing the set of sensor data further comprises segmenting the data into a set of overlapping time windows.
6 . The method of claim 1 , wherein the set of features comprise at least one feature related to cyclicality.
7 . The method of claim 1 , wherein each of the plurality of rolling, overlapping time windows has a duration between 1 and 3 seconds and an overlap between 50 percent and 75 percent.
8 . The method of claim 1 , wherein the threshold percentage is 50 percent, 60 percent, or 70 percent.
9 . The method of claim 1 , wherein the preset interval spans two minutes before and two minutes after the time of the user input.
10 . The method of claim 1 , wherein updating the cyclical individualized movement classifier comprises storing updated parameters in a memory of the wearable device.
11 . A system, the system comprising: a wearable device; at least one movement sensor incorporated into the wearable device; a memory containing machine readable medium comprising machine executable code having stored thereon instructions; a control system coupled to the memory comprising one or more processors, the control system configured to execute the machine executable code to cause the control system to: receive a set of sensor data from the at least one movement sensor; preprocess the set of sensor data to output a set of features; process the set of features with a cyclical individualized movement classifier for a user to output a movement classification, wherein the movement classification comprises one or more of eating, eating with hands, eating with fork and knife, eating with spoon, drinking, lying down, cyclical exercise motions, washing hands, brushing teeth, toilet usage, repeated contact with different anatomic regions, or other cyclical motions that can quantify the time, frequency, and duration of activities of daily living; provide a notification to the user to confirm the movement classification on an interface; receive user input confirming the movement classification; in response to the user input, generate a plurality of rolling, overlapping time windows over the sensor data within a preset interval before and after a time of the user input, and concurrently process the plurality of rolling, overlapping time windows in parallel with the cyclical individualized movement classifier; determine, based at least in part on agreement of at least a threshold percentage of the plurality of rolling, overlapping time windows, a start time A and an end time B of the confirmed movement; define AB windows as those rolling, overlapping time windows of the plurality of rolling, overlapping time windows whose ranges fall between the start time A and the end time B; compute cycle features from the AB windows; update, on the wearable device or a paired device, the cyclical individualized movement classifier for the user by training the cyclical individualized movement classifier using the confirmed movement classification and the cycle features from the AB windows, to generate an updated cyclical individualized movement classifier for the user; compile statistics related to the movement classification; establish a baseline of movements based on the statistics, wherein the baseline is specific to the user; and predict risks of health conditions based on changes of the movements relative to the established baseline of the user.
12 . The system of claim 11 , wherein the control system is further configured to: receive a second set of sensor data from the at least one movement sensor; preprocess the second set of sensor data to output a second set of features; and process the second set of features with the updated cyclical individualized movement classifier to output a second movement classification.
13 . The system of claim 11 , wherein the at least one movement sensor comprises one or more of an accelerometer, a gyroscope, and magnetometer.
14 . The system of claim 11 , wherein the wearable device is one of a smart watch, smart anklet, and smart band.
15 . The system of claim 11 , wherein the preset interval spans two minutes before and after the time of the user input confirming the movement classification.
16 . The system of claim 11 , wherein the threshold percentage is 50 percent, 60 percent, or 70 percent, and the control system determines the start time A and the end time B based on majority agreement.
17 . A method comprising: receiving a set of sensor data comprising data output from at least one movement sensor; preprocessing the set of sensor data to output a set of features; processing the set of features with a cyclical individualized movement classifier for a user to output a movement classification, wherein the movement classification comprises one or more of eating, eating with hands, eating with fork and knife, eating with spoon, drinking, lying down, cyclical exercise motions, washing hands, brushing teeth, toilet usage, repeated contact with different anatomic regions, or other cyclical motions that can quantify the time, frequency, and duration of activities of daily living; providing a notification to the user to confirm the movement classification on an interface; receiving user input confirming the movement classification; in response to the user input, generating a plurality of rolling, overlapping time windows over the sensor data within a preset interval before and after a time of the user input, and concurrently processing the plurality of rolling, overlapping time windows in parallel with the cyclical individualized movement classifier; determining, based at least in part on agreement of at least a threshold percentage of the plurality of rolling, overlapping time windows, a start time A and an end time B of the confirmed movement; defining AB windows as those rolling, overlapping time windows of the plurality of rolling, overlapping time windows whose ranges fall between the start time A and the end time B; computing cycle features from the AB windows; updating, on a device comprising the cyclical individualized movement classifier, the cyclical individualized movement classifier for the user by training the cyclical individualized movement classifier using the confirmed movement classification and the cycle features from the AB windows, to generate an updated cyclical individualized movement classifier for the user; compiling statistics related to the movement classification; establishing a baseline of movements based on the statistics, wherein the baseline is specific to the user; and predicting risks of health conditions based on changes of the movements relative to the established baseline of the user.
18 . The method of claim 17 , wherein the set of features includes one or more of acceleration along an x axis, acceleration along a y axis, acceleration along a z axis, angular velocity along axis x, angular velocity along the y axis, and angular velocity along the z axis, and duration.
19 . The method of claim 17 , wherein the preprocessing the set of sensor data further comprises segmenting the data into a set of overlapping time windows.
20 . The method of claim 17 , wherein the threshold percentage is 50 percent, 60 percent, or 70 percent.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application is a U.S. National Stage of International Application No. PCT/US2020/035265, filed May 29, 2020 which claims priority to and the benefit of U.S. Provisional Application No. 62/855,725 filed May 31, 2019, titled HAND KINETIC VECTOR INFERRED GESTURES TO CLASSIFY ACTIVITIES OF DAILY LIVING, each of which is hereby incorporated herein by reference in their entireties. FIELD The present invention is directed to systems and methods for monitoring movements, including classification of movements based on data output from wearables. BACKGROUND The following description includes information that may be useful in understanding the present invention. It is not an admission that any of the information provided herein is prior art or relevant to the presently claimed invention, or that any publication specifically or implicitly referenced is prior art. Users perform a variety of movements and gestures throughout their lives, including in performing their daily activities. Identifying, tracking and monitoring these movements can provide useful information about a user. SUMMARY Specific or repeated body movements that can be passively and objectively classified have the potential to provide insights into the existence of certain conditions, diseases and/or disorders of a user. For example, anorexia, depression, anxiety, and/or other diseases, disorders and conditions may be inferred from repeated, particularly cyclic, motions. Similarly, behaviors that may lead to diseases, such as smoking, may be inferred from repeated hand motions. Such data may also be combined with other forms of context-based metadata such as time of day, or the amount of time a touch screen is active in order to improve the classification accuracy of such behaviors, or put them into more meaningful context. For instance, the disclosed systems and methods may aid in compliance with prescribed diabetes management, including by monitoring aspects of eating such as frequency, duration, time of day. Metabolic, endocrine, hormonal diseases and disorders or conditions like pregnancy might also be classified with non-binary statistical probability based on changes in behaviors inferred from a user's kinetic body or hand activity. Additionally, certain digestive diseases/disorders such as irritable bowel syndrome (IBS) might be detected based on the monitoring of eating movements of a user as described herein and any statistically significant changes to established homeostatic baselines related thereto. Side-effects/adverse-events in response to novel drug treatments in clinical trials may also be determined based on detection of particular hand motions as described above. Further, weight gain/loss, changes in health or lifestyle, and/or habits (such as binge eating, eating small bites through the day, or sleeping right after eating) might be inferred based on the behaviors indicated by the hand motion detected. A corresponding alert and/or other applications can be provided based on the activities inferred from the hand motions. Depression, pregnancy, sleep disorders, the risk of certain obesity related disease such as diabetes might also be indicated in the alert. Detection and/or prediction of conditions, disorders and/or diseases noted above may serve as an indication of holistic health, providing insights into the multiple facets of disease risk, and/or may be usable in assisting individuals to alter their behaviors. Insurance companies might adopt smart device-derived next-gen metrics akin to “steps” into their risk stratification, but there are many other ways to stratify risk (such as frequency of eating as an “inverse” corollary of steps). Changes in baseline behavior or emergence of novel hand gestures (e.g. tremors) may serve as indications of potentially serious side effects in patients in drug trials. Conversely, reduction in similar motions may prove evidence of efficacy when targeting diseases, disorders and/or conditions. The methods and systems described herein, in some examples, employ user-feedback to label movements. The system monitors sensors data output by inertial measurement unit(s) (“IMU”—a combination of at least one accelerometer and at least one gyroscope) and applies classifiers to the data to determine whether they are likely associated with a particular movement. In some examples, the system may monitor the sensor data and request confirmation from a user that a particular movement has been performed to label the ground truth of the data. These labels allow for the supervised training of software to passively detect similar movements, providing views into movement/gesture patterns and frequency specific to the user. These movements correspond to certain behaviors or activities of daily living (“ADLs”) which affect health. Changes to baseline behaviors provide insights into changes in user health. Health Applications The method and system described herein may aid user hea