EP-4742180-A1 - LYING POSITION DETECTION OF A PERSON IN BED WITH COMPUTER VISION

EP4742180A1EP 4742180 A1EP4742180 A1EP 4742180A1EP-4742180-A1

Abstract

There is provided a method for detecting in an image a lying position of a person in a lying area with a computing device, the computing device comprising a computer program product, wherein the computer program product when running on the data processor causes the computing device to retrieve the image from the input source as image data, analyze the image data, the analyzing comprising: input the image data to a first machine learning model, the first machine learning model detecting the lying area in the image data, transpose the lying area in the predefined position as a transposed lying area, input the transposed lying area to a second machine learning model, the second machine learning model detecting the lying position of the person in the lying area in the transposed lying area, and to output the lying position of the person in the lying area.

Inventors

VAN DE SANDE, KOEN ERIK ADRIAAN
STOKMAN, Henricus Meinardus Gerardus
VAN OLDENBORGH, Marc Jean Baptist

Assignees

Kepler Vision Technologies B.V.

Dates

Publication Date: 20260513
Application Date: 20251108

Claims (15)

A method for detecting in an image a lying position of a person in a lying area with a computing device, the computing device comprising: an input source; a data processor, and a computer program product comprising: - a first machine learning model trained for detecting a lying area in image data, and - a second machine learning model trained for detecting a lying position of a person in the lying area that is transposed in a predefined position, wherein the computer program product when running on the data processor causes the computing device to: - retrieve the image from the input source as image data; - analyze the image data, the analyzing comprising: • input the image data to the first machine learning model; • the first machine learning model detecting the lying area in the image data; • transpose the lying area in the predefined position as a transposed lying area; • input the transposed lying area to the second machine learning model; • the second machine learning model detecting the lying position of the person in the lying area in the transposed bed area, and - output the lying position of the person in the lying area.
The method according to claim 1, wherein the lying position of the person in the lying area is categorized into three categories: person-lying-on-their-back, person-lying-on-their-left-side, and person-lying-on-their-right-side.
The method according to claim 1 or 2, wherein the transposed lying area is either the lying area rotated in a horizontal position or vertical position.
The method according to any one of the preceding claims, wherein the lying area is selected from a bed in a bed area, a couch in a couch area, a carpet in a carpet area, and a part of a floor in a floor area.
The method according to any one of the preceding claims, wherein the computer program product when running on the data processor in addition causes the computing device to: - store an occurrence of a first output of the lying position with a timestamp; - measure, with time intervals, the time elapsed since the timestamp as a duration; - output a signal when the duration exceeds a predefined threshold, and when an occurrence of a subsequent output of the lying position differs from the first output of the lying position: - store the occurrence of the subsequent output of the lying position, and - reset the timestamp to the time of the occurrence of the subsequent output.
A device configured to detect a lying position of a person in a lying area, the device comprising a common housing holding: an image-capturing sensor outputting image data; a computing device comprising a data processor, and a computer program product comprising: - a first machine learning model trained for detecting a lying area in image data, and - a second machine learning model trained for detecting a lying position of a person in the lying area that is transposed in a predefined position, wherein the computer program product when running on the data processor causes the computing device to: - receive the image data from the image-capturing sensor; - analyze the image data, the analyzing comprising: • input the image data to the first machine learning model; • the first machine learning model detecting the lying area in the image data; • transpose the lying area in the predefined position as a transposed lying area; • input the transposed lying area to the second machine learning model; • the second machine learning model detecting the lying position of the person in the lying area in the transposed lying area, and - output the lying position of the person in the lying area.
The device according to claim 6, wherein the lying position of the person is categorized into three categories: person-lying-on-their-back, person-lying-on-their-left-side, and person-lying-on-their-right-side.
The device according to claim 6 or 7, wherein the transposed lying area is either the lying area rotated in a horizontal position or vertical position.
The device according to any one of the preceding claims 6-8, wherein the lying area is selected from a bed in a bed area, a couch in a couch area, a carpet in a carpet area, and a part of a floor in a floor area.
The device according to any one of the preceding claims 6-9, wherein the computer program product when running on the data processor in addition causes the computing device to: - store an occurrence of a first output of the lying position with a timestamp; - measure, with time intervals, the time elapsed since the timestamp as a duration; - output a signal when the duration exceeds a predefined threshold, and when an occurrence of a subsequent output of the lying position differs from the first output of the lying position: - store the occurrence of the subsequent output of the lying position, and - reset the timestamp to the time of the occurrence of the subsequent output.
A computer program product for running on a computing device comprising a data processor, for detecting in image data a lying position of a person in a lying area, the computer program product comprising: a first machine learning model trained for detecting a lying area in image data, a second machine learning model trained for detecting a rotation position of a person in the lying area that is transposed in a predefined position, and a third machine learning model trained for detecting an orientation position of a lying area that is transposed in a predefined position, wherein the computer program product when running on the data processor causes the computing device to: - receive the image data from an input source; - analyze the image data, the analyzing comprising: • input the image data to the first machine learning model; • the first machine learning model detecting the lying area in the image data; • transpose the lying area in the predefined position as a transposed lying area; • input the transposed lying area to the second machine learning model; • the second machine learning model detecting the rotation position of the person in the lying area in the transposed lying area; • input the transposed lying area to the third machine learning model; • the third machine learning model detecting the orientation position of the lying area in the transposed lying area; • deduct the lying position of the person in the lying area in the transposed lying area from the rotation position and the orientation position, and - output the lying position of the person in the lying area.
The computer program product according to claim 11, wherein the lying position of the person is categorized into three categories: person-lying-on-their-back, person-lying-on-their-left-side, and person-lying-on-their-right-side, and/or wherein the transposed lying area is either the lying area rotated in a horizontal position or vertical position.
The computer program product according to any one of the preceding claims 11-12, wherein the lying area is selected from a bed in a bed area, a couch in a couch area, a carpet in a carpet area, and a part of a floor in a floor area.
The computer program product according to any one of the preceding claims 11-13, wherein the computer program product when running on the data processor in addition causes the computing device to: - store an occurrence of a first output of the lying position with a timestamp; - measure, with time intervals, the time elapsed since the timestamp as a duration; - output a signal when the duration exceeds a predefined threshold, and when an occurrence of a subsequent output of the lying position differs from the first output of the lying position: - store the occurrence of the subsequent output of the lying position, and - reset the timestamp to the time of the occurrence of the subsequent output.
The computer program product according to any one of the preceding claims 11-14 13, wherein the computer program product is running on a computing device that is part of an image-capturing sensor.

Description

TECHNICAL FIELD The invention relates to detecting the lying position of a person, in particular the lying position of a person in bed, from one or more images with the use of artificial intelligence and computer vision. BACKGROUND Artificial intelligence (AI) is developing rapidly. AI applications are supporting or will support all industries including the aerospace industry, agriculture, chemical industry, computer industry, construction industry, defense industry, education industry, energy industry, entertainment industry, financial services industry, food industry, health care industry, hospitality industry, information industry, manufacturing, mass media, mining, telecommunication industry, transport industry, water industry and direct selling industry. The ability to monitor and/or control systems is an area wherein AI can be very useful. Another area is the understanding of human behavior and interaction. To do that, AI systems should be able to detect and recognize events in real-time. This requires a smart approach using software, such as deep neural networks, and powerful computer hardware to execute computations within milliseconds. Computer vision is an area of artificial intelligence (AI) wherein machine learning is used to classify or categorize scenes in images of living beings and objects. The science of computer vision seeks to understand what can be seen and what is happening in an image or series of images such as a photo picture, a video, or a live stream by use of a computing device. In particular, the field of care continually faces escalating challenges due to the lack of staff to monitor patients. For instance, monitoring the lying position of patients developing bed sores is crucial. Turning these patients regularly prevents pressure ulcers or decubitus that would severely deteriorate patients' quality of life. Traditionally, care workers need to keep track of a patient's lying position by themselves and turn the patient at regular time intervals. A task that requires dedicated observation and keeping track of the duration a patient is at a certain lying position. This is a labor-intensive job and prone to errors in an environment wherein a care worker is easily distracted by other patients and emergencies. To address the challenge of lack of staff and still provide sufficient and reliable monitoring, automatic detection of a person's lying position with AI can provide significant assistance to care workers. United States patent No. 11,967,101, titled "Method and system for obtaining joint positions, and method and system for motion capture", according to its abstract describes "The present invention provides a motion capture with a high accuracy which can replace an optical motion capture technology, without attaching optical markers and sensors to a subject. A subject with an articulated structure has a plurality of feature points in the body of the subject including a plurality of joints wherein a distance between adjacent feature points is obtained as a constant. A spatial distribution of a likelihood of a position of each feature point is obtained based on a single input image or a plurality of input images taken at the same time. One or a plurality of position candidates corresponding to each feature point are obtained based on the spatial distribution of the likelihood of the position of each feature point. Each join angle is obtained by performing an optimization calculation based on inverse kinematics using the candidates and the articulated structure. Positions of the feature points including the joints are obtained by performing a forward kinematics calculation using the joint angles." United States patent No. 12,112,541, titled "Bed system", according to its abstract describes "A bed system includes: an imaging device; a bed on which the imaging device is to be installed; and a controller configured to process an image acquired by the imaging device to predict a possibility of overturning of a user, in which, when it is determined that a state of the user is a first state, the controller predicts the possibility of overturning of the user based on a first parameter, when it is determined that the state of the user is a second state, the controller predicts the possibility of overturning of the user based on a second parameter, the first state is a state of the user different from the second state, and the first parameter is different from the second parameter." In "A Vision-Based System for In-Sleep Upper-Body and Head Pose Classification", January 2018, by Yan-Ying Li et al,(https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8914692/#B33-sensors-22-02014), according to its abstract describes "Sleep quality is known to have a considerable impact on human health. Recent research shows that head and body pose play a vital role in affecting sleep quality. This paper presents a deep multi-task learning network to perform head and upper-body detection and pose classification durin