Search

JP-7856597-B2 - Equipment control system, equipment control method, and program

JP7856597B2JP 7856597 B2JP7856597 B2JP 7856597B2JP-7856597-B2

Inventors

  • 橋本 裕介
  • 工藤 弘行
  • 梅村 拓矢
  • 林 勇太
  • 張 翔
  • 石川 哲朗

Assignees

  • パナソニック株式会社

Dates

Publication Date
20260511
Application Date
20230217

Claims (16)

  1. A radio wave sensor that performs a transmit and receive operation, transmitting a transmitted wave, which is a radio wave modulated in a predetermined manner, toward real space, receiving a reflected wave from the real space, and outputting an output signal based on the transmitted wave and the reflected wave, A conversion unit performs a conversion process to convert the output signal output by the radio wave sensor into location-identifiable information that can identify the position of each of one or more objects present in the real space. A holding unit that performs a holding process to hold the location-identifiable information converted by the conversion process, A difference acquisition unit performs a difference acquisition process to acquire the difference between a plurality of location-identifiable pieces of information held by the holding process, A motion estimation unit performs motion estimation processing that estimates whether each of the one or more objects is a moving object or a stationary object based on the difference obtained by the difference acquisition process, identifies the position of the object estimated to be a moving object, and acquires motion position information indicating the identified position. A point placement unit performs a point placement process that virtually places points corresponding to the motion position information acquired by the motion estimation process into a virtual space corresponding to the real space, The system comprises an equipment control unit that performs equipment control processing based at least on a point cloud which is a set of points arranged in the virtual space, The moving body is either a human body or a moving object other than a human body. The motion estimation process further estimates, at least based on the change in the difference, whether each of the two or more motion position information pieces obtained with different time differences corresponds to the human body or a moving object other than the human body. The point placement process virtually places points in the virtual space that correspond to the motion position information that the motion estimation process has estimated to correspond to the human body, out of the two or more motion position information acquired by the motion estimation process. Equipment control system.
  2. The motion estimation process includes a posture estimation process that estimates the posture of the human body based on the distribution of the point cloud. The system further includes an action detection unit that performs an action detection process to detect human actions corresponding to the human body based on the posture or changes in posture estimated by the posture estimation process, The device control process controls the device based on the action detected by the action detection process. The equipment control system according to claim 1.
  3. The system further includes an information acquisition unit that performs an information acquisition process to acquire environmental information relating to the environment in which the human body is present, The motion estimation process further detects the action based on the environmental information acquired by the information acquisition process. The equipment control system according to claim 2.
  4. The aforementioned device control process controls the device based on the aforementioned action using pre-stored automatic control information. The equipment control system according to claim 3.
  5. The system further includes a history storage unit that performs a history storage process to store automatic control history information relating to the control history of the device based on the aforementioned actions. The equipment control system according to claim 4.
  6. The automatic control history information associates the control content of the device based on the action with one or more pieces of information from the time the action was performed, the environmental information, and the person identifier that identifies the person. The equipment control system according to claim 5.
  7. The history accumulation process further accumulates manual control history information relating to the control history of the device based on operations performed on the device. The equipment control system according to claim 6.
  8. The manual control history information associates the control content of the device based on the operation with one or more pieces of information from the time the operation was performed, the environmental information, and the person identifier. The system further includes a learning unit that performs a learning process to update the automatic control information when the time and operation indicated by the manual control history information satisfy predetermined update conditions. The equipment control system according to claim 7.
  9. The system further includes a learning unit that performs a learning process to update the automatic control information when the time and control content indicated by the automatic control history information and the time and operation indicated by the manual control history information relating to the control history of the equipment based on the operation of the equipment satisfy predetermined update conditions. The equipment control system according to claim 5.
  10. The aforementioned virtual space is a three-dimensional space, The motion estimation process obtains a cluster group, which is a collection of one or more clusters, by performing clustering on the point cloud, and estimates the posture of the human body based on the distribution of the cluster group in the three-dimensional space. The equipment control system according to claim 2.
  11. The motion estimation process estimates the posture of the human body based on the ratios of the length, width, and height of the three-dimensional figure surrounding the cluster group. The equipment control system according to claim 10.
  12. The behavior detection process detects non-operational behavior, which is different from operation of the device by the human body, based at least on the posture or change in posture estimated by the posture estimation process. The aforementioned device control process controls the device at least based on the detected non-operational behavior. The equipment control system according to claim 2.
  13. A radio wave sensor that performs a transmit and receive operation, transmitting a transmitted wave, which is a radio wave modulated in a predetermined manner, toward real space, receiving a reflected wave from the real space, and outputting an output signal based on the transmitted wave and the reflected wave, A conversion unit performs a conversion process to convert the output signal output by the radio wave sensor into location-identifiable information that can identify the position of each of one or more objects present in the real space. A holding unit that performs a holding process to hold the location-identifiable information converted by the conversion process, A difference acquisition unit performs a difference acquisition process to acquire the difference between a plurality of location-identifiable pieces of information held by the holding process, A motion estimation unit performs motion estimation processing that estimates whether each of the one or more objects is a moving object or a stationary object based on the difference obtained by the difference acquisition process, identifies the position of the object estimated to be a moving object, and acquires motion position information indicating the identified position. A point placement unit performs a point placement process that virtually places points corresponding to the motion position information acquired by the motion estimation process into a virtual space corresponding to the real space, The system comprises an equipment control unit that performs equipment control processing based at least on a point cloud which is a set of points arranged in the virtual space, The difference acquisition process acquires two or more differences with different time differences between three or more location-identifiable pieces of information at different time points held by the retention process, The motion estimation process acquires two or more motion position information corresponding to two or more differences acquired by the difference acquisition process, The point placement process involves virtually placing two or more points corresponding to two or more pieces of motion position information with different time differences, obtained by the motion estimation process, in the virtual space. Equipment control system.
  14. A radio wave sensor that performs a transmit/receive operation, which involves transmitting a transmitted wave, which is a radio wave modulated in a predetermined manner, toward real space, receiving a reflected wave from the real space, and outputting an output signal based on the transmitted wave and the reflected wave, is used to make an estimation regarding a moving object, and to control the device based on the estimation result, A conversion step involves performing a conversion process to convert the output signal output by the radio wave sensor into location-identifiable information that can identify the position of each of one or more objects present in the real space. A holding step which includes a holding process that holds the location-identifiable information converted by the conversion process, A difference acquisition step involves performing a difference acquisition process to acquire the difference between a plurality of location-identifiable pieces of information held by the holding process, A motion estimation step involves performing a motion estimation process based on the difference obtained by the difference acquisition process, estimating whether each of the one or more objects is a moving object or a stationary object, identifying the position of the object estimated to be a moving object, and acquiring motion position information indicating the identified position. A point placement step involves performing a point placement process in which points corresponding to the motion position information acquired by the motion estimation process are virtually placed in a virtual space corresponding to the real space, The device control step includes controlling the device based at least on a point cloud which is a set of points arranged in the virtual space, The moving body is either a human body or a moving object other than a human body. The motion estimation process further estimates, at least based on the change in the difference, whether each of the two or more motion position information pieces obtained with different time differences corresponds to the human body or a moving object other than the human body. The point placement process virtually places points in the virtual space that correspond to the motion position information that the motion estimation process has estimated to correspond to the human body, out of the two or more motion position information acquired by the motion estimation process. Equipment control method.
  15. A radio wave sensor that performs a transmit/receive operation, which involves transmitting a transmitted wave, which is a radio wave modulated in a predetermined manner, toward real space, receiving a reflected wave from the real space, and outputting an output signal based on the transmitted wave and the reflected wave, is used to make an estimation regarding a moving object, and to control the device based on the estimation result, A conversion step involves performing a conversion process to convert the output signal output by the radio wave sensor into location-identifiable information that can identify the position of each of one or more objects present in the real space. A holding step which includes a holding process that holds the location-identifiable information converted by the conversion process, A difference acquisition step involves performing a difference acquisition process to acquire the difference between a plurality of location-identifiable pieces of information held by the holding process, A motion estimation step involves performing a motion estimation process based on the difference obtained by the difference acquisition process, estimating whether each of the one or more objects is a moving object or a stationary object, identifying the position of the object estimated to be a moving object, and acquiring motion position information indicating the identified position. A point placement step involves performing a point placement process in which points corresponding to the motion position information acquired by the motion estimation process are virtually placed in a virtual space corresponding to the real space, The device control step includes controlling the device based at least on a point cloud which is a set of points arranged in the virtual space, The difference acquisition process acquires two or more differences with different time differences between three or more location-identifiable pieces of information at different time points held by the retention process, The motion estimation process acquires two or more motion position information corresponding to two or more differences acquired by the difference acquisition process, The point placement process involves virtually placing two or more points corresponding to two or more pieces of motion position information with different time differences, obtained by the motion estimation process, in the virtual space. Equipment control method.
  16. A program for causing one or more processors to execute the device control method according to claim 14 or 15 .

Description

This disclosure relates to a device control system, a device control method, and a program, and more specifically, to a device control system, a device control method, and a program that uses radio wave sensors to estimate moving objects (motion bodies) such as the human body, and controls the device based on the estimation results. Patent Document 1 describes an FMCW radar device that transmits a frequency-modulated transmission signal such that its frequency changes linearly over time, receives a reception signal reflected by an object, and measures the distance to the object from the frequency of a beat signal obtained by multiplying the reception signal by the frequency-modulated local signal. Japanese Patent Publication No. 2022-108639 Figure 1 is a block diagram of an equipment control system (human body posture estimation system) according to an embodiment of this disclosure.Figure 2 is a conceptual diagram of a room in which the above-mentioned equipment control system is used.Figure 3 is a graph showing the frequency change of the transmitted wave sent by the radio wave sensor that constitutes the above-mentioned equipment control system.Figure 4 is a flowchart illustrating the operation of the control device that constitutes the above-mentioned equipment control system.Figure 5 is a flowchart illustrating the one-to-many frame difference acquisition process included in the above operation.Figure 6 is a flowchart illustrating the posture estimation process included in the above operation.Figure 7 is a waveform diagram illustrating an example of inter-frame difference (one-to-one inter-frame difference).Figure 8A is a frequency spectrum diagram showing the FFT result for the current frame Fr0, Figure 8B is a frequency spectrum diagram showing the FFT result for the subsequent frame Fr1, and Figure 8C is a frequency spectrum diagram showing the difference between the FFT results.Figure 9 is a waveform diagram illustrating another example of inter-frame differences (one-to-many inter-frame differences).Figure 10 is a conceptual diagram illustrating the characteristics of various human body movements.Figure 11 is a waveform diagram illustrating an example of one-to-many frame difference acquisition corresponding to various movements as described above.Figure 12A is a distribution map showing an example of cluster distribution in the standing position, one of the postures of the human body; Figure 12B is a distribution map showing an example of cluster distribution in the sitting position; and Figure 12C is a distribution map showing an example of cluster distribution in the lying position.Figure 13A is a conceptual diagram showing a three-dimensional figure surrounding cluster groups in a standing position, Figure 12B is a conceptual diagram showing a three-dimensional figure surrounding cluster groups in a sitting position, and Figure 12C is a conceptual diagram showing a three-dimensional figure surrounding cluster groups in a supine position.Figure 14 is a data structure diagram of the information group for automatic control.Figure 15 is a data structure diagram of the control history information.Figure 16 is a block diagram of a modified example of the same equipment control system. (1) Overview of the Equipment Control System The motion estimation system of this disclosure is, in this embodiment, an equipment control system 100 that has an equipment control function in addition to a motion estimation function (for example, a human body estimation function, in particular a human body posture estimation function). As shown in Figure 2, the equipment control system 100 of this embodiment uses a radio wave sensor 1 to make estimations regarding moving objects (hereinafter sometimes referred to as "motions"), in particular the human body 301. Estimations regarding the human body 301 include, for example, estimations of whether it is the human body 301, a moving object other than the human body 301, or a stationary object, as well as estimations of the posture of the human body 301. The posture of the human body 301 may be, for example, standing, sitting, or lying down (see Figures 12A to 12C), but is not limited to these. The device control system 100 detects various movements of the human body 301, and consequently, human behavior, based on the estimated changes in posture, and controls the device 200 according to the detection results. Furthermore, changes in posture include, but are not limited to, changes between standing, sitting, and lying positions. Changes in posture also include the continuation of no change in posture (for example, when a predetermined period has passed since changing to a sitting position, and no change back to standing, etc., occurs). The behaviors to be detected are, in particular, non-manipulative behaviors (described later), but manipulative behaviors (described later) may also be included. (1-1) Radio wave sensor The radio wave sensor 1 transmits radio waves (transmitte