US-12622647-B2 - Runtime assessment of sensors

US12622647B2US 12622647 B2US12622647 B2US 12622647B2US-12622647-B2

Abstract

This relates to the use of sensor evaluation in a multi-sensor environment. In a first aspect, this specification describes apparatus comprising: at least one processor; and at least one memory including computer program code. The at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to perform: receive sensor data from a plurality of sensors collected during a first time period; process the received sensor data through a plurality of layers of a neural network to generate an output indicative of the sensing quality of each of the plurality of sensors for a task; and cause a subset of the plurality of sensors to collect data during a second time period based on the output indicative of the suitability of each of the plurality of sensors for the task.

Inventors

Chulhong Min
Alessandro Montanari
Fahim Kawsar
Akhil Mathur

Assignees

NOKIA TECHNOLOGIES OY

Dates

Publication Date: 20260512
Application Date: 20200831

Claims (19)

1 . An apparatus comprising at least one processor; and at least one memory including computer program code; the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus at least to perform: generate ground truth quality data for a plurality of sensors, the ground truth quality data comprising sensor data collected by the plurality of the sensors and a corresponding quality value for an inference task, wherein the corresponding quality value indicates an accuracy of the inference task using the sensor data; process the sensor data from the ground truth quality data through a plurality of layers of a neural network to generate an output indicative of sensing quality of the plurality of the sensors for the inference task; update weights of the neural network in dependence on a comparison of the output indicative of the sensing quality of the plurality of sensors to the quality values of the ground truth quality data; select one or more of the plurality of the sensors based on the output indicative of the sensing quality of the one or more of the plurality of the sensors for the inference task; cause the one or more of the plurality of the sensors to collect data during a second time period for the inference task; and apply a sensing model of the selected one or more sensors to the collected data to generate a probability distribution over a plurality of classes for the inference task.
2 . The apparatus of claim 1 , wherein the generating of the ground truth quality data for the plurality of the sensors comprises, for one or more sensors in the plurality of the sensors: apply a sensing model to sensor data collected from a sensor in the plurality of the sensors to generate a predicted class for said sensor data; and generate the ground truth quality data in dependence on the comparison of the predicted class for said sensor data to a known class for said sensor data.
3 . The apparatus of claim 1 , wherein operations of the processing of the sensor data from the ground truth quality data and the updating of the weights of the neural network are iterated until a threshold condition is met.
4 . The apparatus of claim 1 , wherein the comparison of the output indicative of the sensing quality of the plurality of the sensors for the inference task to the quality values of the ground truth quality data is performed using a loss function.
5 . The apparatus of claim 4 , wherein the weights are determined by applying an optimization procedure to the loss function.
6 . The apparatus of claim 1 , wherein the neural network comprises: a plurality of input sub-networks, each input sub-network configured to receive as input sensor data from one of the plurality of the sensors and to extract one or more features from said input sensor data; a plurality of fully connected layers configured to process features extracted by the plurality of the input sub-networks and generate the output indicative of the suitability of each of the plurality of the sensors for the inference task.
7 . The apparatus of claim 6 , wherein two or more of the input sub-networks have identical weights.
8 . The apparatus of claim 1 , wherein the output indicative of the suitability of the plurality of the sensors for the inference task is a set of binary values, each binary value associated with one of the sensors in the plurality of the sensors and indicative of the suitability of said one of the sensors for performing the inference task.
9 . The apparatus of claim 1 , wherein the apparatus is a user device, a smartphone, a smartwatch, a smart earbud, a tablet device or a server.
10 . The apparatus of claim 1 , wherein the plurality of the sensors are implemented in one or more of the user device, the smartphone, the smartwatch, the smart earbud, the tablet device or the server.
11 . The apparatus of claim 1 , wherein the inference task comprises one or more of human activity recognition, hot-word recognition, health monitoring, environmental monitoring, physiological monitoring, and/or exercise monitoring.
12 . A method comprising: generating ground truth quality data for a plurality of sensors, the ground truth quality data comprising sensor data collected by the plurality of the sensors and a corresponding quality value for an inference task, wherein the corresponding quality value indicates an accuracy of the inference task using the sensor data; processing the sensor data from the ground truth quality data through a plurality of layers of a neural network to generate an output indicative of sensing quality of the plurality of the sensors for the inference task; updating weights of the neural network in dependence on a comparison of the output indicative of the sensing quality of the plurality of sensors to the quality values of the ground truth quality data; selecting one or more of the plurality of the sensors based on the output indicative of the sensing quality of the one or more of the plurality of the sensors for the inference task; causing the one or more of the plurality of the sensors to collect data during a second time period for the inference task; and applying a sensing model of the selected one or more sensors to the collected data to generate a probability distribution over a plurality of classes for the inference task.
13 . The method of claim 12 , wherein the generating of the ground truth quality data for the plurality of the sensors comprises, for one or more sensors in the plurality of the sensors: applying a sensing model to sensor data collected from a sensor in the plurality of the sensors to generate a predicted class for said sensor data; and generating the ground truth quality data in dependence on the comparison of the predicted class for said sensor data to a known class for said sensor data.
14 . The method of claim 12 , wherein operations of the processing of the sensor data from the ground truth quality data and the updating of the weights of the neural network are iterated until a threshold condition is met.
15 . The method of claim 12 , wherein the comparison of the output indicative of the sensing quality of the plurality of the sensors for the inference task to the quality values of the ground truth quality data is performed using a loss function.
16 . The method of claim 15 , wherein the weights are determined by applying an optimization procedure to the loss function.
17 . The method of claim 12 , wherein the neural network comprises: a plurality of input sub-networks, each input sub-network configured to receive as input sensor data from one of the plurality of the sensors and to extract one or more features from said input sensor data; a plurality of fully connected layers configured to process features extracted by the plurality of the input sub-networks and generate the output indicative of the suitability of each of the plurality of the sensors for the inference task.
18 . A non-transitory computer readable medium comprising program instructions for causing an apparatus to perform: generating ground truth quality data for a plurality of sensors, the ground truth quality data comprising sensor data collected by the plurality of the sensors and a corresponding quality value for an inference task, wherein the corresponding quality value indicates an accuracy of the inference task using the sensor data; processing the sensor data from the ground truth quality data through a plurality of layers of a neural network to generate an output indicative of sensing quality of the plurality of the sensors for the inference task; updating weights of the neural network in dependence on a comparison of the output indicative of the sensing quality of the plurality of sensors to the quality values of the ground truth quality data; selecting one or more of the plurality of the sensors based on the output indicative of the sensing quality of the one or more of the plurality of the sensors for the inference task; causing the one or more of the plurality of the sensors to collect data during a second time period for the inference task; and applying a sensing model of the selected one or more sensors to the collected data to generate a probability distribution over a plurality of classes for the inference task.
19 . The apparatus of claim 1 , wherein a sensor of the plurality of sensors is associated with a respective sensor model.

Description

FIELD This relates to the field of sensor evaluation. More particularly, this relates to the use of sensor evaluation in a multi-sensor environment. BACKGROUND The increasing availability of multiple sensory devices on or near a human body has opened brand new opportunities to leverage redundant sensory signals for the development of powerful human sensing applications. For instance, personal-scale sensory inferences with time-varying signals (e.g., motion, audio etc.) can be performed individually on each of a plurality of devices (e.g. a smartphone, a smartwatch, and even an earbud), each offering unique sensor quality, model accuracy, runtime behaviour and usage dynamics. At execution time, however, it is incredibly challenging to assess and compare these characteristics to select the best device for accurate and resource-efficient sensory inferences. Moreover, the presence of redundant sensors is not only limited to human sensing scenarios; in industrial automation systems, the deployment of redundant sensors is widely used to ensure reliability of the system. For example, in a future where self-driving cars are the norm, several redundant sensors will ensure high levels of safety and reliability. Sensors outside and inside the car will work together with wearable sensors on the driver's body to monitor the conditions of the road, of the car and the psychophysical status of the driver. SUMMARY In a first aspect, this specification describes apparatus comprising: at least one processor; and at least one memory including computer program code. The at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to perform: receive sensor data from a plurality of sensors collected during a first time period; process the received sensor data through a plurality of layers of a neural network to generate an output indicative of the sensing quality of each of the plurality of sensors for a task; and cause a subset of the plurality of sensors to collect data during a second time period based on the output indicative of the suitability of each of the plurality of sensors for the task. The neural network may comprise: a plurality of input sub-networks, each input sub-network configured to receive as input sensor data from one of the plurality of sensors and to extract one or more features from said input sensor data; and a plurality of fully connected layers, the plurality of fully connected layers configured to process features extracted by the plurality of input sub-networks and generate the output indicative of the suitability of each of the plurality of sensors for the given task. Each of the input sub-networks may comprise one or more convolutional layers. Two or more of the input sub-networks may have identical weights. The output indicative of the suitability of each of the plurality of sensors for the task may be a set of binary values, each binary value associated with one of the sensors in the plurality of sensors and indicative of the suitability of said one of the sensors for performing the task. The first period of time may be a fixed period of time. The first period of time may be between about 1 and 5 seconds, for example between 2 and 4 seconds. The second period of time may be a fixed period of time, for example between 1 and 60 seconds, such as between 20 and 50 seconds. The second period of time may be a variable period of time dependent on a threshold quality of the sensor data collected at the second period of time. The method performed by the apparatus may be iterated after the second period of time has elapsed. The apparatus may be: a user device; a smartphone; a smartwatch; a smart earbud; or a tablet device. The apparatus may be a remote server. The task may comprise one or more of: human activity recognition; hot-word recognition; health monitoring; environmental monitoring; physiological monitoring; and/or exercise monitoring. The plurality of sensors may comprise one or more of: an accelerometer; a microphone; a gyroscope; a thermometer; a magnetometer; a positioning sensor; a light sensor; a pedometer; a barometer; a heart rate sensor; and/or a humidity sensor. In a further aspect, this specification describes a system comprising: a plurality of user devices, each user device comprising one or more sensors; and any of apparatus described herein. In a further aspect, this specification describes a method comprising: receiving sensor data from a plurality of sensors collected during a first time period; processing the received sensor data through a plurality of layers of a neural network to generate an output indicative of the sensing quality of each of the plurality of sensors for a task; and causing a subset/one or more of the plurality of sensors to collect data during a second time period based on the output indicative of the suitability of each of the plurality of sensors for the task. The neural network may comprise