Search

EP-4004682-B1 - DISTRIBUTED SENSOR DATA PROCESSING USING MULTIPLE CLASSIFIERS ON MULTIPLE DEVICES

EP4004682B1EP 4004682 B1EP4004682 B1EP 4004682B1EP-4004682-B1

Inventors

  • OLWAL, ALEX
  • BALKE, KEVIN
  • VOTINTCEV, Dmitrii

Dates

Publication Date
20260513
Application Date
20201013

Claims (11)

  1. A method for distributed image recognition using a wearable device, the method comprising: receiving (1402), via at least one imaging sensor (1342, 1542a, 1542b, 1742a, 1742b) of the wearable device, image data; detecting (1404), by an image classifier (1303, 1503, 1703) of the wearable device, whether or not an object of interest is included within the image data, the image classifier (1303, 1503, 1703) executing a first machine-learning, ML, model (1326); in response to the object of interest being detected in the image data, transmitting (1406), via a wireless connection, the image data to a computing device, the image data configured to be used by a second ML model (1327) on the computing device for further image classification; receiving (1408), via the wireless connection, object location data from the computing device, the object location data identifying a location of the object of interest in the image data; identifying (1410), by an object tracker (1335) of the wearable device, an image region in subsequent image data captured by the at least one imaging sensor (1342, 1542a, 1542b, 1742a, 1742b) using the object location data, wherein the image region includes the object of interest, and wherein the image region in subsequent image data represents an area within the subsequent image data that has been cropped; and transmitting (1412), via the wireless connection, the image region to the computing device, the image region configured to be used by the second ML model (1327) for further image classification.
  2. The method of claim 1, further comprising: cropping, by the object tracker (1335), the image region from the subsequent image data; and compressing, by the object tracker (1335), the image region, wherein the compressed image region is transmitted to the computing device over the wireless connection.
  3. The method of claims 1 or 2, wherein the object of interest includes facial features.
  4. The method of any of claims 1 to 3, wherein the method further comprises: activating a first imaging sensor (1542a, 1742a) of the wearable device to capture first image data; detecting, by the image classifier (1303, 1503, 1703), whether the first image data includes the object of interest; and activating a second imaging sensor (1542b, 1742b) to capture second image data, the second image data having a quality higher than a quality of the first image data, wherein the second image data is transmitted to the computing device via the wireless connection, the second image data configured to be used by the second ML model (1327) for further image classification.
  5. The method of claim 4, further comprising: receiving, via a light condition sensor of the wearable device, light condition information; and activating the first imaging sensor (1542a, 1742a) based on the light condition information.
  6. The method of claim 4 or 5, further comprising: receiving, via a motion sensor of the wearable device, motion information; and activating the first imaging sensor (1542a, 1742a) based on the motion information.
  7. The method of any of claims 1 to 6, wherein the wireless connection is a short-range wireless connection, the wearable device includes smartglasses, and the computing device includes a smartphone.
  8. A non-transitory computer-readable medium storing executable instructions that when executed by at least one processor of a wearable device comprising an imaging sensor, a radio frequency transceiver, a light condition sensor and a motion sensor, cause the at least one processor to perform a method according to any of claims 1 to 7.
  9. A wearable device for distributed image recognition, the wearable device comprising: at least one imaging sensor (1342, 1542a, 1542b, 1742a, 1742b) configured to capture image data; an image classifier (1303, 1503, 1703) configured to detect whether or not an object of interest is included within the image data, the image classifier (1303, 1503, 1703) configured to execute a first machine-learning, ML, model (1326); and a radio frequency, RF, transceiver (1314) configured to transmit, via a wireless connection, in response to the object of interest being detected in the image data, the image data to a computing device, the image data configured to be used by a second ML model (1327) on the computing device to compute object location data, the object location data identifying a location of the object of interest in the image data; and wherein the RF transceiver (1314) is configured to receive, via the wireless connection, the object location data from the computing device, the object location data identifying a location of the object of interest in the image data; and an object tracker (1335) configured to identify an image region in subsequent image data captured by the at least one imaging sensor (1342, 1542a, 1542b, 1742a, 1742b) using the object location data, wherein the image region includes the object of interest, and wherein the image region in subsequent image data represents an area within the subsequent image data that has been cropped, wherein the RF transceiver (1314) is configured to transmit, via the wireless connection, the image region to the computing device, the image region configured to be used by the computing device for further image classification.
  10. The wearable device of claim 9, wherein the wearable device further comprises: a sensor trigger configured to activate a first imaging sensor (1542a, 1742a) to capture first image data, the image classifier (1303, 1503, 1703) is configured to detect whether the first image data includes the object of interest, the sensor trigger configured to activate a second imaging sensor (1542b, 1742b) to capture second image data in response to the object of interest being detected in the first image data, the second image data having a quality higher than a quality of the first image data, wherein the RF transceiver (1314) is configured to transmit the second image data to the computing device over the wireless connection.
  11. A computing device for distributed image recognition, the computing device including: at least one processor; and a non-transitory computer-readable medium storing executable instructions that when executed by the at least one processor cause the at least one processor to: receive, via a wireless connection, image data from a wearable device, the image data having an object of interest detected by an image classifier (1303, 1503, 1703) of the wearable device, the image classifier executing a first machine-learning, ML, model (1326); compute object location data based on the image data using a second ML model (1327), the object location data identifying a location of the object of interest in the image data; transmit, via the wireless connection, the object location data to the wearable device; receive, via the wireless connection, an image region in subsequent image data, wherein the image region includes the object of interest, and wherein the image region in subsequent image data represents an area within the subsequent image data that has been cropped; and execute, by the second ML model (1327), object classification on the image region.

Description

FIELD This disclosure relates to a distributed sensor data processing using multiple classifiers on multiple devices. BACKGROUND Computing devices (e.g., wearable devices, smartglasses, smart speakers, action cameras, etc.) are often relatively compact devices, and in some examples, may be on or around the body of a person for an extended period of time. However, computer processing requirements for processing sensor data (e.g., image data, audio data) can be relatively high especially for devices that include display and perception capabilities. For example, a device may perform energy-intensive operations (e.g., audio and/or image processing, computer vision, etc.) that requires a number of circuit components, which can cause several challenges. For example, the device may generate a relatively large amount of heat, thereby making the device uncomfortable to be in proximity to the skin for extended periods of time. In addition, the amount of circuit components (including batteries) adds weight to the device, thereby increasing the discomfort of wearing the device over an extended period of time. Further, the energy-intensive operations (in conjunction with the limitations on battery capacity) can cause the battery life to be relatively short. As such, some conventional devices can be used for only short durations throughout the day. An example of an information processing device is described in US 2016/080652 Al. SUMMARY This disclosure relates to a low-power device (e.g., smartglasses, wearable watches, portable action cameras, security cameras, smart speakers, etc.) that connects to a computing device (e.g., smartphone, laptop, tablet, etc.) over a wireless connection, where energy-intensive operations are offloaded to the computing device (or a server computer connected to the computing device), which can cause improvement to the device's performance (e.g., power, bandwidth, latency, computing capabilities, machine learning precision, etc.) and the user's experience. In some examples, the wireless connection is a short-range wireless connection such as a Bluetooth connection or near field communication (NFC) connection. In some examples, the low-power device includes a head-mounted display device such as smartglasses. However, the techniques discussed herein may be applied to other types of low-power devices such as portable action cameras, security cameras, smart doorbells, smart watches, etc. The invention is as set out in the independent claims. Preferable features of the invention are defined in the appended dependent claims. According to an aspect, a method for distributed sound recognition using a wearable device includes receiving, via a microphone of the wearable device, audio data, detecting, by a sound classifier of the wearable device, whether or not the audio data includes a sound of interest, where the sound classifier executes a first machine learning (ML) model, and transmitting, via a wireless connection, the audio data to a computing device in response to the sound of interest being detected within the audio data, where the audio data is configured to be used by a second ML model for further sound classification. According to an aspect, a non-transitory computer-readable medium storing executable instructions that when executed by at least one processor cause the at least one processor to receive audio data via a microphone of a wearable device, detect, by a sound classifier of the wearable device, whether or not the audio data includes a sound of interest, where the sound classifier is configured to execute a first machine learning (ML) model, and transmit, via a wireless connection, the audio data to a computing device in response to the sound of interest being detected within the audio data, where the audio data is configured to be used by a second ML model on the computing device for further sound classification. According to an aspect, a wearable device for distributed sound recognition includes a microphone configured to capture audio data, a sound classifier configured to detect whether or not the audio data includes a sound of interest, the sound classifier including a first machine learning (ML) model, and a radio frequency (RF) transceiver configured to transmit the audio data to a computing device via a wireless connection in response to the sound of interest being detected within the audio data, where the audio data is configured to be used by a second ML model to translate the sound of interest to text data. According to an aspect, a computing device for sound recognition including at least one processor, and a non-transitory computer-readable medium storing executable instructions that when executed by the at least one processor cause the at least one processor to receive, via a wireless connection, audio data from a wearable device, the audio data having a sound of interest detected by a sound classifier executing a first machine-learning (ML) model, determine whether to