EP-4737852-A1 - MODEL MONITORING FOR MACHINE LEARNING-BASED MAPPING SYSTEMS

EP4737852A1EP 4737852 A1EP4737852 A1EP 4737852A1EP-4737852-A1

Abstract

A system for monitoring performance of predictive models within robotic mapping systems. A prediction unit is communicatively coupled to one or more sensors configured to obtain measurements of an environment. The prediction unit comprises a predictive model operable to generate one or more features from a signal obtained from the one or more sensors. A mapping unit is configured to construct a map of the environment based at least in part on data derived from the signal and/or the one or more features. A monitoring unit is configured to determine a performance value associated with the one or more features generated by the predictive model, where the performance value quantifies a performance metric for the predictive model, compare the performance value to a performance criterion, and when the performance value does not satisfy the performance criterion, cause execution of an action to adjust operation of the mapping unit.

Inventors

MAYNARD, Magnus
SEALE, Cate
STEFFENS, Aljoscha

Assignees

Rovco Limited

Dates

Publication Date: 20260506
Application Date: 20240514

Claims (15)

A system for monitoring performance of predictive models within robotic mapping systems, the system comprising: a prediction unit communicatively coupled to one or more sensors configured to obtain measurements of an environment, the prediction unit comprising a first predictive model operable to generate one or more features from a signal obtained from the one or more sensors; a mapping unit configured to construct a map of the environment based at least in part on data derived from the signal obtained from the one or more sensors and/or the one or more features obtained from the first predictive model; and a monitoring unit comprising one or more processors configured to: determine a performance value associated with the one or more features generated by the first predictive model, wherein the performance value quantifies a performance metric for the first predictive model; compare the performance value to a performance criterion; and when the performance value does not satisfy the performance criterion, cause execution of an action to adjust operation of the mapping unit.
The system of claim 1 wherein the performance metric is a data similarity metric.
The system of claim 2 wherein the one or more processors of the monitoring unit are configured to determine the performance value for the data similarity metric by calculating a distance between: (i) one or more embeddings generated based on the signal obtained from the one or more sensors, and (ii) at least one predetermined embedding.
The system of claim 3 wherein the at least one predetermined embedding is generated during training of the first predictive model.
The system of either of claims 3 or 4 wherein the distance is calculated using one of: a Euclidean distance measure, a cosine distance measure, or a dot product distance measure.
The system of any of claims 3 to 5 wherein the distance is weighted according to at least one training performance score related to the at least one predetermined embedding.
The system of any of claims 3 to 6 wherein the distance is an average distance between the one or more embeddings and a plurality of predetermined embeddings.
The system of any preceding claim wherein the performance metric is a confidence metric.
The system of any preceding claim wherein the performance metric is an operational performance metric associated with a resource usage of the system, the resource usage corresponding to one or more of: a graphics processing unit, GPU, usage; a central processing unit, CPU, usage; a memory usage; a bandwidth; a model throughput metric; or a frames per second, FPS, metric.
The system of any preceding claim wherein the system comprises a robotic platform comprising the mapping unit and the one or more sensors.
The system of any preceding claim where execution of the action to adjust operation of the mapping unit causes one or more of: an adjustment to a speed, a movement direction, and/or a depth of the robotic platform; the mapping unit to bypass the first predictive model such that a subsequent map of the environment is generated based on data derived from signals obtained from the one or more sensors; the mapping unit to obtain features from a second predictive model of the prediction unit such that a subsequent map of the environment is generated based at least in part on features from the second predictive model; display of an alert viewable by a user of the system; and/or the mapping unit to regenerate the map of the environment based on the signal obtained from the one or more sensors.
The system of any preceding claim wherein the signal obtained from the one or more sensors include: an RGB image, an RGBD image, a SONAR image, a SONAR point cloud, a LIDAR point cloud, an inertial measurement signal, a GPS signal, or a doppler velocity log signal.
A method for adaptive operation of a mapping unit configured to construct a map of an environment based at least in part on features obtained from a predictive model, the method comprising: identifying, by one or more processors, one or more features generated by the predictive model; determining, by the one or more processors, a performance value associated with the one or more features generated by the predictive model, wherein the performance value quantifies a performance metric for the predictive model; comparing, by the one or more processors, the performance value to a performance criterion; and when the performance value does not satisfy the performance criterion, causing, by the one or more processors, execution of an action to adjust operation of the mapping unit.
A computer-readable medium comprising instructions which, when executed by one or more processors, cause the one or more processors to carry out the method of claim 13.
A robotic mapping system comprising one or more processors configured to carry out the method of claim 13.

Description

Field of Disclosure The present disclosure relates to environment mapping systems using machine learning models. Particularly, but not exclusively, the present disclosure relates to monitoring the performance of machine learning models used within environmental mapping systems; more particularly, but not exclusively, the present disclosure relates to adapting operation of environmental mapping systems based on the performance of machine learning models used within such systems. Background In the field of robotics, it is often necessary to build a map of the robot's environment. There are many techniques for constructing such environmental maps, such as Simultaneous Localization and Mapping (SLAM). Recently, machine learning has proven to be an effective way to enhance these techniques, either in part or as the entire end-to-end mapping process. However, the robustness of these machine learning-based mapping systems relies on the machine learning model performing correctly, otherwise an inaccurate map may be generated and unpredictable behaviours may occur. The accuracy and robustness of mapping systems is of particular importance within harsh environments such as subsea or space environments. As such, there is a need to improve the integration of machine learning models within environmental mapping systems. Summary of Disclosure According to an aspect of the present disclosure there is provided a system for monitoring performance of predictive models within robotic mapping systems. The system comprises a prediction unit communicatively coupled to one or more sensors configured to obtain measurements of an environment, the prediction unit comprising a first predictive model operable to generate one or more features from a signal obtained from the one or more sensors. The system further comprises a mapping unit configured to construct a map of the environment based at least in part on data derived from the signal obtained from the one or more sensors and/or the one or more features obtained from the first predictive model. The system further comprises a monitoring unit comprising one or more processors configured to determine a performance value associated with the one or more features generated by the first predictive model, wherein the performance value quantifies a performance metric for the first predictive model. The one or more processors are further configured to compare the performance value to a performance criterion, and when the performance value does not satisfy the performance criterion, cause execution of an action to adjust operation of the mapping unit. Adjusting the operation of the mapping unit based on the performance of the first predictive model has numerous benefits and advantages. First, it helps to improve the reliability and robustness of the mapping system by helping to ensure that the mapping system does not use sub-optimal or poor quality features/predictions generated by the first predictive model. This helps to improve the accuracy of the generated map by avoiding the degradation in performance that may occur from unreliable features or predictions. Furthermore, the actions performed in response to a drop in performance can be executed quickly without necessarily requiring the predictive model to be re-trained (which can be both computationally expensive and time consuming). These benefits help improve the effectiveness of the mapping unit in generating an accurate environment map which is particularly beneficial within harsh environments (e.g., subsea environments). The performance metric can be an outlier detector, a novelty detector, or a data similarity metric. The one or more processors of the monitoring unit can be configured to determine the performance value for the data similarity metric by calculating a distance between: (i) one or more embeddings generated based on the signal obtained from the one or more sensors, and (ii) at least one predetermined embedding. The at least one predetermined embedding can be generated during training of the predictive model. The distance can be calculated using a distance metric. The distance metric is one of: a Euclidean distance measure, a cosine distance measure, a Manhattan distance, a dot product distance measure, or any other suitable distance metric. The distance can be weighted according to at least one training performance score related to the at least one predetermined embedding. The distance can be an average distance between the one or more embeddings and a plurality of predetermined embeddings. Beneficially, monitoring similarity (e.g., data similarity, concept drift, etc.) allows the mapping system to be more robust in unseen condition by automatically re-configuring the mapping system to expect unreliable (i.e., out of distribution) outputs. This is particularly advantageous when the predictive model has not learned an adequate representation of the environment as it helps to avoid the mapping unit relying on inaccurate f