EP-4738030-A1 - IDENTIFYING MACHINE SUBSTATES

EP4738030A1EP 4738030 A1EP4738030 A1EP 4738030A1EP-4738030-A1

Abstract

State data transmitted from a machine is received by a server (502), where the state data include telemetry data from one or more sensors on the machine, and the telemetry data are collected at a specified frequency. The state data is processed by the server to obtain transformed data (504). The transformed data is input into a machine-learning model (508), where the machine-learning model is configured to output an operational substate of the machine based on the transformed data. A substate prediction of the machine is generated (510) using the machine-learning model, where the substate prediction includes a probability value for each possible substate. The substate prediction is then output to a user interface for display (512).

Inventors

DEO, Mayur
Enderle, John
PEDERSEN, Ryan
LUSTY, WILLIAM
Asonye, Iroh
Hemphill, Tristan
MARQUIS, Timothy

Assignees

Deere & Company

Dates

Publication Date: 20260506
Application Date: 20251007

Claims (15)

A method for predicting a substate of a machine, comprising: receiving, by a server, state data transmitted from the machine, wherein the state data include telemetry data from one or more sensors on the machine, and the telemetry data are collected at a specified frequency; processing, by the server, the state data to obtain transformed data; inputting the transformed data into a machine-learning model, wherein the machine-learning model is configured to output an operational substate of the machine based on the transformed data; generating, using the machine-learning model, a substate prediction of the machine, wherein the substate prediction includes a probability value for each possible substate; and outputting the substate prediction to a user interface for display.
The method of claim 1, further comprising: training the machine-learning model by receiving historical telemetry data from the machine, wherein the historical telemetry data include one or more of sensor readings, machine settings, or operator inputs; assigning ground truth labels to the historical telemetry data, wherein the ground truth labels correspond to operational substates of the machine; grouping consecutive data points of the historical telemetry data that share a same ground truth label into unified segments representing cohesive machine activities; and using the grouped and labeled historical telemetry data to train the machine-learning model to classify the operational substates.
The method of claim 2, wherein assigning the ground truth labels comprises: identifying contexts for association with the historical telemetry data, wherein the contexts include surrounding conditions of a location of the machine, a type of operation being conducted, and machine actions over a time frame before and after a moment of interest; assigning proto labels to data points based on key features extracted from the historical telemetry data; and refining the proto labels based on temporal context and adjacent machine activities to generate definitive substate labels.
The method of claim 2, wherein grouping the consecutive data points comprises: combining consecutive data points classified into a same preliminary substate into unified events; merging related groups that exhibit temporal continuity by eliminating time intervals of a predetermined duration between events of a same classification; and combining the merged groups to form larger operational events representing machine behavior patterns.
The method of claim 1, wherein processing, by the server, the state data to obtain the transformed data comprises: performing data cleaning to remove corrupted, incomplete, or outlier data points; extracting features from raw telemetry data to derive meaningful attributes; and normalizing numerical inputs using mean and standard deviation re-scaling to center values around a mean and scale based on standard deviation.
The method of claim 1, further comprising: selecting the machine-learning model from a plurality of available machine-learning models based on a type of the machine, wherein different machine-learning models are trained for different machine types.
The method of claim 1, wherein the state data further includes telemetry data from an other machine that operates in tandem with the machine, and wherein the telemetry data from the other machine are also input into the machine-learning model for determining the substate of the machine.
The method of claim 1, wherein the telemetry data includes one or more of speed, engine temperature, tank level, location coordinates, or engine load.
The method of claim 1, wherein the machine-learning model is a random forest model.
The method of claim 1, further comprising: determining that the substate indicates an operational status that requires intervention; and transmitting an alert to a remote device, in response to determining that the substate indicates the operational status that requires the intervention.
A system, comprising: a memory subsystem; and processing circuitry, the processing circuitry configured to execute instructions stored in the memory subsystem to: receive state data transmitted from a machine, wherein the state data include telemetry data from one or more sensors on the machine, and the telemetry data are collected at a specified frequency; process the state data to obtain transformed data; input the transformed data into a machine-learning model, wherein the machine-learning model is configured to output an operational substate of the machine based on the transformed data; generate, using the machine-learning model, a substate prediction of the machine, wherein the substate prediction includes a probability value for each possible substate; and output the substate prediction to a user interface for display.
The system of claim 11, the processing circuitry further configured to execute instructions in the memory subsystem to: train the machine-learning model by receiving historical telemetry data from the machine, wherein the historical telemetry data include one or more of sensor readings, machine settings, or operator inputs; assign ground truth labels to the historical telemetry data, wherein the ground truth labels correspond to operational substates of the machine; group consecutive data points of the historical telemetry data that share a same ground truth label into unified segments representing cohesive machine activities; and use the grouped and labeled historical telemetry data to train the machine-learning model to classify the operational substates.
The system of claim 12, wherein, to assign the ground truth labels, the processing circuitry configured to execute instructions stored in the memory subsystem to: identify contexts for association with the historical telemetry data, wherein the contexts include surrounding conditions of a location of the machine, a type of operation being conducted, and machine actions over a time frame before and after a moment of interest; assign proto labels to data points based on key features extracted from the historical telemetry data; and refine the proto labels based on temporal context and adjacent machine activities to generate definitive substate labels.
The system of claim 12, wherein, to group the consecutive data points, the processing circuitry configured to execute instructions stored in the memory subsystem to: combine consecutive data points classified into a same preliminary substate into unified events; merge related groups that exhibit temporal continuity by eliminating time intervals of a predetermined duration between events of a same classification; and combine the merged groups to form larger operational events representing machine behavior patterns.
One or more non-transitory computer-readable storage media comprising instructions that, when executed by one or more processors, perform operations for predicting a substate of a machine, the operations comprising: receiving, by a server, state data transmitted from the machine, wherein the state data include telemetry data from one or more sensors on the machine, and the telemetry data are collected at a specified frequency; processing, by the server, the state data to obtain transformed data; inputting the transformed data into a machine-learning model, wherein the machine-learning model is configured to output an operational substate of the machine based on the transformed data; generating, using the machine-learning model, a substate prediction of the machine, wherein the substate prediction includes a probability value for each possible substate; and outputting the substate prediction to a user interface for display.

Description

This disclosure relates to machine operations and more specifically to identifying substates of machines using machine learning. SUMMARY Disclosed herein are implementations of predicting machine substates. One aspect of the disclosed implementations relates to a method for predicting a substate of a machine. The method includes receiving, by a server, state data transmitted from the machine, wherein the state data include telemetry data from one or more sensors on the machine, and the telemetry data are collected at a specified frequency; processing, by the server, the state data to obtain transformed data; inputting the transformed data into a machine-learning model, wherein the machine-learning model is configured to output an operational substate of the machine based on the transformed data; generating, using the machine-learning model, a substate prediction of the machine, wherein the substate prediction includes a probability value for each possible substate; and outputting the substate prediction to a user interface for display. One aspect of the disclosed implementations relates to a system, including: a memory subsystem and processing circuitry. The processing circuitry is configured to execute instructions stored in the memory subsystem to receive state data transmitted from a machine, wherein the state data include telemetry data from one or more sensors on the machine, and the telemetry data are collected at a specified frequency; process the state data to obtain transformed data; input the transformed data into a machine-learning model, wherein the machine-learning model is configured to output an operational substate of the machine based on the transformed data; generate, using the machine-learning model, a substate prediction of the machine, wherein the substate prediction includes a probability value for each possible substate; and output the substate prediction to a user interface for display. One aspect of the disclosed implementations relates to one or more non-transitory computer-readable storage media including instructions that, when executed by one or more processors, perform operations for predicting a substate of a machine. The operations include receiving, by a server, state data transmitted from the machine, wherein the state data include telemetry data from one or more sensors on the machine, and the telemetry data are collected at a specified frequency; processing, by the server, the state data to obtain transformed data; inputting the transformed data into a machine-learning model, wherein the machine-learning model is configured to output an operational substate of the machine based on the transformed data; generating, using the machine-learning model, a substate prediction of the machine, wherein the substate prediction includes a probability value for each possible substate; and outputting the substate prediction to a user interface for display. These and other aspects of the present disclosure are disclosed in the following detailed description of the embodiments, the appended claims and the accompanying figures. BRIEF DESCRIPTION OF THE DRAWINGS The disclosure is best understood from the following detailed description when read in conjunction with the accompanying drawings. It is emphasized that, according to common practice, the various features of the drawings are not to-scale. On the contrary, the dimensions of the various features are arbitrarily expanded or reduced for clarity. FIG. 1 is a block diagram of a system for determining machine substates.FIG. 2 is a block diagram of a computing device.FIGS. 3A-3B illustrate an example of a technique for training an ML model to identify machine substate.FIG. 4 is a block diagram of example functionality of a substate prediction software.FIG. 5 is an example of a technique for determining a substate of a machine.FIG. 6 illustrates a table that presents examples of machine types, their associated states, and corresponding substates.FIG. 7A illustrates an example of a user interface that serves as an operations center for monitoring and managing agricultural machinery within a field.FIG. 7B illustrates a user interface that provides an example of how machine substates may be displayed on a user interface of a machine.FIG. 8 is a flowchart of an example of a technique for predicting machine substate. DETAILED DESCRIPTION Determining or identifying the state of a farming machine, while crucial, is often not sufficient to fully optimize operations (e.g., farming operations). For example, modern farming involves highly dynamic activities where machines, such as sprayers and harvesters, transition through various operational stages. Simply knowing the current state-whether a machine is idle, moving, or performing a specific function-provides a limited snapshot that lacks the necessary context for timely and effective decision-making. Conventional telemetry systems typically may provide state information that is either too generic or