US-12619883-B2 - Systems and methods for determining time-series feature importance of a model

Abstract

A system described herein may receive a set of outputs of a first model, which have been generated by the first model based on a set of inputs, and identify a set of historical values that correspond to the set of inputs and the set of outputs. The inputs and the historical values may be associated with the same time series. The system may train a second model based on the set of inputs to the first model, the set of outputs of the first model, and the set of historical values that correspond to the set of inputs and the set of outputs. The system may determine, based on training the second model, a set of weights associated with the set of historical values; and refine the first model based on the set of weights associated with the set of historical value.

Inventors

• Kushal SINGLA
• Subham BISWAS

Assignees

• VERIZON PATENT AND LICENSING INC.

Dates

Publication Date

20260505

Application Date

20211222

Claims (20)

1. 1 . A device, comprising: one or more processors configured to: receive a first set of outputs generated by a first model based on a set of inputs, the first set of outputs including one or more network actions and the set of inputs including one or more attributes of a wireless network; identify a set of historical actual values, determined by one or more sensors, that correspond to the set of inputs, wherein the first set of outputs, the set of inputs, and the historical actual values are associated with a particular time series that includes a plurality of time windows, wherein one or more particular historical actual values, one or more particular inputs, and one or more particular outputs of the first model are associated with a particular time window of the plurality of time windows; train a second model based on: the set of inputs to the first model, the first set of outputs of the first model, and the set of historical actual values that correspond to the set of inputs and the first set of outputs, wherein training the second model includes; determining, for particular respective time windows of the plurality of time windows, respective measures of similarity between the first set of outputs of the first model and the set of historical actual values; generating, by the second model, a second set of outputs based on the set of inputs; and comparing the second set of outputs to the set of inputs; determine, based on the determined respective measures of similarity between the first set of outputs of the first model and the set of historical actual values, and further based on comparing the second set of outputs to the first set of outputs, a set of weights associated with the set of historical actual values; refine the first model based on the set of weights associated with the set of historical actual values; use the refined first model to determine a particular modification with respect to the wireless network based on a particular set of attributes of the wireless network; and perform the determined particular modification, determined by the refined first model, with respect to the wireless network.
2. 2 . The device of claim 1 , wherein the set of inputs includes a first value that is associated with a first time window of the time series, and wherein the set of historical actual values includes at least a second value that is associated with a second time window of the time series that occurs prior.
3. 3 . The device of claim 1 , wherein the set of weights is a first set of weights, wherein the one or more processors are further configured to: determine, based on training the second model, a second set of weights associated with the set of inputs; and refine the first model further based on the second set of weights associated with the set of inputs.
4. 4 . The device of claim 1 , wherein performing the determined particular modification includes modifying one or more beamforming parameters of the wireless network.
5. 5 . The device of claim 1 , wherein training the second model includes determining values for the set of weights which, when applied by the second model to the set of inputs and the historical actual values, provide values that correspond to the outputs of the first model.
6. 6 . The device of claim 1 , wherein the first set of outputs of the first model include one or more predicted values that are based on one or more particular inputs of the set of inputs, and wherein the historical actual values include one or more ground truth values that are associated with the one or more particular inputs of the set of inputs.
7. 7 . The device of claim 1 , wherein determining a particular weight, associated with a particular time window of the plurality of time windows, is based on: a particular set of outputs of the first model that correspond to the particular time window, and one or more particular historical actual values, of the set of historical actual values, that correspond to the particular time window.
8. 8 . A non-transitory computer-readable medium, storing a plurality of processor-executable instructions to: receive a first set of outputs generated by a first model based on a set of inputs, the first set of outputs including one or more network actions and the set of inputs including one or more attributes of a wireless network; identify a set of historical actual values, determined by one or more sensors, that correspond to the set of inputs, wherein the first set of outputs, the set of inputs, and the historical actual values are associated with a particular time series that includes a plurality of time windows, wherein one or more particular historical actual values, one or more particular inputs, and one or more particular outputs of the first model are associated with a particular time window of the plurality of time windows; train a second model based on: the set of inputs to the first model, the first set of outputs of the first model, and the set of historical actual values that correspond to the set of inputs and the first set of outputs, wherein training the second model includes; determining, for particular respective time windows of the plurality of time windows, respective measures of similarity between the first set of outputs of the first model and the set of historical actual values; generating, by the second model, a second set of outputs based on the set of inputs; and comparing the second set of outputs to the set of inputs; determine, based on the determined respective measures of similarity between the first set of outputs of the first model and the set of historical actual values, and further based on comparing the second set of outputs to the first set of outputs, a set of weights associated with the set of historical actual values; refine the first model based on the set of weights associated with the set of historical actual values; use the refined first model to determine a particular modification with respect to the wireless network based on a particular set of attributes of the wireless network; and perform the determined particular modification, determined by the refined first model, with respect to the wireless network.
9. 9 . The non-transitory computer-readable medium of claim 8 , wherein the set of inputs includes a first value that is associated with a first time window of the time series, and wherein the set of historical actual values includes at least a second value that is associated with a second time window of the time series that occurs prior to the first time window of the time series.
10. 10 . The non-transitory computer-readable medium of claim 8 , wherein the set of weights is a first set of weights, wherein the plurality of processor-executable instructions further include processor-executable instructions to: determine, based on training the second model, a second set of weights associated with the set of inputs; and refine the first model further based on the second set of weights associated with the set of inputs.
11. 11 . The non-transitory computer-readable medium of claim 8 , wherein performing the determined particular modification includes modifying one or more beamforming parameters of the wireless network.
12. 12 . The non-transitory computer-readable medium of claim 8 , wherein training the second model includes determining values for the set of weights which, when applied by the second model to the set of inputs and the historical actual values, provide values that correspond to the outputs of the first model.
13. 13 . The non-transitory computer-readable medium of claim 8 , wherein the first set of outputs of the first model include one or more predicted values that are based on one or more particular inputs of the set of inputs, and wherein the historical actual values include one or more ground truth values that are associated with the one or more particular inputs of the set of inputs.
14. 14 . The non-transitory computer-readable medium of claim 8 , wherein determining a particular weight, associated with a particular time window of the plurality of time windows, is based on: a particular set of outputs of the first model that correspond to the particular time window, and one or more particular historical actual values, of the set of historical actual values, that correspond to the particular time window.
15. 15 . A method, comprising: receiving a first set of outputs generated by a first model based on a set of inputs, the first set of outputs including one or more network actions and the set of inputs including one or more attributes of a wireless network; identifying a set of historical actual values, determined by one or more sensors, that correspond to the set of inputs, wherein the first set of outputs, the set of inputs, and the historical actual values are associated with a particular time series that includes a plurality of time windows, wherein one or more particular historical actual values, one or more particular inputs, and one or more particular outputs of the first model are associated with a particular time window of the plurality of time windows; training a second model based on: the set of inputs to the first model, the first set of outputs of the first model, and the set of historical actual values that correspond to the set of inputs and the first set of outputs, wherein training the second model includes; determining, for particular respective time windows of the plurality of time windows, respective measures of similarity between the first set of outputs of the first model and the set of historical actual values; generating, by the second model, a second set of outputs based on the set of inputs; and comparing the second set of outputs to the set of inputs; determining, based on the determined respective measures of similarity between the first set of outputs of the first model and the set of historical actual values, and further based on comparing the second set of outputs to the first set of outputs, a set of weights associated with the set of historical actual values; refining the first model based on the set of weights associated with the set of historical actual values; using the refined first model to determine a particular modification with respect to the wireless network based on a particular set of attributes of the wireless network; and performing the determined particular modification, determined by the refined first model, with respect to the wireless network.
16. 16 . The method of claim 15 , wherein the set of inputs includes a first value that is associated with a first time window of the time series, and wherein the set of historical actual values includes at least a second value that is associated with a second time window of the time series that occurs prior.
17. 17 . The method of claim 15 , wherein the set of weights is a first set of weights, the method further comprising: determining, based on training the second model, a second set of weights associated with the set of inputs; and refining the first model further based on the second set of weights associated with the set of inputs.
18. 18 . The method of claim 15 , wherein performing the determined particular modification includes modifying one or more beamforming parameters of the wireless network.
19. 19 . The method of claim 15 , wherein training the second model includes determining values for the set of weights which, when applied by the second model to the set of inputs and the historical actual values, provide values that correspond to the outputs of the first model.
20. 20 . The method of claim 15 , wherein the first set of outputs of the first model include one or more predicted values that are based on one or more particular inputs of the set of inputs, and wherein the historical actual values include one or more ground truth values that are associated with the one or more particular inputs of the set of inputs.

Description

BACKGROUND Systems may utilize models, such as machine learning models, to aid in the performance of various functions such as pattern recognition, analysis of real-world data, automated network remediation, etc. Some models may be refined through training or some other suitable process, in which configuration parameters of the models (e.g., weights for certain features) are modified. BRIEF DESCRIPTION OF THE DRAWINGS FIGS. 1A and 1B illustrate an example overview of one or more embodiments described herein; FIG. 2 illustrates an example of utilizing historical values as a feature based on which a model may be refined, in accordance with some embodiments; FIG. 3 illustrates an example of determining a set of output-keyed features, in accordance with some embodiments; FIG. 4 illustrates an example set of weights that may be determined with respect to a set of output-keyed features associated with a model, in accordance with some embodiments; FIG. 5 illustrates example values based on which a time series of historical values may be determined as having relative significance for a mode, in accordance with some embodiments; FIG. 6 illustrates an example process for refining a model based on identifying weights associated with a time series of historical data associated with a set of inputs to the model, in accordance with some embodiments; FIG. 7 illustrates an example environment in which one or more embodiments, described herein, may be implemented; FIG. 8 illustrates an example arrangement of a radio access network (“RAN”), in accordance with some embodiments; and FIG. 9 illustrates example components of one or more devices, in accordance with one or more embodiments described herein. DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS The following detailed description refers to the accompanying drawings. The same reference numbers in different drawings may identify the same or similar elements. Embodiments described herein provide for the determination of feature importance weights associated with one or more models. In some situations, such models may be associated with private or otherwise unascertainable feature importance weights, such as models that are generated, modified, provided, etc. by one or more entities that make the inner workings of the model (e.g., the feature importance weights and/or other attributes or parameters of the model) private or otherwise not unascertainable. As such, the models may become more explainable, and the predictive ability of the models may be able to be better quantified. Further, embodiments herein may identify historical actual data (e.g., a “ground truth” time series of actual values, as compared to outputs of the models) as a feature based on which the performance of the model may be refined. In this sense, the model may be augmented, enhanced, etc. to take elements of time into account when generating outputs, thus enhancing the accuracy, predictive ability, etc. of the model. Models that are enhanced, refined, etc. according to embodiments described herein may be used in the automated operations of real-world systems, such as wireless networks, autonomous vehicles, smart homes, etc. For example, a particular model may determine a prediction of a set of performance metrics, wireless coverage areas, etc. associated with a wireless network (or a portion thereof, such as a cell, a base station, a sector, etc.) based on a set of input parameters such as locations of base stations of the wireless network, beamforming parameters of one or more antennas associated with the wireless network (e.g., azimuth angle, beam transmit power, beam width, etc.), quantities of connected User Equipment (“UEs”), etc. Such predictions may be used to modify parameters of the wireless network in order to increase one or more measures of yield, such as actual performance metrics experienced by UEs connected to the wireless network. For example, a particular model may generate parameters for the wireless network, such as beamforming parameters (e.g., modifications to azimuth angle, beam transmit power, beam width, etc.) based on a set of input parameters. As discussed herein, time series-related information (e.g., historical actual values), which may not necessarily be accounted for by such models, may further be used to enhance such models, such that such models are further able to utilize historical actual values in predicting attributes or metrics associated with the wireless network, and are further able to perform one or more actions (e.g., modifications to network parameters) based on such predicted attributes or metrics. As shown in FIG. 1A, for example, model 101 may receive, as input, a set of features 103. Model 101 may be, for example, a predictive model, an artificial intelligence/machine learning (“AI/ML”) model, a statistical model, and/or other type of model that generates or otherwise provides one or more outputs 105 based on one or more sets of inputs (e.g., features 103, in thi