EP-4738778-A1 - MODEL TEST METHOD, DEVICES, AND STORAGE MEDIUM

EP4738778A1EP 4738778 A1EP4738778 A1EP 4738778A1EP-4738778-A1

Abstract

The present disclosure relates to a model test method, devices, and a storage medium. The method comprises : determining whether predicted data output by a first model meets a first condition, wherein the first condition is a performance requirement for the predicted data, the predicted data is data obtained by means of the first model performing prediction according to first measured data, the first measured data is data obtained after a terminal device performs measurement on first information, and the first information is information sent by a network device to the terminal device. Thus, a first model can be tested or verified to determine whether the first model meets a first condition, thereby determining the performance and reliability of the model, achieving online verification and testing of the model, and improving the flexibility and reliability of model management.

Inventors

TAO, Xuhua

Assignees

Beijing Xiaomi Mobile Software Co., Ltd.

Dates

Publication Date: 20260506
Application Date: 20230627

Claims (20)

A model test method, comprising: determining whether predicted data output by a first model meets a first condition; wherein the first condition is a performance requirement of the predicted data, the predicted data is data predicted by the first model based on first measurement data, the first measurement data is data obtained after a terminal device measures first information, and the first information is information sent from a network device to the terminal device.
The method according to claim 1, further comprising: receiving second information sent by the terminal device, the second information including the predicted data.
The method according to claim 1, further comprising: sending third information to the terminal device; the third information indicating whether the first model is usable by the terminal device.
The method according to any one of claims 1 to 3, further comprising: receiving fourth information; wherein the fourth information is configured to trigger the network device to send the first information, the fourth information comprises a first model identifier of the first model; and determining, based on the fourth information, that the first model meets a test initiation condition, and sending the first information to the terminal device.
The method according to claim 4, wherein the test initiation condition comprises at least one of: the first model has not been tested; the first model is not in a model management set; the model management set comprises models that have already been tested; or the fourth information is configured to indicate the execution of a test for the first model.
The method according to claim 4 or 5, wherein the fourth information comprises at least one of: indication information for model update; indication information for model addition; or indication information for model test.
The method according to any one of claims 1 to 6, wherein the first model is a model for performing beam prediction, the predicted data comprises a predicted signal quality of a first beam, the first beam is at least one beam configured by the network device for the terminal device; and the first condition comprises a difference between the predicted signal quality and a first signal quality being within a first difference range; wherein the first signal quality is a signal quality of the first beam generated by the network device according to beam configuration information, or the first signal quality is a signal quality of the first beam actually measured by the terminal device.
The method according to any one of claims 1 to 6, wherein the first model is a model for performing beam prediction, and the predicted data comprises a second beam that meets a signal condition and is obtained by the beam prediction; the signal condition comprises at least one of: a signal quality of the second beam is greater than or equal to a preset signal quality threshold; the second beam is a beam with the best signal quality among multiple beams configured by the network device for the terminal device.
The method according to claim 8, wherein the first condition is that a number of times a prediction success condition is met in N beam predictions is greater than or equal to M; wherein N is a natural number, and M is a natural number less than or equal to N; the prediction success condition is that the second beam and a third beam are the same, and the second beam is a beam that meets the signal condition and is obtained by the beam prediction; and the third beam is a beam that meets the signal condition determined by the network device according to beam configuration information; or the third beam is a beam that meets the signal condition determined according to second measurement data, and the second measurement data is data obtained by the terminal device after the terminal device measures multiple beams configured by the network device for the terminal device.
The method according to any one of claims 7 to 9, wherein the signal quality comprises at least one of: RSRP, SINR, RSSI and RSRQ.
The method according to any one of claims 1 to 12, wherein the first information comprises at least one of: first data for testing the first model; or a first signal for testing the first model.
A model test method, comprising: using a first model, performing prediction based on first information and obtaining predicted data; wherein the first information is information sent from a network device to a terminal device; and sending second information, the second information comprising the predicted data, the predicted data being configured to determine whether the first model meets a first condition, the first condition being a performance requirement of the predicted data.
The method according to claim 12, further comprising: receiving the first information; wherein using the first model, performing prediction based on the first information and obtaining the predicted data comprises: measuring the first information, and obtaining first measurement data; and inputting the first measurement data into the first model and obtaining the predicted data output by the first model.
The method according to claim 12 or 13, further comprising: receiving third information; the third information being configured to indicate whether the first model is usable by the terminal device; and determining whether the first model is usable based on the third information.
The method according to any one of claims 12 to 14, further comprising: sending fourth information; wherein the fourth information is configured to trigger the network device to send the first information, and the fourth information comprises a first model identifier of the first model.
The method according to claim 15, wherein the fourth information comprises at least one of: indication information for model update; indication information for model addition; or indication information for model test.
The method according to any one of claims 12 to 16, wherein the first model is a model for performing beam prediction, the predicted data comprises a predicted signal quality of a first beam, and the first beam is at least one beam configured by the network device for the terminal device; the first condition comprises a difference between the predicted signal quality and a first signal quality being within a first difference range; wherein the first signal quality is a signal quality of the first beam generated by the network device according to beam configuration information, or the first signal quality is a signal quality of the first beam actually measured by the terminal device.
The method according to any one of claims 12 to 16, wherein the first model is a model for performing beam prediction, the predicted data comprises a second beam that meets the signal condition and is obtained from the beam prediction; the signal condition comprises at least one of: a signal quality of the second beam is greater than or equal to a preset signal quality threshold; or the second beam is a beam with the best signal quality among a plurality of beams configured by the network device for the terminal device.
The method according to claim 18, wherein the first condition is that a number of times a prediction success condition is met in N beam predictions is greater than or equal to M; wherein N is a natural number and M is a natural number less than or equal to N; the prediction success condition is that the second beam and a third beam are the same, and the second beam is a beam that meets the signal condition and is obtained from the beam prediction; the third beam is a beam that meets the signal condition and is determined by the network device according to beam configuration information; or the third beam is a beam that meets the signal condition and is determined according to second measurement data, and the second measurement data is data obtained by the terminal device after the terminal device measures multiple beams configured by the network device for the terminal device.
The method according to any one of claims 17 to 19, wherein the signal quality comprises at least one of: RSRP, SINR, RSSI and RSRQ.

Description

TECHNICAL FIELD The present disclosure relates to the field of communication technology, and in particular to a model test method, a model test device and a storage medium. BACKGROUND With the advancement of communication technology, predictive models have been introduced into communication systems, such as Artificial Intelligence (AI) models, Machine Learning (ML) models, or other models. Predictive models can obtain predicted data for certain scenarios to improve network performance. However, the reliability of the predicted data significantly impacts network performance; therefore, how to determine the reliability of the predicted data from predictive models is a pressing issue that needs to be addressed. SUMMARY The present disclosure provides a model test method, a model test device, and storage medium. According to a first aspect of the embodiments of the present disclosure, there is provided a model test method, including: determining whether predicted data output by a first model meets a first condition; where the first condition is a performance requirement of the predicted data, the predicted data is data predicted by the first model based on first measurement data, the first measurement data is data obtained after a terminal device measures first information, and the first information is information sent from a network device to the terminal device. According to a second aspect of the embodiments of the present disclosure, there is provided a model test method, including: using a first model, performing prediction based on first information and obtaining predicted data; where the first information is information sent from a network device to a terminal device; andsending second information, the second information including the predicted data, the predicted data being configured to determine whether the first model meets a first condition, the first condition being a performance requirement of the predicted data. According to a third aspect of the embodiments of the present disclosure, there is provided a model test method, including: a terminal device using a first model, performing prediction based on first information and obtaining predicted data; where the first information is information sent from a network device to the terminal device;the terminal device sending second information to the network device, the second information including the predicted data, the predicted data being configured to determine whether the first model meets a first condition, the first condition being a performance requirement of the predicted data; andthe network device determining whether the predicted data output by the first model meets the first condition; where the predicted data is data obtained by the first model based on first measurement data, the first measurement data is data obtained after the terminal device measures the first information, and the first information is information sent from the network device to the terminal device. According to a fourth aspect of the embodiments of the present disclosure, there is provided a network device, including: a processing module configured to determine whether predicted data output by a first model meets a first condition; where the first condition is a performance requirement of the predicted data, the predicted data is data predicted by the first model based on first measurement data, the first measurement data is data obtained after a terminal device measures first information, and the first information is information sent by the network device to the terminal device. According to a fifth aspect of the embodiments of the present disclosure, there is provided a terminal device, including: a processing module configured to use a first model, perform prediction based on first information and obtain predicted data; where the first information is information sent by a network device to the terminal device; anda transceiver module configured to send second information, where the second information includes the predicted data, the predicted data is configured to determine whether the first model meets a first condition, the first condition is a performance requirement of the predicted data. According to a sixth aspect of the embodiments of the present disclosure, there is provided a network device, including one or more processors; where the network device is configured to execute the optional implementations according to the first aspect. According to a seventh aspect of the embodiments of the present disclosure, there is provided a terminal device, including one or more processors; where the terminal device is configured to execute the optional implementations according to the second aspect. According to an eighth aspect of the embodiments of the present disclosure, there is provided a communication system, including a terminal device and a network device, where the network device is configured to implement the method as described in the optional implementation