US-12621223-B2 - Industrial 5G service quality assurance via Markov decision process mapping

US12621223B2US 12621223 B2US12621223 B2US 12621223B2US-12621223-B2

Abstract

A quality assurance method, performed in a device, includes defining a set of states of a joint system of a communication service provider and a consumer of the service, defining a set of actions each corresponding to a change of one or more QoS characteristics of the service, using a partially observable Markov decision process (POMDP) to define a control policy mapping beliefs about the state of the joint system to preferred actions taken on the service, receiving observations indicative of performance indicators of a process performed by the consumer, updating a belief about the state of the joint system based on the observations, and using the control policy to control the service to make the service operate more efficiently while fulfilling a service level agreement (SLA) with the consumer. A device, joint system, computer program and computer program product are also provided.

Inventors

Ajay KATTEPUR
Anil RAMACHANDRAN NAIR
Merve SAIMLER
Yunus Donmez

Assignees

TELEFONAKTIEBOLAGET LM ERICSSON (PUBL)

Dates

Publication Date: 20260505
Application Date: 20220629
Priority Date: 20210814

Claims (20)

1 . A method of quality assurance performed in a computing device associated with a communication service provider, the method comprising: defining a set of states of a joint system comprising a communication service provided by the communication service provider and a process performed by a consumer while consuming the communication service, each state of the set of states being a hidden state of the joint system, and each such hidden state corresponding to a degree of fulfillment of a service level agreement, SLA, with the consumer of the communication service and a degree of efficiency of the communication service; defining a set of actions, each action of the set of actions corresponding to a changing of one or more Quality of Service, QoS, characteristics of the communication service; using a partially observable Markov decision process, POMDP, to define a control policy mapping beliefs about the state of the joint system to preferred actions in order to at least eventually move the joint system towards a state corresponding to one or both of a higher degree of SLA-fulfillment and a higher degree of efficiency; receiving observations indicative of one or more performance indicators for the process performed by the consumer; based on the received observations, establishing or updating a belief about a current state of the joint system; and based on the belief about the current state of the joint system, controlling the communication service by performing one or more actions of the set of actions as suggested by the control policy.
2 . The method according to claim 1 , wherein the one or more QoS characteristics are selected from: packet delay, packet loss, priority, and guaranteed/non-guaranteed bit rate.
3 . The method according to claim 1 , wherein the process performed by the consumer is a manufacturing process, and wherein the one or more performance indicators are selected from: throughput rate, availability/capacity, efficiency, energy consumption, cycle time, downtime, and machine production capability.
4 . The method according to claim 1 , wherein defining the control policy includes using a point-based POMDP algorithm.
5 . The method according to claim 1 , wherein the set of states of the joint system includes at least: a first state corresponding to the SLA not being fulfilled and to the communication service not being efficient; a second state corresponding to the SLA being fulfilled and to the communication service not being efficient; a third state corresponding to the SLA not being fulfilled and to the communication service being efficient, and a fourth state corresponding to the SLA being fulfilled and to the communication service being efficient.
6 . The method according to claim 1 , wherein the communication service is a fifth-generation, 5G, mobile communications service.
7 . The method according to claim 6 , wherein the method includes one or both of receiving the observations and controlling the communication service using one or more digital twin interfaces.
8 . The method according to claim 1 , wherein the process performed by the consumer, and the performance indicators for the process, are as defined in any one of ISO 22400, IEC 62264, ISO 22737, and IEEE P2048.
9 . The method according to claim 1 , wherein the method further includes implementing a reward structure of the POMDP by assigning positive or higher rewards for states having a higher degree of SLA-fulfillment and a higher degree of efficiency, and by assigning negative or lower rewards for states having one or both of a lower degree of SLA-fulfillment and a lower degree of efficiency.
10 . A computing device for quality assurance, comprising processing circuitry and a memory storing instructions, the instructions are such that they, when executed by the processing circuitry, cause the computing device to: define a set of states of a joint system comprising a communication service provided by the communication service provider and a process performed by a consumer while consuming the communication service, each state of the set of states being a hidden state of the joint system, and each such hidden state corresponding to a degree of fulfillment of a service level agreement, SLA, with the consumer of the communication service and a degree of efficiency of the communication service; define a set of actions, each action of the set of actions corresponding to a changing of one or more Quality of Service, QoS, characteristics of the communication service; use a partially observable Markov decision process, POMDP, to define a control policy mapping beliefs about the state of the joint system to preferred actions in order to at least eventually move the joint system towards a state corresponding to one or both of a higher degree of SLA-fulfillment and a higher degree of efficiency; receive observations indicative of one or more performance indicators for the process performed by the consumer; based on the received observations, establish or update a belief about a current state of the joint system; and based on the belief about the current state of the joint system, control the communication service by performing one or more actions of the set of actions as suggested by the control policy.
11 . The computing device according to claim 10 , wherein the one or more QoS characteristics are selected from: packet delay, packet loss, priority, and guaranteed/non-guaranteed bit rate.
12 . The computing device according to claim 10 , wherein the process performed by the consumer is a manufacturing process, and wherein the one or more performance indicators are selected from: throughput rate, availability/capacity, efficiency, energy consumption, cycle time, downtime, and machine production capability.
13 . The computing device according to claim 10 , wherein defining the control policy includes using a point-based POMDP algorithm.
14 . The computing device according to claim 10 , wherein the set of states of the joint system includes at least: a first state corresponding to the SLA not being fulfilled and to the communication service not being efficient; a second state corresponding to the SLA being fulfilled and to the communication service not being efficient; a third state corresponding to the SLA not being fulfilled and to the communication service being efficient, and a fourth state corresponding to the SLA being fulfilled and to the communication service being efficient.
15 . The computing device according to claim 10 , wherein the communication service is a fifth-generation, 5G, mobile communications service.
16 . The computing device according to claim 15 , wherein the instructions are further such that they, when executed by the processing circuitry, cause the computing device to perform one or both of receiving the observations and controlling the communication service using one or more digital twin interfaces.
17 . A joint system comprising a first system configured to provide a communication service and a second system configured to consume the communication service, and further comprising at least one computing device, the computing device being communicatively coupled to both the first system and the second system, and configured to receive the one or more performance indicators from the second system to control the communication service provided by the first system, the at least one computing device comprising: processing circuitry and a memory storing instructions, the instructions are such that they, when executed by the processing circuitry, cause the computing device to: define a set of states of a joint system comprising a communication service provided by the communication service provider and a process performed by a consumer while consuming the communication service, each state of the set of states being a hidden state of the joint system, and each such hidden state corresponding to a degree of fulfillment of a service level agreement, SLA, with the consumer of the communication service and a degree of efficiency of the communication service; define a set of actions, each action of the set of actions corresponding to a changing of one or more Quality of Service, QoS, characteristics of the communication service; use a partially observable Markov decision process, POMDP, to define a control policy mapping beliefs about the state of the joint system to preferred actions in order to at least eventually move the joint system towards a state corresponding to one or both of a higher degree of SLA-fulfillment and a higher degree of efficiency; receive observations indicative of one or more performance indicators for the process performed by the consumer; based on the received observations, establish or update a belief about a current state of the joint system; and based on the belief about the current state of the joint system, control the communication service by performing one or more actions of the set of actions as suggested by the control policy.
18 . The joint system according to claim 17 , wherein the second system forms part of a Fourth-industrial revolution, Industry 4.0, solution.
19 . A non-transitory computer storage medium storing a computer program for quality assurance, the computer program comprising computer code which, when run on a processing circuitry of a computing device, causes the computing device to: define a set of states of a joint system comprising a communication service provided by the communication service provider and a process performed by a consumer while consuming the communication service, each state of the set of states being a hidden state of the joint system, and each such hidden state corresponding to a degree of fulfillment of a service level agreement, SLA, with the consumer of the communication service and a degree of efficiency of the communication service; define a set of actions, each action of the set of actions corresponding to a changing of one or more Quality of Service, QoS, characteristics of the communication service; use a partially observable Markov decision process, POMDP, to define a control policy mapping beliefs about the state of the joint system to preferred actions in order to at least eventually move the joint system towards a state corresponding to one or both of a higher degree of SLA-fulfillment and a higher degree of efficiency; receive observations indicative of one or more performance indicators for the process performed by the consumer; based on the received observations, establish or update a belief about a current state of the joint system; and based on the belief about the current state of the joint system, control the communication service by performing one or more actions of the set of actions as suggested by the control policy.
20 . The non-transitory computer program storage medium according to claim 19 , wherein the one or more QoS characteristics are selected from: packet delay, packet loss, priority, and guaranteed/non-guaranteed bit rate.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application is a Submission Under 35 U.S.C. § 371 for U.S. National Stage Patent Application of International Application No.: PCT/TR2022/050677, filed Jun. 29, 2022 entitled “INDUSTRIAL 5G SERVICE QUALITY ASSURANCE VIA MARKOV DECISION PROCESS MAPPING,” which claims priority to Indian Application No.: 202111036877, filed Aug. 14, 2021,” the entireties of both of which are incorporated herein by reference. TECHNICAL FIELD The present disclosure relates to communication networks when used to provide communication services to consumers. In particular, the present disclosure relates to how a computing device associated with such networks can use partially observable Markov decision processes in order to increase network robustness and reliability. BACKGROUND Modern and future industrial deployments are likely to become increasingly complex, especially if implementing changes conceptualized by the so-called Fourth industrial revolution (or “Industry 4.0”). As an example, so-called smart manufacturing may involve a linking of multiple subsystems such as machinery, robots, internet-of-things (IoT), various machine learning algorithms, and current and next-generation mobile telecommunications systems. With such an increased complexity, the critical role of quality assurance (QA) becomes more difficult. In order to provide a unified view of quality assurance across multiple use-cases in Industry 4.0-related processes, accurate parametrization, mapping, data analysis and closed loop controls are needed. To ensure optimal performance, Industry 4.0-related applications require robust and reliable connectivity, and the emergence of Fifth-generation (5G) private networks attempts to provide differentiated services for various industrial traffic requirements. To guarantee service quality, it is imperative to correctly map how various metrics of the Industry 4.0 domain are affected by changes made in the 5G network domain, as such mapping is needed to ensure optimal system performance without the need for e.g. 5G network over-provisioning or requirement violations. To assess the performance of e.g. Industry 4.0-related manufacturing processes, various key performance indices (KPIs) are typically defined in ISO/IEC formats. Examples of such KPIs include production rates, failure rates, and machine stoppage times. These KPIs are then acted upon by a network service quality assurance layer in order to ensure that the 5G network meets these requirements. The 5G networks are in turn evaluated in terms of other metrics such as packet delay, packet drop rates, packet throughput, and e.g. network availability. In accordance with commonly available technology, understanding how the metrics of the 5G networks relate to (and affect) the KPIs of manufacturing processes is not an easy task, and often requires both analysis and intervention by experts trained in the field (i.e. manual tuning of one or more parameters of the 5G network, and often in a trial-and-error-like process). Consequently, how to configure a 5G network in terms of optimal resource allocation, usage efficiency and reliability while still meeting the demands for service quality posed by a consumer (such as e.g. a smart manufacturing plant) is a difficult task. As a result, 5G networks often end up being over-dimensioned and/or operating at a sub-optimal level in order to not risk accidentally failing to deliver in accordance with an agreement established with the consumer. SUMMARY In light of the above, there is therefore a need for an improved way of configuring a communication service (such as a 5G network service) in order to meet a particular service level agreement (SLA) with a consumer of the communication service. For this purpose, the present disclosure provides an improved method of quality assurance performed in a computing device associated with a communication service provider, and a corresponding computing device, joint system, computer program and computer program product as defined by the accompanying independent claims. Various embodiments of the improved method, computing device, joint system, computer program and computer program product are defined by the accompanying dependent claims. According to a first aspect, there is provided a method of quality assurance performed in a computing device associated with a communication service provider. The method includes defining a set of states (S) of a joint system comprising a communication service provided by the communication service provider and a process performed by a consumer while consuming the communication service. In the envisaged method, each state (s∈S) is a hidden state of the joint system, and each such hidden state corresponds to a degree of fulfillment of a service level agreement (SLA) with the consumer of the communication service and a degree of efficiency of the communication service. The method further includes defining a set of actions (A)