Search

CN-122001937-A - Multi-dimensional weight scheduling data center service system and method

CN122001937ACN 122001937 ACN122001937 ACN 122001937ACN-122001937-A

Abstract

A multi-dimension weight dispatching data center service system and method, wherein the system comprises a gateway module, an API service processing module, an asynchronous dispatching and callback module, a rights management module and a data storage module; the method comprises the steps of first data primary processing, second data preprocessing, third rights verification and pre-deduction, fourth asynchronous task scheduling, fifth task callback and sixth data storage, wherein the method not only ensures the quick response of high-weight requests, but also ensures that low-weight requests are not shelved for a long time through low-level proportion reservation by combining a multi-dimensional callback weight calculation model of at least user weight and request weight and dynamic level pulling proportion, thereby effectively solving the contradiction of 'high-weight starvation low-weight' or 'low-weight slow high-weight' in the traditional scheduling scheme and realizing the good balance between efficiency and fairness of asynchronous task scheduling.

Inventors

  • MAO FANGYI
  • ZHOU TAO
  • XU JIAN

Assignees

  • 浙江亿保软件有限公司

Dates

Publication Date
20260508
Application Date
20260316

Claims (10)

  1. 1. The data center service system for multi-dimensional weight scheduling is characterized by comprising a gateway module, a rights management module, an API service processing module, an asynchronous scheduling and callback module and a data storage module; The gateway module is used for receiving an external request, performing flow control and security verification on the external request, packaging information of the request after the operation, and forwarding the packaged request to the API service processing module; The API service processing module is used for receiving the request from the gateway module, calling the rights management module to perform user authentication verification and rights pre-deduction after the rule verification is performed on the request, performing service processing on the request data after the rights pre-deduction is successful, and sending the request information needing asynchronous callback together with callback weight obtained by calculation according to the user weight and the request weight to a callback pool to be called of the data storage module; The asynchronous scheduling and callback module is used for regularly pulling the tasks to be callback from the pool to be callback according to the preset weight level proportion, distributing different processing resources for the tasks of different levels according to the callback weight, and distributing the pulled tasks to the API service processing module for callback execution; The rights management module is used for responding to the call of the API service processing module, executing user authentication, performing an atomic pre-deduction operation on the rights of the user in a Redis cache, monitoring service processing result information in an information queue, and if monitoring the information of failure of service processing, recovering and compensating the rights of the corresponding user in the Redis cache; The data storage module comprises a Redis cache for storing equity pre-deduction state and user balance thermal data, a MySQL database for storing request events and callback task persistence data, a message queue MQ for transmitting service processing results and compensation instructions, and a database table serving as the callback waiting pool.
  2. 2. A multi-dimensional weight-scheduled data center service method applied to the system as claimed in any one of claims 1 to 4, comprising the steps of: Firstly, receiving an external request through a gateway module, and forwarding the external request to an API service processing module after current limiting, safety verification and information encapsulation; Secondly, calling a rights management module through an API service processing module to carry out rule verification, then carrying out user authentication verification and rights pre-deduction, if the rights are insufficient or the authentication fails, directly returning a response, carrying out rights pre-deduction after the authentication is successful, and carrying out subsequent service processing; Thirdly, for a service request requiring asynchronous callback, calculating callback weight of the service request through an API service processing module according to user weight and request weight of the request, and storing task information of the request and the callback weight into a to-be-called-back pool; Pulling a batch of tasks to be called back from the pool to be called back at regular time through the asynchronous scheduling and callback module according to the preset weight level proportion; fifthly, distributing the pulled task to an API service processing module for instance processing, and executing callback to the third party service; And sixthly, the rights management module monitors the message queue, and if the service processing failure message associated with the rights deduction in the step two is monitored, the rights of the corresponding user in the Redis cache is recovered.
  3. 3. The multi-dimensional weight-scheduled data center service system according to claim 2, wherein in the first step, the external request received by the gateway module needs to be processed in advance as follows: 1.1 SM4 encrypting the request to obtain an encrypted request body; 1.2 Screening, sorting and splicing original public request heads carried by the request provider, and then combining the original public request heads with an encryption request body; 1.3 And then carrying out signature calculation by using an SM3 algorithm and combining an application key to generate a request carrying a complete public request header, an SM3 signature and an SM4 encryption request body, and using the request as an external request.
  4. 4. A multi-dimensional weight-scheduled data center service system according to claim 2 or 3, wherein in the second step, the rule content includes: 2.1 Header parameter verification) 2.1.1 Checking whether the parameters of the request header meet the field types; 2.1.2 Checking whether the request head part must be filled with the content or not; 2.2 Signature verification) Comparing the signature with the signature calculated according to the request head SM3 according to the X-IPUT-Sign in the request head; 2.3 Idempotent of request Checking when in request, intercepting repeatedly, and ensuring that the same Nonce only executes business logic once within preset time; 2.4 Anti-replay verification) Relying on X-IPUT-nonces and time stamps ensures that requests are both disposable and time-efficient.
  5. 5. The method for serving a data center for multi-dimensional weight scheduling according to claim 4, wherein the specific contents in the fifth step are as follows: 5.1 Using XxlJob timing tasks to obtain task data needing asynchronous scheduling; 5.2 Performing different business processes according to the event processing type of the task data; 5.3 After the processing is completed, the service data is assembled, and the interface is called according to the X-IPUT-Callback-Url in the previous request head; 5.4 Pushing the final business processing result to the client.
  6. 6. The method according to claim 2 or 5, wherein in the third step, the formula for calculating the callback weight is callback weight=user weight×a+request weight×b, wherein a and B are preset weighting coefficients, and a+b=1, a > B.
  7. 7. The method for serving a data center of multi-dimensional weight scheduling according to claim 6, wherein in the fourth step, the specific content of the task to be called back is pulled according to a preset weight level ratio through an asynchronous scheduling and callback module, which is as follows: 4.1 Dividing the callback weight into three weight levels of high, medium and low according to the maximum range of the callback weight; 4.2 Setting the total number N of single pulling tasks; 4.3 The method comprises the steps of) carrying out pulling according to the proportion of high-level task pulling quantity=N×P1, middle-level task pulling quantity=N×P2 and low-level task pulling quantity=N×P3, wherein P1, P2 and P3 are preset level pulling proportions, P1> P2> P3 and P1+P2+P3=1.
  8. 8. The method according to claim 2 or 7, wherein in the fourth step, when pulling tasks, the asynchronous scheduling and callback module is further capable of dynamically adjusting the execution frequency of the timed task for fetching tasks from the to-be-callback Chi La according to the serial or parallel feature of the target callback service support, and distributing the callback tasks to different API service processing instances by adopting a combination of polling routing and global lock.
  9. 9. The method for multi-dimension weighted scheduling of data center service according to claim 8, wherein in the fourth step, the time interval of the pulling task is dynamically adjusted according to the average time consumption of the processing request of the target callback service, and if the target service only supports serial processing, the time interval of the pulling task is set to be not less than the average time consumption.
  10. 10. The method for serving a multi-dimensional weight-scheduled data center according to claim 2 or 9, wherein in the second step, the equity pre-deduction is completed in the Redis cache through an atomic operation; In the step six, the rights and interests restoring operation is also completed in the Redis cache, and the relevant task state records in the MySQL database are synchronously updated.

Description

Multi-dimensional weight scheduling data center service system and method Technical Field The invention relates to the field of data processing, in particular to a data center service system and a method for multi-dimensional weight scheduling. Background In the enterprise digital transformation process, the data center station is used as a core carrier for integrating multi-source service and processing massive service requests, and the multi-dimensional requirements are simultaneously met, so that the cooperation of a gateway, rights management, API service and other multi-modules is realized, the efficient dispatching of asynchronous requests is ensured, and the rights data consistency and the interface call security are ensured. Along with the increase of the service scale, the enterprise-level data center generally faces challenges such as the increase of service request magnitude, the diversification of service types, the upgrading of safety protection requirements and the like, and higher requirements are put forward on the stability, the high efficiency, the safety and the expandability of a system architecture. In terms of asynchronous request scheduling, a common scheme is to adopt a single weight ordering mechanism or a first-in first-out (FIFO) mechanism, the former is used for carrying out inverse priority processing on high-weight requests according to request weight values, the latter is sequentially processed according to request arrival time sequences, the two mechanisms have a certain effect under a simple scene, but no effective balance is realized between high-weight priority processing and low-weight requests which are not placed for a long time, in terms of rights management, most architectures separate rights verification from service call, and carry out rights deduction operation directly through a MySQL (request for interest) relational database, the scheme does not introduce a buffer pre-deduction and message queue compensation mechanism, the problem of data inconsistency easily occurs under the abnormal scene of high concurrency or service call, in terms of interface security, the common scheme depends on a single signature algorithm (such as an MD 5) or a basic symmetric encryption algorithm (such as AES), full link verification mechanism for request uniqueness, timeliness and completeness is not realized, a full link verification mechanism is difficult to realize effective parameter, repetition, a callback mechanism is not used for the time-out, the callback mechanism is used for the callback frequency is not matched with a callback mechanism, the callback (request) can be carried out in terms of the method, and the callback policy is not used for carrying out the callback-back-off, the request can be stably transferred in terms of the request, and the callback policy is not matched with a callback-side, and the callback policy is not used for transferring the request has a regular policy, and the request can be well-scheduled, and the request is well has a policy is well-scheduled, and is well-scheduled is not scheduled, and is well has a policy is well-ordered, and is easy, core information such as application ID is easy to miss or error in the transmission process, and accuracy and efficiency of cooperation between services are affected. Based on the prior art scheme, when the high concurrency, high safety and high consistency business requirements of the enterprise-level data center are met, the defects that scheduling fairness is lost, a single weight ordering mechanism easily causes that a low weight request cannot be processed for a long time, a starvation problem occurs, a first-in first-out mechanism cannot guarantee the response speed of a high priority request, and the two cannot realize effective balance between scheduling efficiency and fairness of request processing; the method has the advantages of poor consistency of rights data, direct operation of a database for carrying out rights deduction, lack of a cache pre-deduction and abnormal compensation mechanism, easiness in occurrence of inconsistent data of 'rights deduction but incomplete service' when service call overtime or fails, influence on service reliability and user experience, insufficient interface safety protection, easiness in falsification of parameters, incapability of effectively identifying and intercepting repeated requests and overtime requests, lower interface call safety, low callback efficiency, incapability of dynamically distributing callback resources according to request weights by fixed frequency polling and random routing strategies, delay callback of high weight requests due to insufficient resource allocation, influence on service response speed and overall throughput of the system, low service cooperation efficiency, lack of standardized request encapsulation and core information transmission mechanism between a gateway and a back-end service, easiness in losing or being wrong in a