Search

CN-121998740-A - Big data intelligent recommendation method and system based on multi-scene dynamic adaptation

CN121998740ACN 121998740 ACN121998740 ACN 121998740ACN-121998740-A

Abstract

The application discloses a big data intelligent recommendation method and system based on multi-scene dynamic adaptation. The method comprises the steps of constructing a three-dimensional scene feature space, fusing multi-source heterogeneous data to generate a dynamic user portrait, realizing scene-feature weight self-adaptive adjustment based on reinforcement learning, combining a 'sharing-private' neural network and transfer learning to train a multi-scene model, finally dynamically distributing resources according to scene priority to generate a recommended result, and continuously optimizing through feedback closed loop. The method solves the problems of insufficient scene adaptation, poor real-time performance and the like of the traditional recommendation system, remarkably improves the recommendation precision and the resource utilization rate, and is suitable for multiple fields of electronic commerce, content distribution and the like.

Inventors

  • ZHANG ZHONGHUA
  • HUANG AIHUA
  • YIN JUEHUI

Assignees

  • 上海趣致网络科技有限公司

Dates

Publication Date
20260508
Application Date
20260128

Claims (10)

  1. 1. The intelligent big data recommendation method based on multi-scene dynamic adaptation is characterized by comprising the following steps: S1, constructing a three-dimensional scene feature space based on user attributes, environment parameters and a behavior sequence, and generating a scene feature vector, wherein the scene feature vector at least comprises a user ID, a scene ID, a time feature, a place feature, a device feature, a behavior activity level and a scene priority; s2, fusing the offline historical data and the real-time scene data by adopting a streaming batch integrated architecture to generate a unified user portrait containing static portraits and dynamic portraits; S3, taking the scene feature vector output by the S1 and the unified user portrait output by the S2 as inputs, obtaining scene-feature weight vectors through on-line training of the deep reinforcement learning DQN, and smoothly updating the scene-feature weight vectors through an EMA algorithm when the scene is switched, so as to realize direct mapping of scene demand change and weight adjustment; S4, constructing a shared-private double-layer neural network, and migrating shared layer parameters of the big data scene to a private layer of the small data scene by using knowledge distillation to obtain a scene exclusive recommendation model; S5, dynamically distributing GPU/CPU computing resources according to scene priority fields of scene feature vectors, performing scene filtering, fine arrangement and diversity rearrangement on candidate commodities, and outputting a final recommendation list; And S6, collecting feedback behaviors of the user on the recommendation list in real time, and triggering incremental updating and rollback of the standby model to realize closed loop iteration if the scene level click rate in a preset time interval is lower than a threshold value.
  2. 2. The big data intelligent recommendation method according to claim 1, wherein the behavior sequence dimension in S1 comprises a behavior type, a behavior activity degree and a behavior sequence entropy value, and the behavior activity degree is obtained by summing operation frequency multiplied by behavior weight in unit time.
  3. 3. The big data intelligent recommendation method according to claim 1, wherein the dynamic representation in S2 encodes a near 10 minute behavior sequence by LSTM algorithm to generate a short-term demand vector with a fixed dimension; The short-term demand vector is spliced with the age, sex and long-term preference vector of the commonly purchased goods class in the static portrait according to the user ID main key in the memory snapshot to form a unified user portrait.
  4. 4. The big data intelligent recommendation method according to claim 1, wherein the reward function R = α x click rate + β x conversion rate + γ x diversity score of DQN in S3; wherein alpha, beta and gamma are dynamic coefficients and the sum is 1, and the system automatically fine-adjusts the weight of the three coefficients every 10 minutes according to the scene-level business target.
  5. 5. The big data intelligent recommendation method according to claim 1, wherein the shared-private dual-layer network in S4 comprises a ReLU-activated 3-layer fully connected shared layer and a scene-specific GRU private layer; After the shared layer pretrains the scene with large data volume, parameters are migrated to a new scene private layer with sparse data through knowledge distillation temperature coefficients, so that cross-scene parameter migration is realized.
  6. 6. The intelligent big data recommendation method according to claim 1, wherein the scene priority field in S5 dynamically routes a high priority scene request to the GPU with a target response delay less than or equal to 100ms, routes a low priority scene request to the CPU with a target response delay less than or equal to 300ms, and the resource scheduler automatically increases or decreases GPU/CPU instances per minute according to QPS fluctuations.
  7. 7. The intelligent big data recommendation method according to claim 1, wherein the preset time interval in S6 is 3 continuous 10 minute windows, when the scene-level click rate is lower than a threshold value, the system pulls the feedback data of the last 1 hour, performs incremental fine adjustment on the special GRU private layer of the scene in S4, and automatically switches to a backup model of the previous day, so that the recommendation effect is ensured to be restored to be stable within 30 minutes.
  8. 8. Big data intelligent recommendation system based on multi-scene dynamic adaptation, characterized in that the system comprises: The system comprises a construction module, a behavior sequence generation module and a behavior sequence generation module, wherein the construction module is used for constructing a three-dimensional scene feature space based on user attributes, environment parameters and the behavior sequence to generate scene feature vectors, and the scene feature vectors at least comprise user IDs, scene IDs, time features, place features, equipment features, behavior liveness and scene priorities; the fusion module is used for fusing the offline historical data and the real-time scene data by adopting a streaming batch integrated architecture to generate a unified user portrait containing static portraits and dynamic portraits; the training module is used for obtaining a scene-feature weight vector W through on-line training of the depth reinforcement learning DQN by taking the output scene feature vector and the output unified user portrait as inputs, and smoothly updating the W through an EMA algorithm when the scene is switched, so as to realize direct mapping of scene demand change and weight adjustment; the migration module is used for constructing a shared-private double-layer neural network, and migrating shared layer parameters of a big data scene to a private layer of a small data scene by using knowledge distillation to obtain a scene exclusive recommendation model; The distribution module is used for dynamically distributing GPU/CPU computing resources according to scene priority fields of scene feature vectors, performing scene filtering, fine arrangement and diversity rearrangement on candidate commodities, and outputting a final recommendation list; and the iteration module is used for collecting feedback behaviors of the user on the recommendation list in real time, and triggering increment updating and rollback of the standby model to realize closed loop iteration if the scene level click rate in the preset time interval is lower than the threshold value.
  9. 9. An electronic device comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, implements the intelligent big data recommendation method based on multi-scenario dynamic adaptation according to any of claims 1 to 7.
  10. 10. A computer readable storage medium, characterized in that it has stored thereon a computer program, which when executed by a processor, implements a multi-scenario dynamic adaptation based big data intelligent recommendation method according to any of claims 1 to 7.

Description

Big data intelligent recommendation method and system based on multi-scene dynamic adaptation Technical Field The application relates to the technical field of big data processing, reinforcement learning and intelligent recommendation systems, in particular to a big data intelligent recommendation method and system based on multi-scene dynamic adaptation. Background Along with the rapid development of mobile internet and big data technology, the recommendation system is widely applied to multiple scenes such as electronic commerce, content distribution, online education, local life service and the like. However, existing recommendation systems have significant technical bottlenecks in multi-scenario adaptation: The scene modeling is static, namely a dynamic scene feature system is not constructed depending on a fixed feature space (such as a user history behavior), and the demand difference of users in space-time heterogeneous scenes (such as workday commute/weekend home) cannot be distinguished, so that a recommendation strategy is disjointed with scene demands; The data fusion granularity is insufficient, namely, fusion barriers exist between offline data (historical behaviors) and real-time data (current environment and operation sequence), dynamic user portraits of 'long-term preference-short-term requirements' are not formed, and recommendation accuracy is limited due to single data dimension; The real-time response mechanism is lacking, wherein the updating period of the model parameters is at an hour level or a day level, the change of the requirements caused by scene switching (such as switching from 4G commute to WiFi office) of a user cannot be captured in real time, and the response delay exceeds 500ms; The cross-scene migration capability is weak, namely, the recommendation accuracy rate is reduced by more than 40% in a data sparse scene (such as a new class recommendation and a holiday exclusive scene) by adopting an independent scene model training mode and not utilizing the characteristic relevance among scenes; resource scheduling non-suitability, namely, a unified computing resource allocation strategy is adopted, and a high-value scene (such as a transaction conversion scene) and a low-value scene (such as a browsing scene) share computational power, so that the core scene recommendation delay rate exceeds 15%. Disclosure of Invention Based on the above, the embodiment of the application provides a big data intelligent recommendation method and system based on multi-scene dynamic adaptation, which aim to solve the technical problems of scene dynamic modeling deficiency, low efficiency of multi-source data fusion, real-time response lag, poor inter-scene mobility, unreasonable resource scheduling and the like in the existing recommendation system. In a first aspect, a big data intelligent recommendation method based on multi-scene dynamic adaptation is provided, the method includes: S1, constructing a three-dimensional scene feature space based on user attributes, environment parameters and a behavior sequence, and generating a scene feature vector, wherein the scene feature vector at least comprises a user ID, a scene ID, a time feature, a place feature, a device feature, a behavior activity level and a scene priority; s2, fusing the offline historical data and the real-time scene data by adopting a streaming batch integrated architecture to generate a unified user portrait containing static portraits and dynamic portraits; S3, taking the scene feature vector output by the S1 and the unified user portrait output by the S2 as inputs, obtaining scene-feature weight vectors through on-line training of the deep reinforcement learning DQN, and smoothly updating the scene-feature weight vectors through an EMA algorithm when the scene is switched, so as to realize direct mapping of scene demand change and weight adjustment; S4, constructing a shared-private double-layer neural network, and migrating shared layer parameters of the big data scene to a private layer of the small data scene by using knowledge distillation to obtain a scene exclusive recommendation model; S5, dynamically distributing GPU/CPU computing resources according to scene priority fields of scene feature vectors, performing scene filtering, fine arrangement and diversity rearrangement on candidate commodities, and outputting a final recommendation list; And S6, collecting feedback behaviors of the user on the recommendation list in real time, and triggering incremental updating and rollback of the standby model to realize closed loop iteration if the scene level click rate in a preset time interval is lower than a threshold value. Optionally, the behavior sequence dimension in S1 comprises a behavior type, behavior liveness and a behavior sequence entropy value, wherein the behavior liveness is obtained by summing operation frequency multiplied by behavior weight in unit time. Optionally, the dynamic image in S2 encodes a near