Search

CN-121998019-A - AI marking closed-loop method based on active learning and difficult case mining

CN121998019ACN 121998019 ACN121998019 ACN 121998019ACN-121998019-A

Abstract

The invention discloses an AI marking closed-loop method based on active learning and difficult case mining, which comprises the steps of after an initial model is trained, calculating uncertainty, representativeness and diversity three-dimensional value indexes in parallel, dynamically weighting and screening high-value samples through a closed-loop feedback controller, identifying difficult cases, adopting DBSCAN clusters, searching similar samples for expansion, storing the similar samples in a dynamic pool, distributing labels, triggering intelligent judgment on the disputed samples after consistency verification, fusing node reputation, model confidence and feature similarity to determine labels, carrying out incremental training by adopting a mixed loss function, carrying out linkage adjustment on sampling, difficult cases and training parameters after performance is not up to standard, carrying out retry, evaluating the validity of the difficult cases after each round of iteration, eliminating the invalid cases, merging redundant clusters, self-adaptively maintaining pool capacity, and monitoring model performance iterative optimization until the performance is up to standard. The method reduces the labeling cost and improves the recognition capability of boundary samples and long tail categories.

Inventors

  • WANG BINBIN

Assignees

  • 无线生活(北京)信息技术有限公司

Dates

Publication Date
20260508
Application Date
20260129

Claims (10)

  1. 1. An AI marking closed loop method based on active learning and difficult case mining is characterized by comprising the following steps: step S1, receiving original data, extracting feature vectors to form an unlabeled sample set, and training an initial model based on the unlabeled sample set; step S2, calculating value indexes of multiple dimensions of unlabeled samples according to the initial model, dynamically adjusting weight of each dimension according to historical iteration performance feedback, and screening high-value samples as target to-be-labeled sets after fusion; step S3, automatically identifying difficult cases from the target to-be-marked set and the model error samples according to preset multidimensional rules, performing unsupervised clustering on the difficult cases and searching similar sample expansion to form difficult case sets and storing the difficult case sets in a dynamic pool; Step S4, marking the target set to be marked and the difficult case set according to priority, obtaining multiple marking results for the same batch of samples, and checking consistency; Step S5, extracting new data from the annotation database, constructing an incremental training set by combining the difficult sample in the dynamic pool, locking partial parameters of the current model, and training by adopting a mixed loss function to generate a candidate model; s6, evaluating the effectiveness of the difficult cases in the dynamic pool by using the candidate model after each iteration, eliminating the invalid difficult cases and merging redundant clusters to enable the capacity of the dynamic pool to be self-adaptively maintained in a preset proportion range; And S7, monitoring the performance of the candidate model, taking the candidate model as a next round of iteration base model if the performance of the candidate model does not reach the target, adaptively adjusting key parameters based on historical iteration information, and then jumping to the step S2, otherwise, stopping and outputting the target model.
  2. 2. The AI-marking closed-loop method based on active learning and difficult-to-case mining according to claim 1, wherein the process of step S2 includes: For each sample in the unlabeled sample set, acquiring the prediction probability distribution of the sample through the iterative basis model of the round, and calculating value indexes of three dimensions of uncertainty, representativeness and diversity in parallel; Carrying out orthogonality constraint preprocessing on the value indexes of the three dimensions to eliminate information redundancy so as to obtain each value component; Acquiring the boundary sample error rate, the distribution deviation rate and the sampling redundancy rate of the previous iteration as performance feedback signals, and inputting the performance feedback signals into a closed-loop feedback controller to generate dynamic weight coefficients of each value component; Weighting and fusing the value components according to the dynamic weight coefficients to obtain comprehensive information value scores of the samples and sorting the comprehensive information value scores according to the descending order of the scores; And adaptively determining the number of the samples of the round according to the lifting amplitude of the macro average F1 value of the candidate model of the previous round, and selecting high-value samples which are ranked at the front to form a target to-be-marked set.
  3. 3. The AI-marking closed-loop method based on active learning and difficult-to-case mining according to claim 2, wherein the process of calculating the value index of three orthogonal dimensions in parallel includes: And performing twice forward reasoning on each sample in the unlabeled sample set through the iterative basis model of the round to obtain the original and disturbed prediction probability distribution, fusing the prediction entropy, the confidence coefficient complement value and mutual information of the two distributions to calculate an uncertainty score, estimating the similarity of the sample and the data global distribution by adopting the kernel density estimation of the self-adaptive bandwidth to obtain a representative score, calculating the characteristic cosine distance between the sample and the sampled sample set of the round, and applying time attenuation weighting aggregation to obtain a diversity score.
  4. 4. The AI-marking closed-loop method based on active learning and difficult-to-find mining of claim 3, wherein the multi-dimensional rule preset in step S3 specifically includes: For each sample in the target to-be-annotated set and the model error samples, calculating the difficult contribution values of three dimensions in parallel: acquiring performance attenuation rate of the candidate model of the previous round on the difficult sample, if the attenuation rate exceeds a first threshold value, lifting the uncertainty dimension weight, if the long-tail class F1 value is not lifted, lifting the long-tail dimension weight, and if the accuracy of the boundary sample is reduced, lifting the confidence dimension weight, so as to form a difficult judgment weight vector of the previous round; And weighting and fusing the difficult case contribution values of the three dimensions according to the weight vectors to generate a comprehensive difficult case index, and if the index exceeds a difficult case judgment threshold value, taking the sample into a difficult case initial set.
  5. 5. The AI-marking closed-loop method based on active learning and difficult-to-case mining according to claim 4, wherein the process of step S4 includes: Preferentially distributing the difficult case set and the target set to be marked to a plurality of marking nodes, wherein nodes with the history marking accuracy higher than a preset threshold value preferentially distribute the difficult case set; Collecting independent labeling results of at least two labeling nodes for each sample, calculating a consistency coefficient, directly adopting the coefficient if the coefficient reaches a preset standard, and otherwise triggering an intelligent judging mechanism; In the judging process, obtaining dynamic credit scores of all labeling nodes, fusing the historical accuracy with the labeling stability on the difficult sample, obtaining the prediction confidence of the base model on the sample, carrying out temperature scaling calibration based on the statistical characteristics of the difficult pool, and calculating the feature similarity and the local density weight of the sample and the similar cluster center in the difficult pool; Weighting and fusing the dynamic reputation score, the calibrated confidence coefficient and the feature similarity to generate reliability scores of each candidate label, and selecting the highest score as a final label, wherein the fusion weight is adaptively adjusted according to the resolution of the disputed sample in the previous iteration; And storing the final label into a labeling database, and reversely updating the dynamic reputation scores of all labeling nodes to serve as a next round of task allocation basis.
  6. 6. The AI-marking closed-loop method based on active learning and difficult-to-case mining according to claim 5, wherein the process of using the dynamic reputation score of each standard node updated reversely as the basis for the next round of task allocation comprises: Counting historical accuracy difference values of all marking nodes on a common sample and a difficult sample, and judging that the difficult marking deviation exists in the nodes when the difference values exceed a preset threshold value; introducing an exponential decay factor to enable the near-term labeling stability of the node on the difficult sample to have higher weight than the long-term expression; And (3) carrying out weighted fusion on the calibrated accuracy and stability to generate a dynamic credit score, wherein the score is automatically updated after each round of iteration and is used as a quantization basis for task allocation of the next round.
  7. 7. The AI-marking closed-loop method based on active learning and difficult-to-case mining according to claim 6, wherein the process of step S5 includes: extracting newly added annotation data from an annotation database, and constructing an incremental training set by combining difficult sample in a dynamic difficult sample pool, wherein the difficult sample is weighted and sampled by giving an exponential increment weight according to the resident turn of the difficult sample in the pool; the bottom layer parameters with converged gradient changes in the frozen base model are updated only for the top layer and the middle layer with the contribution degree exceeding a preset threshold value for misjudgment of difficult cases; the difficult-case loss weight in the mixed loss function is adjusted in real time according to the dynamic ratio of the difficult-case sample loss falling rate to the common sample loss falling rate in the training process to obtain a self-adaptive loss weight coefficient; And if the performance of the candidate model does not reach the preset lifting condition, automatically calculating the parameter adjustment step length of each module based on the performance gap vector, and returning to the step S2 for re-execution after updating the sampling weight, the difficult case judgment threshold value and the self-adaptive loss weight coefficient in a linkage manner.
  8. 8. The AI-marking closed-loop method based on active learning and difficult-case mining according to claim 7, wherein the step of automatically calculating the parameter adjustment step of each module based on the performance gap vector, and returning to step S2 for re-execution after updating the sampling weight, the difficult-case decision threshold and the adaptive loss weight coefficient in a linkage manner comprises: analyzing the performance gap vector of the candidate model, if the recall rate of the boundary sample is lower than a preset recall reference value and the long tail class F1 value is higher than a preset F1 reference value, judging that the recognition of the difficult case is insufficient, at the moment, preferentially increasing the adjustment step length of the difficult case judgment threshold, and locking the sampling weight and the self-adaptive loss weight coefficient of the multi-target dynamic sampling; if the first level judgment is not established, and the sampling distribution deviation rate is higher than the preset deviation reference value while the boundary sample recall rate is lower than the recall reference value, judging that the boundary sample sampling deviation is generated, and at the moment, increasing the uncertainty weight of the multi-target dynamic sampling and synchronously reducing the similarity retrieval threshold value; if the first two levels are judged not to be established, judging that the training strategies are mismatched, at the moment, increasing the weight loss of the difficult case to a first preset weight value, thawing the two layers of parameters of the top layer of the basic model to restore the trainability, and increasing the K value sampling number of the multi-target dynamic sampling to the upper limit value of the preset sampling number, wherein the three levels are sequentially triggered by the adjustment operation, and the performance gap vector is recalculated after each layer of adjustment to carry out iterative diagnosis until the performance improvement condition is met.
  9. 9. The AI-marking closed-loop method based on active learning and difficult-to-case mining according to claim 8, wherein the process of step S6 includes: For each refractory sample in the dynamic pool, obtaining the prediction confidence coefficient by using a candidate model, and executing three-level effectiveness judgment by combining the resident turn of the sample in the pool and the long tail marking state of the category to which the resident turn belongs, wherein if the confidence coefficient is higher than a first threshold and is not the long tail category, the refractory sample is marked as a failure refractory sample, if the marked refractory sample is lower than a second threshold but the consistency coefficient of the cross turn is lower than a second threshold, the refractory sample is marked as a dispute refractory sample and the priority of the refractory sample is reduced; Performing elimination operation on the difficult sample marked as invalid, expired or redundant, simultaneously calculating the inter-cluster feature distance and intra-cluster mark dispersion between every two clusters, and triggering cluster merging operation if both the inter-cluster feature distance and the intra-cluster mark dispersion are lower than the corresponding threshold; and dynamically adjusting the upper limit of the capacity of the dynamic pool according to the performance lifting gradient of the candidate model.
  10. 10. The AI-marking closed-loop method based on active learning and refractory excavation of claim 9, wherein dynamically adjusting the dynamic pool capacity upper limit according to the candidate model performance boost gradient comprises: If the gradient exceeds the preset reference, the capacity upper limit is raised to 20% of the total sample size, if the gradient is lower than the preset reference, the capacity upper limit is lowered to 10% of the total sample size, otherwise, the current capacity proportion is maintained.

Description

AI marking closed-loop method based on active learning and difficult case mining Technical Field The invention relates to the technical field of artificial intelligence, in particular to an AI marking closed-loop method based on active learning and difficult case mining. Background With the deep application of artificial intelligence technology in the professional fields of industrial quality inspection, medical diagnosis, legal text analysis and the like, high-quality labeling data has become a core bottleneck for restricting the performance of a model. The traditional supervised learning method relies on large-scale randomly sampled labeling data, so that labeling cost is high, efficiency is low, and particularly the method has obvious defects in long-tail category identification and boundary sample learning. The existing active learning technology mainly screens unlabeled samples through single strategies such as uncertainty sampling, representative sampling or diversity sampling, but has obvious defects that the uncertainty sampling easily causes the samples to be concentrated on local decision boundaries and ignores global data distribution, the representative sampling can cover the distribution but is difficult to accurately locate high-value difficult cases, and the diversity sampling reduces redundancy but possibly omits critical boundary samples. According to the method, multiple targets are fused by adopting fixed weights, dynamic response to model performance change is lacked, and a sampling strategy cannot be adaptively adjusted according to feedback signals such as boundary sample error rate, distribution deviation rate and the like, so that the model performance is improved only limited under the same labeling cost. In the aspect of difficult cases mining, the prior art relies on manual screening or simple threshold judgment, and lacks an automatic recognition and expansion mechanism for difficult cases. The traditional method does not establish a dynamic maintenance mechanism of the difficult case pool, so that the learned invalid difficult case continuously occupies storage resources, and the emerging difficult case cannot be brought into a training closed loop in time. In addition, the capacity of the difficult pool is mostly static, gradient expansion and contraction cannot be improved according to model performance, and resource waste or insufficient samples are caused. The quality control of the labeling link depends on manual rechecking or simple majority voting, and a dynamic reputation evaluation system of the labeling node is not established. The existing method cannot quantify the stability difference of the annotators on the difficult sample, and the disputed sample processing lacks a weighted fusion mechanism of the model confidence and the feature similarity, so that the annotation quality fluctuates greatly and the model optimization effect is poor after the data backflow training. The incremental training stage usually adopts a full-parameter updating or fixed freezing strategy, and a parameter layer is selectively unfrozen according to the misjudgment contribution degree of the difficult cases. The weight of the difficult cases in the mixed loss function is a fixed value, and cannot be dynamically adjusted according to the loss dropping rate difference between the difficult cases and the common samples. When the performance of the model does not reach the standard, the prior art lacks a cross-module parameter linkage diagnosis mechanism, and the sampling weight, the difficult case judgment threshold value and the training strategy are adjusted in an isolated mode, so that collaborative optimization is difficult. In conclusion, the prior art has core problems of stiff sampling strategy, difficult closed loop deletion, weak parameter self-adaption capability, imperfect quality control mechanism and the like, so that the marking cost is high, the model is insufficient in learning boundary samples and long tail categories, and the data utilization rate is low. What is needed is an AI marking method integrating dynamic feedback, closed loop iteration and cross-module cooperation, which achieves the dual goals of reduced marking cost and improved model performance. Disclosure of Invention Therefore, the invention provides an AI marking closed-loop method based on active learning and difficult case mining, which is used for solving the problems in the prior art. In order to achieve the above purpose, the present invention provides an AI marking closed loop method based on active learning and difficult case mining, comprising: step S1, receiving original data, extracting feature vectors to form an unlabeled sample set, and training an initial model based on the unlabeled sample set; step S2, calculating value indexes of multiple dimensions of unlabeled samples according to the initial model, dynamically adjusting weight of each dimension according to historical iteration performance f