CN-122023792-A - Point prompt segmentation method and device based on attack defense mechanism and readable storage medium

CN122023792ACN 122023792 ACN122023792 ACN 122023792ACN-122023792-A

Abstract

The invention relates to a point prompt segmentation method and device based on an attack defense mechanism and a readable storage medium, belonging to the field of computer vision; the method comprises the following steps of constructing an initial prompt point heterogram integrating space and semantic information, inputting the prompt point heterogram as an environment state to an attack agent to finish noise prompt point injection and instant rewarding calculation, executing prompt point screening, repairing and rewarding calculation by the defense agent, performing game opposition and cooperative training by the double agents, performing iterative prompt point assessment and deletion by the aid of the defense agent after solidification training, outputting an optimized prompt point set, and applying to image segmentation.

Inventors

LIU XUEYU
WANG YUJIE
ZHANG XIAOYI
WEI MINGQIANG
WU YONGFEI
WANG RUI

Assignees

太原理工大学

Dates

Publication Date: 20260512
Application Date: 20260109

Claims (10)

1. The point prompt segmentation method based on the attack defense mechanism is characterized by comprising the following steps of: s1, constructing an initial prompt point heterogram fusing space and semantic information; s2, inputting the prompt point heterogeneous map as an environmental state to an attack intelligent agent to finish noise prompt point injection and instant rewarding calculation; S3, the current updated prompt point abnormal graph and segmentation performance feedback information are input to the defending agent together, the defending agent evaluates the harmfulness of each prompt point in the current updated prompt point abnormal graph, the prompt point with the highest harmfulness is selected to execute the deleting action so as to optimize the prompt set, and the instant rewards of the defending agent are calculated based on the segmentation performance improvement caused by the deleting action; Step S4, based on the instant rewards of the attack intelligent agent and the instant rewards of the defending intelligent agent, the strategy network parameters of the attack intelligent agent and the defending intelligent agent are updated by adopting a deep reinforcement learning algorithm, so that the attack intelligent agent and the defending intelligent agent are synergistically optimized in the countermeasure game to finish the twin-intelligent countermeasure game and the cooperative training, wherein the target of the attack intelligent agent is the minimum segmentation performance, and the target of the defending intelligent agent is the maximum segmentation performance; and S5, adopting the defending agent after curing training is completed, carrying out iterative prompt point evaluation and deletion on the new input image and the initial prompt point set, and outputting the optimized prompt point set.
2. The point prompt segmentation method based on the attack defense mechanism according to claim 1, wherein in step S1, an initial prompt point iso-composition of merging a spatial physical edge and a semantic feature edge is constructed, comprising the following steps: S11, extracting offline features, namely extracting features of an input image by utilizing a pre-trained visual basic model, acquiring a global feature map of the input image, extracting visual feature vectors of all prompt points according to initial prompt point positions of the input image, pre-calculating a spatial distance matrix and a semantic similarity matrix between all the prompt points, and storing the spatial distance matrix and the semantic similarity matrix in a lasting mode; Step S12, constructing an online environment, namely loading pre-stored features in a reinforcement learning environment, initializing all the features of the prompting points by fusing position coordinate codes, positive/negative label embedding and corresponding visual features of each prompting point, and respectively setting a physical edge and a semantically similar feature edge which are adjacent in space based on a pre-calculated space distance matrix and a semantically similar matrix to form an initial prompting point heterogram which simultaneously characterizes the spatial layout of the prompting points and the semantic association.
3. The point prompt segmentation method based on the attack defense mechanism according to claim 1, wherein in step S2, completing the noise prompt point injection and the instant prize calculation comprises the following steps: S21, inputting an initial prompt point isomerism graph as an environment state to an attack intelligent agent, evaluating the value of executing attack actions such as adding positive prompt points or negative prompt points on a preset candidate position network based on a built-in depth Q network of the attack intelligent agent, screening target attack actions according to an epsilon-greedy strategy, and finally generating noise prompt points in a current prompt point isomerism graph; S22, adding noise prompt points to an initial prompt point set to finish updating the abnormal prompt point composition; S23, inputting the prompt point set subjected to attack action disturbance into a downstream segmentation model, and outputting a disturbance segmentation mask by the downstream segmentation model; calculating to obtain a second similarity index based on the disturbance segmentation mask and the real segmentation mask, wherein the second similarity index represents the segmentation performance of the downstream segmentation model after disturbance of the attack action; And S24, a preset first similarity index is called and compared with a second similarity index, wherein the first similarity index is obtained by inputting an initial prompt point set before attack into a downstream segmentation model, and obtaining a real-time rewarding of an attack intelligent agent by a similarity quantification result between an initial segmentation mask and a real segmentation mask output by the model.
4. The point prompt segmentation method based on attack defense mechanism according to claim 1, wherein in step S3, calculating the instant prize of the defending agent comprises the steps of: step S31, the prompt point abnormal pattern in the current state and the segmentation performance feedback information are input to the defending intelligent agent together; S32, analyzing the abnormal pattern of the prompt points in the current state by the defense intelligent agent through the built-in depth Q network, and simultaneously encoding each indication point in the abnormal pattern and the relation of edges between the indication points by using the graph neural network; step S33, based on the coding result, the defending agent evaluates the 'harmfulness' degree of each existing prompting point in the prompting point heterogram under the current state, and outputs a Q value for each existing prompting point, wherein the Q value is used for representing the expected value of improving the splitting performance after deleting the corresponding existing prompting point; step S34, the defense intelligent agent screens out the prompt point with the highest Q value and executes the deleting action, removes the prompt point and all the relevant edges thereof from the prompt point heterogram under the current state, and finishes the purification and repair of the prompt point set; step S35, calculating the segmentation performance improvement quantity after the execution of the defense actions, and taking the segmentation performance improvement quantity as the instant rewards of the defense intelligent agent.
5. The point prompt segmentation method based on the attack defense mechanism according to claim 4, wherein in step S31, the segmentation performance feedback information specifically includes a second similarity index corresponding to the attack, a first similarity index corresponding to the attack, and an instant reward of the attack agent, and the core is used for feeding back the segmentation performance change condition caused by the current prompt point set.
6. The point prompt segmentation method based on the attack defense mechanism according to claim 4, wherein the segmentation performance improvement amount is obtained by comparing a third similarity index of a repaired segmentation mask and a real segmentation mask obtained by inputting a post-defense prompt point set into a downstream segmentation model with a second similarity index before defense.
7. The point hint segmentation method based on the attack defense mechanism according to claim 4, wherein the outputting the optimized hint point set in step S5 includes the steps of: Step S51, after the double-agent countermeasure game and collaborative training are completed, only the defending agent is reserved, the defending agent parameters are cured, and a final prompt point optimization model is formed based on the cured defending agent and then an reasoning stage is carried out; step S52, after entering an reasoning stage, repeating the initial prompt point heterograph constructed in the step S1 aiming at a new input image and an initial prompt point set, and calling the solidified defensive intelligent body to perform multiple iterative prompt point evaluation and deletion operations, wherein each round of prompt points with highest harmfulness are deleted until a preset stop condition is reached; And step S53, outputting the final residual prompt point set as an optimization result, and inputting the downstream segmentation model.
8. The method of claim 7, wherein in step S52, the preset stopping condition is that the maximum number of deletion rounds or the highest Q value of the remaining indication points is lower than a threshold.
9. A point hint splitting device based on an attack defense mechanism for implementing the method according to any one of claims 1 to 8, comprising: The composition module is used for fusing the space and the prompt point heterogeneous map of the semantic information; The countermeasure training module is used for constructing and training attack intelligent agents and defense intelligent agents; The model solidifying module is used for solidifying the strategy network of the defending agent after the countermeasure training is finished; And the reasoning optimization module is used for loading the defending agent after the solidification training is finished and executing iterative deletion optimization on the new input image and the prompt point.
10. A readable storage medium, characterized in that it has stored thereon a computer program which, when executed by a processor, implements the method according to any of claims 1 to 9.

Description

Point prompt segmentation method and device based on attack defense mechanism and readable storage medium Technical Field The invention relates to the field of computer vision and image segmentation, in particular to a point prompt segmentation method and device based on an attack defense mechanism and a readable storage medium. Background In the existing image segmentation technology, the segmentation performance of a segmentation Model such as SEGMENT ANYTHING Model (SAM) is greatly dependent on the quality of input prompt points, and the segmentation accuracy of the Model is obviously reduced by wrong prompt points or redundant prompt points, especially in the case of cross-domain application or noise of prompt information, the existing Model often shows an unstable segmentation effect. To improve the quality of the cue points, various methods for generating the cue points statically have been proposed, including strategies based on feature matching, geometric heuristic methods, generating the cue points by means of auxiliary models, and the like. However, these methods are generally unidirectional generation mechanisms, and lack adaptive adjustment capability for SAM model segmentation results. Therefore, when the initial cue point is poor in quality, uneven in distribution or contains misleading content, problems such as drift of the segmentation boundary and target miss-segmentation are easily caused. In recent years, part of the learnable prompt optimization methods, such as VRP-SAM, maskSAM and the like, optimize prompt generation by training a special prompt encoder or introducing an additional network, and achieve a certain effect on a specific task. However, such methods often rely on task-related supervisory data or require fine-tuning of the model, and are difficult to apply directly in non-labeling or cross-domain scenarios. In addition, since the prompt generation module and the decoding process of the SAM are mutually independent, the prompt points cannot be dynamically adjusted in real time according to the actual errors of the current segmentation, the robustness to the prompt noise is insufficient, and the practical application range of the method is limited. There have been other studies attempting to optimize the hint layout using reinforcement learning methods, such as PPO-based methods that adjust hint distribution by learning structural relationships between points. However, such methods typically have a single training environment, and the reward function is not directly related to the SAM segmentation result, resulting in a weak adaptability of the optimization strategy to initial prompt quality and scene changes. More importantly, the existing methods generally lack effective recognition and inhibition mechanisms for the 'harmful prompt points'. When the prompt points are derived from feature matching, weak supervision segmentation or manual labeling, a large number of misleading points are often introduced, and the points are difficult to effectively filter in the prior art. In summary, the current technology does not provide a task independent prompt optimization mechanism capable of performing adaptive adjustment by using segmentation feedback and having noise prompt cleaning capability. Disclosure of Invention The invention provides a point prompt segmentation method, a point prompt segmentation device and a readable storage medium based on an attack defense mechanism, and aims to solve the technical problems that the existing prompt driving segmentation technology is sensitive to prompt noise, lacks an automatic prompt quality evaluation and optimization mechanism and is insufficient in cross-scene generalization capability. In order to solve the technical problems, the technical scheme adopted by the invention is that the point prompt segmentation method based on the attack defense mechanism comprises the following steps: s1, constructing an initial prompt point heterogram fusing space and semantic information; s2, inputting the prompt point heterogeneous map as an environmental state to an attack intelligent agent to finish noise prompt point injection and instant rewarding calculation; S3, the current updated prompt point abnormal graph and segmentation performance feedback information are input to the defending agent together, the defending agent evaluates the harmfulness of each prompt point in the current updated prompt point abnormal graph, the prompt point with the highest harmfulness is selected to execute the deleting action so as to optimize the prompt set, and the instant rewards of the defending agent are calculated based on the segmentation performance improvement caused by the deleting action; Step S4, based on the instant rewards of the attack intelligent agent and the instant rewards of the defending intelligent agent, the strategy network parameters of the attack intelligent agent and the defending intelligent agent are updated by adopting a deep reinforceme