CN-121809487-B - Cognitive state evaluation and intervention method and system based on semantic dynamic weighting, electronic equipment and storage medium

CN121809487BCN 121809487 BCN121809487 BCN 121809487BCN-121809487-B

Abstract

The application discloses a cognitive state evaluation and intervention method, a system, electronic equipment and a storage medium based on semantic dynamic weighting. The method comprises the steps of establishing personalized baseline data, continuously collecting behavior data and storing the behavior data in a ring buffer area in real time, when semantic tags in interaction are identified, backtracking and extracting behavior data segments aligned with the semantic tags from the ring buffer area, extracting behavior features from the behavior data segments, searching feature weights from a preset weight mapping table according to the semantic tags, carrying out weighted calculation on the behavior features to obtain dynamic weighted feature indexes, obtaining a cognitive state classification result according to the dynamic weighted feature indexes based on a classification model, calculating contribution degree to determine main attribution features, and finally matching and executing an intervention strategy from a strategy library based on the semantic tags and the main attribution features. The application realizes accurate attribution, reliable evaluation and accurate intervention on the cognitive state.

Inventors

Ren Peishi
HU LIPING
LI JING
GAO JIAN

Assignees

上海浩宜信息科技有限公司

Dates

Publication Date: 20260508
Application Date: 20260309

Claims (9)

1. The cognitive state evaluation and intervention method based on semantic dynamic weighting is characterized by comprising the following steps of: S1, acquiring initial behavior data of a user at an initial stage of an interaction process, and establishing personalized baseline data according to the initial behavior data, wherein the initial behavior data comprises a first initial modal data stream and a second initial modal data stream, and the personalized baseline data comprises a first baseline characteristic value calculated and stored based on the first initial modal data stream and a second baseline characteristic value calculated and stored based on the second initial modal data stream; s2, continuously collecting behavior data of a user after an initial stage of the interaction process, and storing the behavior data and time stamps corresponding to the behavior data into a ring buffer area in real time to form a behavior data stream, wherein the behavior data comprises a first mode data stream and a second mode data stream, and the ring buffer area stores the time stamps of the first mode data stream and the time stamps of the second mode data stream and the second mode data stream; S3, when the semantic tags in the interaction process are identified, analyzing the second-mode data stream to obtain the semantic tags and time intervals corresponding to the semantic tags; extracting a first modal data segment and a second modal data segment which are aligned with the semantic tag in time from the ring buffer based on the time interval, carrying out feature extraction on the first modal data segment to obtain a first modal feature sequence, comparing and normalizing the first modal feature sequence with the first baseline feature value to obtain a first modal behavior feature; S4, respectively acquiring a first feature weight corresponding to the first modal behavior feature and a second feature weight corresponding to the second modal behavior feature from a preset semantic-feature weight mapping table according to the semantic tag; performing multiplication weighted calculation on the first modal behavior characteristics by using the first characteristic weights to obtain first dynamic weighted characteristic indexes; performing multiplication weighted calculation on the second modal behavior characteristics by using the second characteristic weights to obtain second dynamic weighted characteristic indexes; S5, based on a preset cognitive state classification model, obtaining a classification result of the cognitive state according to the dynamic weighted feature indexes, and calculating the contribution degree of each dynamic weighted feature index to the classification result to determine main attribution features; And S6, matching and executing corresponding intervention strategies from a preset strategy library based on the semantic tags and the main attribution features.
2. The method according to claim 1, characterized in that the method further comprises the step of: and S7, evaluating the cognitive state change of the user as a feedback signal, and optimally updating the feature weights in the mapping table of the corresponding relation between the semantic tags and the feature weights and/or the intervention strategies in the strategy library in an online incremental learning mode according to the feedback signal, wherein the optimization updating comprises the steps of adjusting the feature weights corresponding to the main attribution features under the semantic tags according to the difference between the main attribution features and the actual cognitive state of external feedback, and/or adjusting the priority or parameters of the corresponding intervention strategies in the strategy library according to the instant feedback effect or the delay feedback effect obtained by the executed intervention strategies.
3. The method of claim 1, wherein the first initial modality data stream and the first modality data stream are both video streams, wherein the second initial modality data stream and the second modality data stream are both audio streams, and wherein: The first modal feature sequence comprises a head attitude angle sequence, and the first dynamic weighting feature index comprises a dynamic weighting attitude micro-tension index, wherein the dynamic weighting attitude micro-tension index is obtained by performing 4-12 Hz band-pass filtering on the head attitude angle sequence, extracting a high-frequency micro-fibrillation component, calculating the current energy value of the high-frequency micro-fibrillation component, comparing and normalizing the current energy value with a baseline energy value obtained by the first initial modal data stream to obtain a relative change rate, and performing the weighting calculation on the relative change rate; Or alternatively The first modal feature sequence comprises a facial optical flow change sequence, the second modal feature sequence comprises a harmonic-to-noise ratio sequence of voice, and the second dynamic weighted feature index comprises a multidimensional micro-expression-acoustic synchronicity index, wherein the multidimensional micro-expression-acoustic synchronicity index is obtained by calculating a decorrelation measure of the facial optical flow change sequence and the normalized harmonic-to-noise ratio sequence in time, normalizing the decorrelation measure, and carrying out the weighted calculation on a normalization result; Or alternatively The second modal feature sequence comprises real-time speech speed, the second baseline feature value comprises a speech speed baseline, the second dynamic weighting feature index comprises a semantic weighting speech speed change index, and the semantic weighting speech speed change index is obtained by comparing and normalizing the real-time speech speed with the speech speed baseline to obtain a speech speed change rate, and carrying out weighting calculation on the speech speed change rate.
4. The method of claim 1, wherein step S4 is followed by inputting the semantic tags, the dominant attribution features, and a cognitive imbalance score calculated from the dynamically weighted feature indicators to a pre-trained decision tree model to determine and execute a corresponding intervention strategy, wherein the cognitive imbalance score is obtained by weighting and summing the dynamically weighted feature indicators or by inputting the dynamically weighted feature indicators to a preset regression model, or In step S6, the policy repository is a double-index intervention matrix with the semantic tags and the main attribution features as double indexes.
5. The method according to claim 4, wherein when the policy repository is a dual-index intervention matrix, the method further comprises, after step S6, evaluating a cognitive state change of a user as a feedback signal, and performing optimization updating on feature weights in the semantic tag and feature weight correspondence mapping table and/or intervention policies in corresponding units in the dual-index intervention matrix in an online incremental learning manner according to the feedback signal, wherein the optimization updating comprises adjusting feature weights corresponding to the main attribution features under the semantic tag according to differences between the main attribution features and actual cognitive states of external feedback, and/or adjusting priorities or parameters of the corresponding intervention policies in the dual-index intervention matrix according to an instant feedback effect or a delay feedback effect obtained by an executed intervention policy.
6. A semantic dynamic weighting-based cognitive state assessment and intervention system, comprising: the baseline calibration module is used for collecting initial behavior data of a user in an initial stage of an interaction process and establishing personalized baseline data according to the initial behavior data; The data updating module is used for continuously collecting behavior data of a user after the initial stage of the interaction process, and storing the behavior data and a time stamp corresponding to the behavior data into a ring buffer area in real time to form a behavior data stream; the time sequence alignment module is used for identifying semantic tags in the interaction process, and extracting behavior data segments which are aligned in time in a time interval corresponding to the semantic tags by means of the annular buffer area in a backtracking mode; the dynamic weighting module is used for carrying out feature extraction on the behavior data segment extracted by the time sequence alignment module to obtain a feature sequence, and comparing and normalizing the feature sequence with the personalized baseline data to obtain at least one behavior feature; according to the semantic tag, searching a feature weight corresponding to the behavior feature under the semantic tag from a preset semantic-feature weight mapping table; performing multiplication weighted calculation on the behavior characteristics by using the characteristic weights to obtain at least one dynamic weighted characteristic index; The strategy execution module is used for receiving the dynamic weighted feature indexes output by the dynamic weighting module, obtaining classification results of the cognitive states according to the dynamic weighted feature indexes based on a preset cognitive state classification model, calculating the contribution degree of each dynamic weighted feature index to the classification results to determine main attribution features, and matching and executing corresponding intervention strategies from a preset strategy base based on the semantic tags and the main attribution features.
7. The system of claim 6, further comprising: The optimization updating module is used for evaluating the cognitive state change of a user as a feedback signal, and carrying out optimization updating on the feature weight in the mapping table of the corresponding relation between the semantic tag and the feature weight and/or the intervention strategy in the strategy library in an online incremental learning mode according to the feedback signal, wherein the optimization updating comprises the steps of adjusting the feature weight corresponding to the main attribution feature under the semantic tag according to the difference between the main attribution feature and the actual cognitive state fed back externally and/or adjusting the priority or parameter of the corresponding intervention strategy in the strategy library according to the instant feedback effect or the delay feedback effect obtained by the executed intervention strategy.
8. An electronic device comprising a processor and a memory, the memory having stored therein a computer program, the processor being configured to implement the method of any of claims 1-5 when the computer program is executed.
9. A computer readable storage medium, characterized in that a computer program is stored thereon, which computer program, when being executed by a processor, implements the method according to any of claims 1 to 5.

Description

Cognitive state evaluation and intervention method and system based on semantic dynamic weighting, electronic equipment and storage medium Technical Field The application relates to the technical field of artificial intelligence, emotion calculation and man-machine interaction, in particular to a cognitive state evaluation and intervention method, a system, electronic equipment and a storage medium based on semantic dynamic weighting. Background In the fields of intelligent training, man-machine conversation and the like, it is important to evaluate the cognitive state of a user accurately in real time. Existing cognitive state recognition technologies based on multimodal data (e.g., voice, video) typically determine a user's mental state by analyzing his behavioral characteristics (e.g., gestures, expressions, sounds, etc.). However, these techniques have the following drawbacks in general: First, the mapping relationship between the features and the states is stiff. The prior art generally builds a static "behavioral characteristics to cognitive state" mapping model. For example, "body fibrillation" is simply mapped to "tension". However, in complex interaction scenarios, the same behavioral appearance may originate from disparate internal states. For example, in price negotiations, "body fibrillation" may represent stress, but the same "body fibrillation" is more likely to represent deep thinking under high cognitive load when recall complex technical details. The prior art cannot distinguish such "homonymy" situations, resulting in frequent erroneous decisions on the user's status. Second, the timing of the multimodal data is mismatched. In multi-modality analysis, the processing speed of data of different modalities varies greatly. For example, capturing visual features such as facial micro-expressions is on the order of milliseconds, while speech recognition and natural language understanding (semantic analysis) typically have delays of several seconds. The prior art is often to simply match the semantic analysis result at the current moment with the behavior characteristics at the current moment, so that behaviors before a few seconds are related to semantic errors after a few seconds, and cause 'causal inversion', thereby seriously affecting the analysis accuracy. Again, there is a lack of personalization adaptation capability. Different users have their own physiological and behavioral patterns, such as basal speech rate, habitual body sloshing amplitude, etc. Most of the existing systems adopt global unified thresholds for judgment, which can lead to continuous false alarms for some users with specific behavior habits, so that the signal-to-noise ratio of the system is low and the reliability is poor. Thus, the prior art still has significant shortcomings in the accuracy, adaptability and accuracy of intervention in cognitive state assessment. Disclosure of Invention The application aims to provide a cognition state evaluation and intervention method, a system, electronic equipment and a storage medium based on semantic dynamic weighting, which aim to solve the problems of cognition state misjudgment caused by incapability of distinguishing 'homonymy and heteronymy', time sequence mismatching caused by data processing delay, poor adaptability caused by lack of a personalized model, difficulty in identifying emotion camouflage and the like in the prior art, thereby realizing more accurate attribution and effective intervention on the cognition state of a user. In order to achieve the aim, the application provides a cognitive state assessment and intervention method based on semantic dynamic weighting, which comprises the following steps of S1, collecting initial behavior data of a user in an initial stage of an interaction process, and establishing personalized baseline data according to the initial behavior data, wherein the initial behavior data comprises a first initial modal data stream and a second initial modal data stream, and the personalized baseline data comprises a first baseline characteristic value calculated and stored based on the first initial modal data stream and a second baseline characteristic value calculated and stored based on the second initial modal data stream; s2, continuously collecting behavior data of a user after an initial stage of the interaction process, and storing the behavior data and time stamps corresponding to the behavior data into a ring buffer area in real time to form a behavior data stream, wherein the behavior data comprises a first mode data stream and a second mode data stream, and the ring buffer area stores the time stamps of the first mode data stream and the time stamps of the second mode data stream and the second mode data stream; S3, when the semantic tags in the interaction process are identified, analyzing the second-mode data stream to obtain the semantic tags and the time intervals corresponding to the semantic tags; extracting a first moda