Search

CN-122018671-A - Intelligent vehicle-mounted interaction system and method based on multi-sensor fusion

CN122018671ACN 122018671 ACN122018671 ACN 122018671ACN-122018671-A

Abstract

The invention discloses an intelligent vehicle-mounted interaction system and method based on multi-sensor fusion, and relates to the technical field of multi-source data fusion.A time synchronization model is established by acquiring an interaction record with characteristic action codes, calculating a difference value of time stamps between continuous image frames and recognition actions and based on the difference value of characteristic light scores, motion scores and time stamps; and judging whether the gesture is abnormal or not according to the real-time data, if the gesture is abnormal, calculating a real-time timestamp difference value, synchronizing continuous image frames and recognition actions, and improving robustness and response speed of the gesture recognition in a complex environment.

Inventors

  • LIU WEI

Assignees

  • 江苏德海汽车科技股份有限公司

Dates

Publication Date
20260512
Application Date
20251208

Claims (10)

  1. 1. The intelligent vehicle-mounted interaction method based on the multi-sensor fusion is characterized by comprising the following steps of: Step 100, calculating to obtain characteristic light scores according to light parameters of the history identification records, counting abnormal conditions of the history identification records, calculating to obtain label values of the history identification records, determining the history abnormal identification records, assigning values to the history identification records, and establishing a light abnormal judgment model based on the characteristic light scores and the assignment values of the history identification records; Step 200, setting a training period, determining a light anomaly identification record based on a light anomaly judgment model, acquiring action information acquired by radar sensing equipment in the light anomaly identification record, identifying actions of a user according to an action database, calculating a motion score according to motion data of the actions, combining the actions with the motion score grade, coding, and determining characteristic action codes according to the occurrence frequency of the action codes; Step 300, obtaining an interactive record with characteristic action codes, calculating the difference value of time stamps between continuous image frames and recognition actions, and establishing a time synchronization model based on the characteristic ray scores, the motion scores and the difference value of the time stamps; Step 400, judging whether the light is abnormal or not according to the real-time data, if the light is abnormal, calculating a real-time timestamp difference value, and synchronizing the continuous image frames with the recognition action.
  2. 2. The intelligent vehicle-mounted interaction method based on multi-sensor fusion according to claim 1, wherein the step S100 comprises the following steps: Step S101, arranging a plurality of devices in a vehicle-mounted interactive device, presetting an action recognition period, collecting a current time point as an initial time point when a voice sensor recognizes a voice call of a user to the vehicle-mounted interactive device, starting radar sensing equipment and vision acquisition equipment in the vehicle-mounted interactive device, collecting action videos of the user according to the action recognition period, analyzing the collected videos to obtain action instructions of the user, meanwhile, collecting light parameters in a vehicle through the illumination sensing equipment, combining the collected action videos, the analyzed action instructions and the light parameters to generate interaction records corresponding to the action recognition period, collecting the current time point as an end time point when the vehicle-mounted interactive device judges that the voice call is ended, setting a time period between the initial time point and the end time point as an interaction time period, numbering the interaction records sequentially according to time stamps of the interaction records in the interaction time period, constructing complete recognition records, and uploading the recognition records to a vehicle-mounted interaction platform; step S102, acquiring a history identification record set, acquiring light parameters of each interaction record in the history identification record, carrying out normalization calculation on each light parameter, presetting a weight of each light parameter, carrying out weighted summation on the normalized light parameter and the corresponding weight, and calculating to obtain a light score of each interaction record; step S103, collecting the identification result of each interaction record in the history identification record, marking the interaction record with the normal identification result as the normal interaction record, marking the interaction record with the abnormal identification result as the abnormal interaction record, and calculating the characteristic light score of the history identification record according to the following formula: ; Wherein A is represented as a characteristic ray score, B a is represented as a ray score of the a-th normal interaction record, B d is represented as a ray score of the d-th abnormal interaction record, C1 is represented as a weight of the normal interaction record, C2 is represented as a weight of the abnormal interaction record, B is represented as the total number of the normal interaction records, and e is represented as the total number of the abnormal interaction records; Step S104, counting the number of normal interaction records and abnormal interaction records in each history identification record, and calculating the label value of the history identification record according to the following formula: ; Wherein D is represented as a label value, F1 is represented as a number weight of normal interaction records, F2 is represented as a number weight of abnormal interaction records, historical identification records of abnormal interaction records are summarized, an average value F1 and a standard deviation F2 of the label values are calculated, k is preset as a threshold coefficient, and a label value threshold d1=f1+kxf2 is calculated; And step 105, marking the history identification record as a history normal identification record if the label value of the history identification record does not exceed the label value threshold, marking the history identification record as a history abnormal identification record if the label value of the history identification record exceeds the label value threshold, assigning the history identification record according to the history normal identification record and the history abnormal identification record, combining the characteristic light scores and the assignments of each history identification record to obtain a characteristic light score training set, taking the characteristic light scores as input, taking the assignments as output, training through a random forest model, presetting a plurality of candidate probability thresholds in the training process, evaluating the performance index of the corresponding model according to each candidate probability threshold, calculating the performance index score corresponding to each candidate probability threshold, and selecting the candidate probability threshold corresponding to the best performance index score as the probability threshold to generate the light abnormal judgment model.
  3. 3. The intelligent vehicle-mounted interaction method based on multi-sensor fusion according to claim 2, wherein the step S200 comprises the following steps: Step S201, selecting a plurality of continuous days as a training period, acquiring identification records in the training period, acquiring light parameters of each interaction record in each identification record, calculating to obtain characteristic light scores of the identification records, inputting the characteristic light scores into a light abnormality judgment model to obtain light abnormality probability, and marking the identification records as light abnormality identification records and summarizing if the light abnormality probability exceeds a probability threshold; Step S202, acquiring action information acquired by radar sensing equipment in a light anomaly identification record in the light anomaly identification record set, wherein the action information comprises the distance, azimuth angle and pitch angle of a target point, acquiring action information in each action identification period, converting the distance, azimuth angle and pitch angle of each target point into three-dimensional space coordinates through conversion from polar coordinates to Cartesian coordinates, acquiring continuous three-dimensional space coordinates in the action identification period, and constructing three-dimensional space distribution of the action identification period; Step 203, extracting characteristic data of each target point based on three-dimensional space distribution of a continuous motion recognition period, matching the characteristic data with a motion database according to a preset motion database, recognizing a motion made by a user, calculating motion data of the motion, wherein the motion data comprises speed and acceleration of a motion space track, carrying out normalization calculation on the motion data, presetting a weight of the motion data, carrying out weighted summation on the normalized motion data, calculating to obtain motion scores, presetting a plurality of motion score grades, and determining a motion score range corresponding to each motion score grade; Step S204, obtaining abnormal identification records in the light abnormal identification record set, extracting actions corresponding to abnormal interaction records in the abnormal identification records and corresponding motion grading grades, combining the actions with the motion grading grades, coding, counting the number of the abnormal interaction records corresponding to each action code, calculating to obtain the occurrence frequency of the action codes, presetting an occurrence frequency threshold, and marking the action codes exceeding the occurrence frequency threshold as characteristic action codes.
  4. 4. The intelligent vehicle-mounted interaction method based on multi-sensor fusion according to claim 3, wherein the step S300 comprises the following steps: Step S301, in a light anomaly identification record set, acquiring an interactive record with characteristic motion codes, extracting motion videos acquired by a visual acquisition device, identifying and extracting continuous image frames corresponding to each motion of a user by a visual human body dynamic capture technology, acquiring a time stamp of each continuous image frame, extracting a time stamp corresponding to the identification motion of a radar sensing device, matching the continuous image frames with the identification motion, and calculating a difference value of the time stamps between the continuous image frames and the identification motion; Step S302, classifying the interaction records according to different actions, taking characteristic light scores and motion scores as inputs and taking the difference value of time stamps as output in an interaction record set of a certain action, and training through a linear regression model to generate a time synchronization model.
  5. 5. The intelligent vehicle-mounted interaction method based on multi-sensor fusion according to claim 4, wherein the step S400 comprises the following steps: Step S401, when a user calls the real-time voice of the vehicle-mounted interaction device, collecting real-time light parameters in the vehicle of the action recognition period, calculating to obtain real-time light scores, determining real-time light abnormality probability of the current action recognition period through a light abnormality judgment model, executing step S402 if the real-time light abnormality probability exceeds a probability threshold, and repeatedly executing step S401 if the real-time light abnormality probability does not exceed the probability threshold; step S402, acquiring real-time action information through radar sensing equipment, calculating to obtain a real-time motion score, inputting the real-time light score and the real-time motion score into a time synchronization model, calculating to obtain a real-time timestamp difference value, and synchronizing continuous image frames with the recognition action according to the real-time timestamp difference value.
  6. 6. An intelligent vehicle-mounted interaction system based on multi-sensor fusion, which is used for realizing the intelligent vehicle-mounted interaction method based on multi-sensor fusion as claimed in any one of claims 1 to 5, and is characterized in that the system comprises a light anomaly judgment model module, a characteristic action coding module, a time synchronization model module and a real-time synchronization module; The light anomaly judgment model module calculates and obtains characteristic light scores according to light parameters of the history identification records, counts the anomaly condition of the history identification records, calculates and obtains the label value of the history identification records, determines the history anomaly identification records, assigns values to the history identification records, and establishes a light anomaly judgment model based on the characteristic light scores and the assignment of the history identification records; The characteristic action coding module is used for setting a training period, determining a light abnormality identification record based on a light abnormality judgment model, acquiring action information acquired by radar sensing equipment in the light abnormality identification record, identifying actions of a user according to an action database, calculating a motion score according to motion data of the actions, combining the actions with the action score grade, coding, and determining characteristic action codes according to the occurrence frequency of the action codes; the time synchronization model module is used for acquiring an interaction record with characteristic action codes, calculating the difference value of time stamps between continuous image frames and recognition actions, and establishing a time synchronization model based on the difference value of characteristic ray scores, motion scores and time stamps; And the real-time synchronization module judges whether the light is abnormal according to the real-time data, if the light is abnormal, calculates a real-time timestamp difference value, and synchronizes the continuous image frames with the recognition action.
  7. 7. The intelligent vehicle-mounted interaction system based on multi-sensor fusion according to claim 6, wherein the light anomaly determination model module comprises a calculation feature light scoring unit and a light anomaly determination model building unit: the system comprises a calculation characteristic light scoring unit, a light marking unit and a light marking unit, wherein a plurality of devices are arranged in vehicle-mounted interaction equipment, a motion recognition period is preset, when a voice sensor recognizes a voice call of a user to the vehicle-mounted interaction equipment, a current time point is collected as an initial time point, radar sensing equipment and vision collecting equipment in the vehicle-mounted interaction equipment are started, a motion video of the user is collected according to the motion recognition period, the collected video is analyzed to obtain a motion instruction of the user, meanwhile, through the light sensing equipment, light parameters in a vehicle are collected, the collected motion video, the analyzed motion instruction and the light parameters are combined to generate an interaction record corresponding to the motion recognition period, when the vehicle-mounted interaction equipment determines that the voice call is finished, the current time point is taken as an end time point, a time period between the initial time point and the end time point is set as an interaction time period, the interaction record is sequentially numbered according to a time stamp in the interaction time period, a recognition record is constructed, and the recognition record is uploaded to the vehicle-mounted interaction platform; The method comprises the steps of establishing a light anomaly judgment model unit, counting the number of normal interaction records and abnormal interaction records in each history identification record, calculating the number of the label values of the history identification records, summarizing the history identification records with the abnormal interaction records, calculating the average value and standard deviation of the label values, presetting a threshold coefficient, calculating a label value threshold, marking the history identification records as history normal identification records if the label values of the history identification records do not exceed the label value threshold, marking the history identification records as history anomaly identification records if the label values of the history identification records exceed the label value threshold, assigning values to the history identification records according to the history normal identification records and the history anomaly identification records, combining the characteristic light scores and the assignment of each history identification record to obtain a characteristic light score training set, taking the characteristic light scores as input, assigning values as output, training through a random forest model, presetting a plurality of candidate probability thresholds in the training process, evaluating the performance indexes of the corresponding models according to each candidate probability threshold, calculating the performance index scores corresponding to each candidate probability threshold, selecting the candidate probability score corresponding to the best performance index score as a probability judgment model.
  8. 8. The intelligent vehicle-mounted interactive system based on multi-sensor fusion according to claim 6, wherein the characteristic action coding module comprises a determining light ray abnormality identification recording unit and a determining characteristic action coding unit: selecting a plurality of continuous days as a training period, acquiring identification records in the training period, acquiring light parameters of each interaction record in each identification record, calculating to obtain characteristic light scores of the identification records, inputting the characteristic light scores into a light abnormality judgment model to obtain light abnormality probability, and marking the identification records as light abnormality identification records and summarizing if the light abnormality probability exceeds a probability threshold; The method comprises the steps of acquiring action information acquired by radar sensing equipment in a light abnormal identification record set, acquiring action information in each action identification period, wherein the action information comprises distance, azimuth angle and pitch angle of a target point, converting the distance, azimuth angle and pitch angle of each target point into three-dimensional space coordinates through conversion from polar coordinates to Cartesian coordinates, acquiring continuous three-dimensional space coordinates in the action identification period, constructing three-dimensional space distribution of the action identification period, extracting characteristic data of each target point based on the three-dimensional space distribution of the continuous action identification period, matching the characteristic data with an action database according to a preset action database, identifying actions made by a user, calculating motion data of the actions, wherein the motion data comprises speed and acceleration of an action space track, carrying out normalization calculation on the motion data, carrying out weighting summation on the normalized motion data, presetting a plurality of action scoring grades, determining an action scoring range corresponding to each action scoring grade, acquiring an action scoring range corresponding to the action scoring grade, carrying out interaction scoring, carrying out the frequency of the corresponding to the identification record in the continuous action identification record set, carrying out the interaction scoring, carrying out the corresponding action scoring, and carrying out the frequency of the action is higher than the corresponding action scoring, and the action grade is coded, and the number of the corresponding action scoring is calculated, and the action is coded, and the number of the occurrence frequency of the corresponding to the action scoring is calculated.
  9. 9. The intelligent vehicle-mounted interactive system based on multi-sensor fusion according to claim 6, wherein the time synchronization model module comprises a calculation difference unit and a time synchronization model building unit: The difference calculating unit is used for acquiring interactive records with characteristic motion codes in a light abnormal identification record set, extracting motion videos acquired by a visual acquisition device, identifying and extracting continuous image frames corresponding to each motion of a user through a visual human body dynamic capture technology, acquiring a time stamp of each continuous image frame, extracting a time stamp corresponding to a radar sensing device identification motion, matching the continuous image frames with the identification motion, and calculating a difference value of the time stamps between the continuous image frames and the identification motion; The time synchronization model building unit classifies the interaction records according to different actions, takes characteristic light scores and motion scores as input and takes the difference value of time stamps as output in an interaction record set of a certain action, and trains through a linear regression model to generate a time synchronization model.
  10. 10. The intelligent vehicle-mounted interactive system based on multi-sensor fusion according to claim 6, wherein the real-time synchronization module comprises a light anomaly determination unit and an implementation synchronization unit: The judging light abnormal unit is used for collecting real-time light parameters in the vehicle of the action recognition period when a user calls the real-time voice of the vehicle-mounted interaction device, calculating to obtain a real-time light score, determining the real-time light abnormal probability of the current action recognition period through a light abnormal judgment model, executing the step S402 if the real-time light abnormal probability exceeds a probability threshold value, and repeatedly executing the step S401 if the real-time light abnormal probability does not exceed the probability threshold value; The implementation synchronization unit acquires real-time action information through radar sensing equipment, calculates to obtain a real-time motion score, inputs the real-time light score and the real-time motion score into a time synchronization model, calculates to obtain a real-time timestamp difference value, and synchronizes continuous image frames with the recognition action according to the real-time timestamp difference value.

Description

Intelligent vehicle-mounted interaction system and method based on multi-sensor fusion Technical Field The invention relates to the technical field of multi-source data fusion, in particular to an intelligent vehicle-mounted interaction system and method based on multi-sensor fusion. Background The existing gesture recognition technology is widely applied in the field of multi-sensor data fusion, particularly for the combination of a visual camera and a radar sensor, and becomes an important component part in an intelligent vehicle-mounted gesture interaction system, and the existing gesture recognition system is mostly based on a traditional single sensor or a simple sensor data fusion strategy and cannot always stably run in a complex environment; The existing gesture recognition system generally depends on a simple feature level or decision level fusion method, the time sequence synchronization problem of different sensor data cannot be fully considered, in a multi-sensor system, time deviation exists between the data due to clock drift and sampling frequency difference, error accumulation and response delay in the fusion process are caused, recognition precision and instantaneity are affected, and the existing space-time synchronization method has the problems of high algorithm complexity, poor instantaneity and the like when facing the accurate synchronization of heterogeneous data, so that the high-efficiency and stable gesture recognition requirement cannot be met; Therefore, how to solve the problem of space-time synchronization among multiple sensors, reduce errors in the data fusion process, and realize deep coordination of vision and millimeter wave radar so as to improve robustness and response speed of gesture recognition in a complex environment becomes a technical problem to be broken through in the current intelligent interaction field. Disclosure of Invention The invention aims to provide an intelligent vehicle-mounted interaction system and method based on multi-sensor fusion, so as to solve the problems in the prior art. In order to solve the technical problems, the invention provides the following technical scheme that the intelligent vehicle-mounted interaction method based on multi-sensor fusion comprises the following steps: Step 100, calculating to obtain characteristic light scores according to light parameters of the history identification records, counting abnormal conditions of the history identification records, calculating to obtain label values of the history identification records, determining the history abnormal identification records, assigning values to the history identification records, and establishing a light abnormal judgment model based on the characteristic light scores and the assignment values of the history identification records; Step 200, setting a training period, determining a light anomaly identification record based on a light anomaly judgment model, acquiring action information acquired by radar sensing equipment in the light anomaly identification record, identifying actions of a user according to an action database, calculating a motion score according to motion data of the actions, combining the actions with the motion score grade, coding, and determining characteristic action codes according to the occurrence frequency of the action codes; Step 300, obtaining an interactive record with characteristic action codes, calculating the difference value of time stamps between continuous image frames and recognition actions, and establishing a time synchronization model based on the characteristic ray scores, the motion scores and the difference value of the time stamps; Step 400, judging whether the light is abnormal or not according to the real-time data, if the light is abnormal, calculating a real-time timestamp difference value, and synchronizing the continuous image frames with the recognition action. Further, step S100 includes: Step S101, arranging a plurality of devices in a vehicle-mounted interactive device, presetting an action recognition period, collecting a current time point as an initial time point when a voice sensor recognizes a voice call of a user to the vehicle-mounted interactive device, starting radar sensing equipment and vision acquisition equipment in the vehicle-mounted interactive device, collecting action videos of the user according to the action recognition period, analyzing the collected videos to obtain action instructions of the user, meanwhile, collecting light parameters in a vehicle through the illumination sensing equipment, combining the collected action videos, the analyzed action instructions and the light parameters to generate interaction records corresponding to the action recognition period, collecting the current time point as an end time point when the vehicle-mounted interactive device judges that the voice call is ended, setting a time period between the initial time point and the end time point as an interact