Search

CN-121789911-B - Interactive autism children training system, method, electronic equipment and readable storage medium

CN121789911BCN 121789911 BCN121789911 BCN 121789911BCN-121789911-B

Abstract

The application discloses an interactive autism children training system, an interactive autism children training method, electronic equipment and a readable storage medium. The interactive autism child training system comprises a training configuration module, a TOF radar, a multi-mode sensing fusion camera set, a gesture training recognition module, an entity interaction behavior recognition module and a prompting module, wherein the training configuration module is used for configuring training actions/training elements corresponding to autism life self-care skills, the TOF radar is used for collecting gesture training data of palms and knuckles, the gesture training recognition module is used for judging the gesture training data, the multi-mode sensing fusion camera set is used for collecting entity interaction behavior data of trained objects, the entity interaction behavior recognition module is used for judging the entity interaction behavior data, and the prompting module is used for outputting the objects according to the individuation capability models of the trained objects. The application is used for improving the scientificity and systematically of training.

Inventors

  • HUANG ZIHAO
  • ZHONG XUESONG
  • QU SHIHAO
  • CHEN YANHUI

Assignees

  • 广东晔生科技股份有限公司

Dates

Publication Date
20260508
Application Date
20260305

Claims (9)

  1. 1. The interactive autism child training system is characterized by comprising a training configuration module, a TOF radar, a multi-mode sensing fusion camera set, a gesture training recognition module, an entity interactive behavior recognition module, a prompt module and an end-side intelligent computing module; the training configuration module is used for establishing a personalized capability model based on the cognition level, limb movement capability and perception sensitivity of a trained object, configuring training actions/training elements corresponding to the autism life self-care skills according to the personalized capability model, and supporting dynamic self-adaptive parameter adjustment of training difficulty, training rhythm and feedback modes; the TOF radar is used for carrying out dynamic background suppression, palm point cloud segmentation and three-dimensional world coordinate system registration on the hand area of the trained object, and collecting gesture exercise data of the palm and the knuckle; The gesture exercise recognition module is used for carrying out 3D space-time feature extraction, feature dimension reduction and similarity matching on gesture exercise data, judging the matching degree of the gesture exercise data and configured training actions/training elements, and evaluating the action proficiency of a trained object through a multidimensional quantization model; the multi-mode sensing fusion camera set is used for collecting whole body refined skeleton point three-dimensional space-time coordinate data stream of a trained object, multi-scale enhanced image data of an entity object and interaction contact data of limbs and entities as entity interaction behavior data; The entity interaction behavior recognition module is used for carrying out limb-entity space-time correlation feature fusion and interaction logic reasoning on entity interaction behavior data, judging the matching degree of the entity interaction behavior data and configured training actions/training elements, and evaluating the action proficiency of a trained object; the prompting module is used for outputting a training guiding object with visual-auditory-tactile multi-mode linkage and a recognition result feedback object according to the personalized capability model of the trained object; The intelligent end-side computing module adopts a heterogeneous computing architecture to realize local real-time preprocessing of all sensing data, low-delay reasoning of an AI model, privacy desensitization of training data and collaborative scheduling of multiple modules; The entity interaction behavior recognition module comprises a limb track modeling unit with a space-time attention mechanism, an entity object recognition unit with small sample learning and a limb-entity interaction logic reasoning unit; The limb track modeling unit is used for carrying out attention weight distribution on three-dimensional space-time coordinate sequence data streams of whole body refined skeleton points, focusing key skeleton points related to training actions, extracting the motion track, angle change and speed/acceleration characteristics of the key skeleton points through track characteristic decoupling, and judging the coincidence degree of the limb action track of a trained object with the set training actions on spatial form, motion time sequence and action amplitude after calculating the limb action track of the trained object; The entity object recognition unit is used for extracting visual features and semantic features of the multi-scale fusion image data of the entity object, accurately recognizing a scarce entity sample in self-care training of autism life based on a small sample learning algorithm, and judging the placement posture, interaction state and identity/similarity of the entity object and training elements of the entity object; The limb-entity interaction logic reasoning unit is used for carrying out interaction logic reasoning on the limb track characteristics and the entity object state characteristics based on the Bayesian network, and judging whether the limb actions of the trained object complete the interaction behavior meeting the training requirements on the target entity object.
  2. 2. The interactive autism child training system of claim 1, wherein the prompting module comprises an AR spatial perspective projection unit, a multi-channel spatial audio unit, a haptic feedback unit; the AR space perspective projection unit is used for anchoring the visual object of the training action/training element in a physical entity interaction area based on a space calibration and plane reprojection algorithm, realizing virtual-real fusion display, outputting an identification result feedback object with dynamic visual enhancement, and adaptively adjusting a visual enhancement strategy according to the visual perception sensitivity of the trained object; The multichannel spatial audio unit is used for outputting a training guiding auditory object matched with the spatial position of the visual object by adopting a customized TTS voice synthesis and spatial sound image positioning technology, supporting the adaptation of the frequency, the volume, the speech speed and the feedback interval of the audio according to the auditory perception threshold of the trained object, and outputting a recognition result feedback object with voice emotion; the haptic feedback unit is used for outputting haptic feedback when the trained object makes matching/non-matching actions through the vibration module with the vibration frequency/intensity being graded, and the visual-auditory-haptic multi-mode feedback closed loop is realized.
  3. 3. The interactive autism child training system according to claim 2, wherein the TOF radar is an area array solid-state TOF radar with a sampling frame rate of not less than 120fps and a point cloud resolution of not less than 640 x 480; the TOF radar is provided with a point cloud preprocessing unit, the accurate segmentation of palm and knuckle areas is realized through European clustering and an area growth algorithm, a three-dimensional space-time coordinate sequence of 21 fine skeleton points of a trained object palm is collected to be used as corresponding gesture exercise data, and inter-frame drift of hand motions is eliminated through inter-frame point cloud registration; the gesture exercise recognition module pre-builds a 3D space-time feature template library of training actions, performs space-time feature extraction on a three-dimensional space-time coordinate sequence of palm fine skeleton points through an improved 3DCNN+LSTM hybrid network, generates a high-dimensional gesture feature vector, and then judges the matching degree of the gesture feature vector and corresponding training action features in the feature template library by combining a dynamic distance threshold constraint through a two-dimensional matching algorithm of Euclidean distance and cosine similarity, and supports differential recognition and matching of single-hand/double-hand fine gestures.
  4. 4. The interactive autism child training system of claim 2, wherein the multimodal sensory fusion camera set includes a somatosensory depth camera, a high definition visible light camera, an infrared interactive contact camera; The motion sensing depth camera collects three-dimensional space-time coordinate sequence data streams of more than or equal to 25 refined skeleton points of the whole trained object, and smoothing and complementing skeleton point tracks are achieved through Kalman filtering; the method comprises the steps that a high-definition visible light camera collects multi-scale fusion image data of an entity object, and the multi-scale fusion image data are used as visual data of the entity object after dim light enhancement and motion blur removal pretreatment; The infrared interactive contact camera collects contact area, contact time sequence and contact pressure trend data of the limb of the trained object and the entity object; The entity interaction behavior recognition module deeply fuses space-time characteristics of limb skeleton points, visual characteristics of entity objects and interaction contact characteristics of limb-entity through a cross-modal characteristic fusion network to generate joint characteristic vectors of limb-entity interaction, focuses on key interaction characteristics through an attention mechanism, judges the matching degree of the joint characteristic vectors and configured training actions/training elements, and supports triple judgment on sequential logic, contact accuracy and action amplitude of limb-entity interaction.
  5. 5. The interactive autism children training system according to claim 4, wherein the limb space coordinate data stream is a microsecond-level time stamp three-dimensional space-time coordinate sequence data stream of more than or equal to 25 refined skeleton points of the whole trained object, and the tracking precision of the skeleton points is less than or equal to 0.5mm; The limb track modeling unit is used for carrying out track feature modeling and limb action track calculation on the skeleton point space coordinate sequence data stream based on an improved dynamic time warping algorithm with feature weighting and time window constraint by combining a hidden Markov model, realizing the probabilistic reasoning of an action time sequence, and judging whether the limb action track coincides with the set training action; the entity object identification unit adopts a light-weight YOLOv-Lite combined with migration learning and small sample fine-tuning visual deep learning network, and the visual deep learning network supports the end-side incremental update of model parameters through the pre-training and fine-tuning optimization of a self-closing child life self-care training exclusive entity data set; The action proficiency is judged based on a four-dimensional quantitative evaluation model, the four-dimensional quantitative evaluation model fuses four dimensions of matching accuracy ratio, motion fluency change of matched actions in a set time period, time sequence compliance of action execution and interaction accuracy of limbs and entities when a trained object makes actions according to training actions, the variation trend of the action proficiency is predicted through an LSTM time sequence prediction model, and meanwhile, the self-adaptive dynamic adjustment of a proficiency evaluation threshold is realized according to a training curve of the trained object.
  6. 6. The interactive autism child training system of claim 1, further comprising a reporting module, a medical record module, a privacy calculation module; The report module is used for carrying out deep analysis on training data through machine learning cluster analysis and trend fitting algorithm according to the judgment result of the gesture exercise recognition module and/or the entity interactive behavior recognition module, generating a training report containing training quantized data, proficiency trend analysis, interactive behavior characteristics and intelligent training effect diagnosis conclusion, and generating customized follow-up training scheme optimization suggestions based on the diagnosis conclusion combined with the individuation capability model of the trained object, wherein the training proposal comprises self-adaptive adjustment of training content, training difficulty and training duration; The medical record file module is used for carrying out blockchain storage type encryption storage on basic information, individuation capability models, training reports and training data of trained objects, and supporting the authorized hierarchical access of multiple terminals and the standardized synchronization of the multi-center training data; The privacy calculation module is used for carrying out differential privacy desensitization and federal learning processing on all training data, so as to realize multi-equipment/multi-center model combined training and optimization.
  7. 7. An interactive autism child training method, applied to the interactive autism child training system of any one of claims 1-6, comprising: establishing a personalized capability model based on the cognitive level, limb movement capability and perception sensitivity of a trained object, configuring training actions/training elements corresponding to autism life self-care skills according to the personalized capability model, and supporting dynamic self-adaptive parameter adjustment of training difficulty, training rhythm and feedback modes; performing dynamic background suppression on a trained object hand area, performing palm point cloud segmentation and three-dimensional world coordinate system registration, and collecting gesture exercise data of palms and knuckles; 3D space-time feature extraction, feature dimension reduction and similarity matching are carried out on gesture exercise data, the matching degree of the gesture exercise data and configured training actions/training elements is judged, and the action proficiency of a trained object is evaluated through a multidimensional quantization model; collecting whole body refined skeleton point three-dimensional space-time coordinate data stream of a trained object, multi-scale enhanced image data of an entity object and interaction contact data of limbs and entities as entity interaction behavior data; Carrying out limb-entity space-time correlation feature fusion and interaction logic reasoning on entity interaction behavior data, judging the matching degree of the entity interaction behavior data and configured training actions/training elements, and evaluating the action proficiency of a trained object; outputting a training guiding object with visual-auditory-tactile multi-mode linkage and an identification result feedback object according to the personalized capability model of the trained object; The method comprises the steps of local real-time preprocessing of all sensing data, low-delay reasoning of an AI model, privacy desensitization of training data and collaborative scheduling of multiple modules.
  8. 8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, the processor implementing the computer program implementing the interactive autism child training system of any of claims 1-6.
  9. 9. A readable storage medium storing computer instructions for causing a computer to implement the interactive autism child training system of any of claims 1-6.

Description

Interactive autism children training system, method, electronic equipment and readable storage medium Technical Field The application belongs to the technical field of man-machine interaction equipment, and particularly relates to an interactive autism children training system, an interactive autism children training method, electronic equipment and a readable storage medium. Background Autism spectrum disorder (Autism Spectrum Disorder, ASD) children often face significant difficulties in learning and developing life-style self-care skills, requiring systematic, highly repeatable and structurally rigorous training equipment to effectively guide the course of treatment and monitor patient compliance. In the prior art, there are at least the following problems: (1) The traditional training adopts a one-to-one manual guiding mode, highly depends on personal experience of a trainer, has a boring teaching process, is difficult to maintain interest and concentration of autism children for a long time, and lacks objective and consistent training effect evaluation means; (2) Part of the prior art uses a simple video observation learning mode, has poor interactivity, is in a passive receiving state for children, and lacks an implementation mode for transferring skills from cognition to practice; (3) In part of the interactive projection training modes in the prior art, a simple touch control or basic action induction technology is adopted, the interactive dimension is single, fine action details when a child operates real objects cannot be effectively captured and quantified, and systematic data recording and quantitative analysis are difficult to carry out on the whole training process, so that the treatment process cannot be accurately guided for a personalized training scheme, and the compliance of patients cannot be detected. Disclosure of Invention The application aims to provide an interactive autism children training system, an interactive autism children training method, electronic equipment and a readable storage medium, which are used for improving the scientificity and systemicity of training. In a first aspect, the application provides an interactive autism child training system, which comprises a training configuration module, a TOF radar, a multi-mode sensing fusion camera set, a gesture training recognition module, an entity interactive behavior recognition module, a prompt module and an end side intelligent computing module; the training configuration module is used for establishing a personalized capability model based on the cognition level, limb movement capability and perception sensitivity of the trained object, configuring training actions/training elements corresponding to the autism life self-care skills according to the personalized capability model, and supporting dynamic self-adaptive parameter adjustment of training difficulty, training rhythm and feedback modes; the TOF radar is used for carrying out dynamic background suppression, palm point cloud segmentation and three-dimensional world coordinate system registration on the hand area of the trained object, and collecting gesture exercise data of the palm and the knuckle; The gesture exercise recognition module is used for carrying out 3D space-time feature extraction, feature dimension reduction and similarity matching on gesture exercise data, judging the matching degree of the gesture exercise data and configured training actions/training elements, and evaluating the action proficiency of a trained object through a multidimensional quantization model; the multi-mode sensing fusion camera set is used for collecting whole body refined skeleton point three-dimensional space-time coordinate data stream of a trained object, multi-scale enhanced image data of an entity object and interaction contact data of limbs and entities as entity interaction behavior data; The entity interaction behavior recognition module is used for carrying out limb-entity space-time correlation feature fusion and interaction logic reasoning on entity interaction behavior data, judging the matching degree of the entity interaction behavior data and configured training actions/training elements, and evaluating the action proficiency of a trained object; the prompting module is used for outputting a training guiding object with visual-auditory-tactile multi-mode linkage and a recognition result feedback object according to the personalized capability model of the trained object; and the intelligent end-side computing module adopts a heterogeneous computing architecture to realize local real-time preprocessing of all sensing data, low-delay reasoning of an AI model, privacy desensitization of training data and collaborative scheduling of multiple modules. Preferably, the prompting module comprises an AR space perspective projection unit, a multichannel space audio unit and a touch feedback unit; the AR space perspective projection unit is used for anchoring the visu