CN-122024550-A - Infant care AR practical training system for home carers and skill assessment method

CN122024550ACN 122024550 ACN122024550 ACN 122024550ACN-122024550-A

Abstract

The invention discloses an infant care AR practical training system and a skill assessment method for home carers, and relates to the technical field of augmented reality. The method mainly comprises the steps of sensing a real operation environment by using a mobile terminal and aligning virtual space and real space, capturing and analyzing motion tracks of hands of a user and general care props in real time, calculating key characteristic indexes, judging compliance based on a preset nursing operation flow state machine and a digital rule, combining health degree scores output by a deep learning model to form a fusion evaluation result, and finally intelligently scheduling multi-mode feedback according to the evaluation result to provide real-time error correction and operation guidance in a visual-audio-tactile cooperative mode. The invention effectively overcomes the defects of high cost, cracking sense of actual operation and lack of intelligent feedback in the traditional training, and realizes the high-immersion, evaluable and low-cost infant nursing household skill training.

Inventors

WEI LILI
LI QIANQIAN
PAN YANZHUO
Pan Yueshuai
GU RUTING
WANG JINGYUAN

Assignees

青岛大学

Dates

Publication Date: 20260512
Application Date: 20260203

Claims (8)

1. Infant care AR practical training system towards family caregivers, characterized in that, this system includes: the virtual-real fusion guiding module is used for creating and managing an augmented reality environment, superposing virtual guide information in a real operation scene and providing a virtual infant model corresponding to the physical prop; The visual perception module is used for capturing the hand actions and the physical prop states of the user in real time through the camera and outputting the key point positions and the motion track data; The intelligent evaluation and interaction feedback module is used for evaluating the data output by the visual perception module in real time according to a preset nursing operation rule and scheduling multi-mode feedback based on an evaluation result; the virtual-real fusion guide module, the visual perception module and the intelligent evaluation and interaction feedback module are integrated on the mobile intelligent terminal; the output of the visual perception module is imported to an intelligent evaluation and interaction feedback module; And the output of the intelligent evaluation and interaction feedback module is fed back to the virtual-real fusion guide module and the mobile intelligent terminal.
2. The infant care AR practical training system for home caregivers according to claim 1, wherein the virtual-real fusion guiding module comprises: invoking an augmented reality frame of the mobile terminal, performing instant positioning and map construction on a real operation environment where a user is located through a camera, and identifying and locking a horizontal reference surface for placing entity props; Detecting a solid baby doll positioned on the horizontal reference plane, determining the position and the gesture of the solid baby doll, and registering and rendering a three-dimensional virtual baby model matched with the gesture and the size of the solid baby doll on corresponding space coordinates; according to the current training task and the user operation stage, graphic operation guide information and state feedback information are overlapped and rendered on the three-dimensional virtual infant model and the real environment picture in real time, wherein the operation guide information comprises standard operation animation, direction indication marks or key part highlight areas; Continuously tracking the position change of the solid infant doll, dynamically adjusting the gesture of the three-dimensional virtual infant model according to the position change, and performing translucency or contour strengthening treatment on the part which is shielded in the virtual model when the solid infant doll is detected to be shielded.
3. The home care provider-oriented infant care AR practical training system of claim 1, wherein the visual perception module comprises: Synchronously loading and initializing a hand gesture estimation model and a general object detection model, and synchronously inputting a real-time video stream captured by a mobile terminal camera into the hand gesture estimation model and the general object detection model for processing; Reasoning each frame of image through the hand gesture estimation model, identifying and outputting a three-dimensional space coordinate sequence of a user hand joint point, and generating hand key point data; Reasoning the same frame of image through the universal object detection model, identifying nursing props in the image, positioning three-dimensional space coordinates of functional key points, and generating prop key point data; and converting the hand key point data and the prop key point data into the same world coordinate system established by the virtual-real fusion guide module, performing time synchronization, and generating a fusion sensing data packet containing the relative position, posture and motion vector of the hand and the prop, and outputting the fusion sensing data packet as key point position and motion track data.
4. The home care provider-oriented infant care AR practical training system of claim 3, wherein the visual perception module performs an optimization method in data processing, comprising: smoothing and correcting the originally identified discrete hand key point data and prop key point data by using basic physical and kinematic constraint as an optimization condition to generate a motion track conforming to a physical rule; Calculating real-time confidence scores for each identified hand key point data and prop key point data according to the original output confidence of the comprehensive model, the motion consistency of the current point in the continuous frames and the rationality of the preset geometric relationship formed by the current point and the adjacent key points; When the confidence score is lower than a preset threshold, a context repair mechanism based on a graph model is started, and the current position data is deduced and complemented in real time by utilizing the topological relation of other point positions meeting the confidence threshold in the same frame and the motion trend of the current point in the previous and subsequent frames; And for the identified non-rigid care prop, identifying and tracking a characteristic texture area or a wrinkle pattern on the surface of the non-rigid care prop, analyzing deformation, displacement and relative movement of the characteristic texture area or the wrinkle pattern in continuous frames, generating deformation track data describing the operation action of a user on the non-rigid prop, and supplementing the deformation track data into the fusion perception data packet.
5. The infant care AR practical training system for home caregivers according to claim 1, wherein the intelligent evaluation and interaction feedback module, Loading a corresponding evaluation rule set from a preset digital nursing rule base according to training items selected by a user, and initializing the context state of a current training session; Receiving a fusion perception data packet from the visual perception module, and calculating key characteristic indexes for evaluation in real time, wherein the key characteristic indexes comprise a spatial relationship index, a time sequence dynamic index and a track conformity index; Inputting the calculated key characteristic indexes into a rule evaluation engine, wherein the engine carries out parallel logic judgment according to a layering rule corresponding to the current operation stage, and fuses operation trend analysis in a time sequence window to generate a real-time evaluation state vector containing judgment results of each dimension and quantization deviation degree; inputting the real-time evaluation state vector into a multi-modal feedback scheduler, generating and synchronously driving a graphic rendering, an audio playing and a control instruction of a touch vibration device of the mobile terminal according to a preset feedback mapping strategy, and performing interactive feedback of audio-visual touch coordination; And recording the evaluation state and user response data in the training process, and dynamically fine-tuning the threshold parameters of the evaluation rules based on the recorded data.
6. The home care provider oriented infant care AR practical training system of claim 5, wherein the rule evaluation engine decomposes each care action into several sequential operational phases and builds a corresponding operational phase state machine; When the state machine confirms that the operation precondition of the current stage is met, starting the operation rule corresponding to the next stage for evaluation, and driving the state machine to perform state transition; The method comprises the steps that a virtual infant physiological response model capable of predicting the physiological response of a virtual infant according to real-time operation data is preset, the rule evaluation engine compares the predicted physiological response with the correct physiological response which accords with standard operation and is triggered, and if the predicted physiological response and the correct physiological response accord with standard operation, high-level warning feedback is triggered.
7. The home carer-oriented infant care AR practical training system according to claim 6, wherein the rule evaluation engine adopts an intelligent evaluation method based on deep learning and operation phase state machine fusion, comprising: Inputting a time sequence formed by key characteristic indexes into a multi-mode time sequence characteristic encoder based on an attention mechanism, and generating a high-order time sequence characteristic vector representing a short-time operation mode; Comparing the high-order time sequence feature vector with a preset stage transfer condition by using an operation layering state machine comprising a plurality of operation stage nursing in a preset sequence, and dynamically driving the state transfer of the state machine; In the current operation stage, according to a preset rule threshold corresponding to the current stage, carrying out logic judgment on original key feature indexes, inputting current and historical high-order time sequence feature vectors into a trend prediction and scoring network in parallel, predicting future development trend of operation features, integrating the feature mode of the current operation fragment and the concentration degree and uncertainty of a prediction track, and outputting continuous health degree scores reflecting operation fluency, stability and predictability; And fusing the logic judgment result and the health degree score to generate a real-time evaluation state vector comprising a specific error label, a quality evaluation label and a quantitative deviation degree.
8. A method for evaluating infant care AR practical training skills for home caregivers, which is used for the infant care AR practical training system for home caregivers according to any one of claims 1 to 7, comprising the steps of: The method comprises the steps that a real operation environment comprising a solid baby doll and a nursing prop is spatially registered and modeled by utilizing an augmented reality frame and a camera of a mobile terminal, and video streams in the operation process are synchronously acquired; based on the video stream, three-dimensional motion tracks of key point points of a user hand and space pose of key points of a prop are recognized and tracked in real time, and key characteristic indexes for evaluating nursing operation skills are calculated based on the motion tracks and the space pose, wherein the key characteristic indexes comprise space relation indexes, time sequence dynamic indexes and track coincidence indexes; According to the current training nursing item, loading a preset nursing operation flow state machine and a corresponding digital evaluation rule base, combining a time sequence feature vector formed by the key feature indexes, driving the state machine to identify the current operation stage, calling an evaluation rule corresponding to the stage to carry out logic judgment, and simultaneously inputting the time sequence feature vector into a pre-training trend prediction and scoring network to obtain continuous health degree scores; Fusing the logic judgment result and the health degree score to generate a real-time evaluation state vector containing specific error types, quality evaluation and quantization scores; And according to the content of the real-time evaluation state vector, dispatching the optimal feedback combination from a preset multi-mode feedback strategy library, driving the graphic rendering, audio playing and control instructions of the touch vibration device of the mobile terminal, performing interactive feedback of audio-visual touch coordination, and outputting correction and guide information to a user.

Description

Infant care AR practical training system for home carers and skill assessment method Technical Field The invention relates to the technical field of augmented reality, in particular to an infant care AR practical training system and a skill assessment method for home carers. Background The infant scientific care is an important guarantee for the early development of children and is a skill with high professional and practical requirements. At present, a significant gap still exists in infant care skill training resources for home carers, and the practical requirements of vast families cannot be effectively met. Most families generally hope to obtain scientific child-care guidance, but the existing service system is obviously insufficient in digital and intelligent support, and large-scale coverage is difficult to realize. Meanwhile, the market professional care service is often polluted due to high cost and different quality, and the necessity of home autonomous learning and training is further highlighted. The current common training mode mainly comprises three types of high-simulation professional simulation equipment, off-line centralized training and traditional theory teaching. The high simulation equipment can provide a professional training environment, but depends on expensive special hardware, is not suitable for popularization in families, faces the problems of insufficient teaching materials, obvious space-time limitation and the like in offline training, is difficult to develop standardized and repeatable training, and is convenient to propagate in traditional image-text or video teaching, but lacks an actual operation environment and a real-time feedback mechanism, and cannot form effective muscle memory and skill consolidation. In general, the existing scheme is difficult to meet the requirements of cost controllability, operation reality and intelligent evaluation in a home scene. In recent years, augmented reality and artificial intelligence technologies have advanced in the fields of education and medical training, but in the field of practical training for infant home care, there is still a lack of systematic, low-threshold solutions with real-time interaction and intelligent assessment capabilities. Therefore, developing an AR practical training system that can utilize a common intelligent device, operate in conjunction with real props, and provide standardized real-time feedback is a common need for current technological development and home care capability improvement. Disclosure of Invention In order to solve the technical problems, the invention provides the infant care AR practical training system and the skill assessment method for the home carers, which effectively overcome the defects of high cost, split feeling of practical operation and lack of intelligent feedback in the traditional training, realize the practical training of the infant care household skills with high immersion and assessment on the popular mobile equipment, and are beneficial to improving the operation standardization and the correspondence confidence of the home carers. The first aspect of the invention provides an infant care AR practical training system for home caregivers, which comprises: the virtual-real fusion guiding module is used for creating and managing an augmented reality environment, superposing virtual guide information in a real operation scene and providing a virtual infant model corresponding to the physical prop; The visual perception module is used for capturing the hand actions and the physical prop states of the user in real time through the camera and outputting the key point positions and the motion track data; The intelligent evaluation and interaction feedback module is used for evaluating the data output by the visual perception module in real time according to a preset nursing operation rule and scheduling multi-mode feedback based on an evaluation result; the virtual-real fusion guide module, the visual perception module and the intelligent evaluation and interaction feedback module are integrated on the mobile intelligent terminal; the output of the visual perception module is imported to an intelligent evaluation and interaction feedback module; And the output of the intelligent evaluation and interaction feedback module is fed back to the virtual-real fusion guide module and the mobile intelligent terminal. In this scheme, the virtual-real fusion guide module includes: invoking an augmented reality frame of the mobile terminal, performing instant positioning and map construction on a real operation environment where a user is located through a camera, and identifying and locking a horizontal reference surface for placing entity props; Detecting a solid baby doll positioned on the horizontal reference plane, determining the position and the gesture of the solid baby doll, and registering and rendering a three-dimensional virtual baby model matched with the gesture and the size o