CN-121999659-A - Mine safety training system and method based on mixed reality and large space positioning

CN121999659ACN 121999659 ACN121999659 ACN 121999659ACN-121999659-A

Abstract

The invention provides a mine safety training system based on mixed reality and large space positioning, which belongs to the technical field of mine safety and comprises a cabin structure, a plurality of visual positioning markers, MR head equipment, a positioning module, a gesture interaction module, a scene fusion module and a training control module, wherein the visual positioning markers are arranged in the cabin structure and each visual positioning marker has a unique coding identifier, the MR head equipment comprises a high-precision camera and a display unit, the positioning module comprises an image processing unit, the image processing unit calculates the real-time position and the gesture of the MR head equipment in a virtual scene, the gesture interaction module recognizes the professional gesture in mine operation, high-precision large space positioning is realized, natural interaction of the professional gesture is supported, and virtual and real scenes can be dynamically fused according to a training flow, so that immersed, efficient and large-scale popularization of mine safety training experience is provided.

Inventors

CHEN PENGGANG
MENG HAOJIE
ZHANG GUILIANG
DAI WEI

Assignees

鞍山五矿陈台沟矿业有限公司
辽宁科技大学

Dates

Publication Date: 20260508
Application Date: 20260119

Claims (10)

1. The mine safety training system based on mixed reality and large space positioning is characterized by comprising a cabin structure, a plurality of visual positioning markers, MR head-mounted equipment, a positioning module, a gesture interaction module, a scene fusion module and a training control module; the cabin structure is used for providing a physical training space; a plurality of visual positioning markers are arranged in the cabin structure, and each visual positioning marker is provided with a unique coding identifier; the MR head-mounted device comprises a high-precision camera and a display unit, wherein the high-precision camera is used for acquiring an environment image containing a visual positioning marker; The positioning module comprises an image processing unit, wherein the image processing unit recognizes a visual positioning marker in an environment image through an image recognition technology and calculates the real-time position and the real-time gesture of the MR head equipment in a virtual scene; the gesture interaction module is used for identifying professional gestures in mine operation; the scene fusion module reserves a specified real image area in the real-time video captured by the high-precision camera, and fuses virtual teaching contents with the reserved real image area; The training control module is used for integrating and scheduling real-time position data of the positioning module, gesture recognition results of the gesture interaction module and rendering parameters of the scene fusion module so as to run a mine safety training process comprising large-space free movement, gesture interaction and virtual-real fusion scenes.
2. The mine safety training system based on mixed reality and large space localization of claim 1, wherein the visual localization marker is ArUco markers or AprilTag markers.
3. The mine safety training system based on mixed reality and large space positioning according to claim 1, wherein the positioning module is used for detecting and identifying visual positioning markers through OpenCV library functions integrated in a Unity engine, and calculating real-time positions and postures through algorithm solving PERSPECTIVE-n-Point problems.
4. The mine safety training system based on mixed reality and large space positioning according to claim 1, wherein the gesture recognition model in the gesture interaction module is a convolutional neural network model based on MEDIAPIPE frames and training optimization is performed by using a user-defined hand image dataset containing mine operation actions.
5. The mine safety training system based on mixed reality and large space positioning according to claim 1, wherein the scene fusion module is used for dynamically generating a rectangular mask in a rendering pipeline according to training procedure instructions, the rectangular mask acts on rendering textures of a high-precision camera so that real environment elements in a corresponding rectangular area are kept for display, and the outside of the area is covered by a virtual mine scene.
6. The mine safety training system based on mixed reality and large space positioning according to claim 1, wherein the training process provided by the training control module comprises mine entry preparation, crowbar process experience, high-altitude falling accident simulation and emergency accident simulation.
7. A mine safety training method based on mixed reality and large space positioning, based on the mine safety training system based on mixed reality and large space positioning as claimed in claims 1-6, characterized by comprising the following steps: s1, arranging visual positioning markers in a physical training space, and constructing a virtual scene model corresponding to a real mine 1:1; s2, a user wears the MR head equipment to enter a physical training space; S3, calculating the position and the gesture of the user in real time by identifying the visual positioning marker, so as to realize large-scale free walking and positioning in the virtual scene; S4, capturing and identifying operation gestures of a user on the virtual tool through the optimized gesture identification model; S5, reserving and fusing a real environment image and virtual teaching contents in a designated area through a rectangular mask dynamically generated by a scene fusion module; And S6, guiding the user to complete the immersed security training course of S3-S5.
8. The mine safety training method based on mixed reality and large space localization according to claim 7, wherein in step S3, visual localization markers are identified through OpenCV library functions integrated in a Unity engine, and real-time positions and attitudes are calculated by solving PERSPECTIVE-n-Point problems.
9. The mine safety training method based on mixed reality and large space localization according to claim 7, wherein in step S4, the gesture recognition model is a convolutional neural network model based on MEDIAPIPE frames and training optimization is performed by using a custom hand image dataset including mine operation actions.
10. The mine safety training method based on mixed reality and large space positioning according to claim 7, wherein step S5 specifically comprises dynamically generating a rectangular mask in a rendering pipeline according to training procedure instructions, applying the rectangular mask to reserve real environment images in corresponding rectangular areas in a camera picture, rendering virtual scenes outside the areas, and superposing virtual teaching contents on the reserved real images.

Description

Mine safety training system and method based on mixed reality and large space positioning Technical Field The invention belongs to the technical field of mine safety, and particularly relates to a mine safety training system based on mixed reality and large space positioning. Background The mine safety production is a basic stone for sustainable development of mining industry, and high-efficiency and vivid safety training is important for preventing accidents and improving the safety literacy of operators. With the development of technology, virtual Reality (VR), augmented Reality (AR) and Mixed Reality (MR) technologies are beginning to be introduced into the training field, but the limitations still exist that (one) most VR and MR systems rely on a laser base station or an inertial sensor built in an MR head-mounted device for positioning. The method has the advantages that the method is complex in deployment, high in cost and limited in moving range, positioning signals are easy to lose and accumulate and drift in a mine simulation environment with sparse characteristics and similar structures, so that a user can get stuck in a picture and have disordered positions when moving in a virtual scene, immersion sense and space cognition are seriously damaged, and real large-space free walking training cannot be realized. And secondly, the conventional training system mostly adopts the traditional peripherals such as a handle, a control rod and the like to interact with the actual freehand operation tool in the pit, such as a crowbar mode to disconnect. Even if preliminary gesture recognition is introduced, the algorithm is general and is not optimized for professional actions, so that the recognition accuracy of fine and continuous operations such as holding, prying, pressing and the like is low, the delay is high, the interaction feedback is distorted, and muscle memory and operation skills cannot be effectively trained. Third, in emergency simulation training, the existing AR and MR techniques tend to simply superimpose virtual information on a real frame, have poor fusion, or completely cover a real object with a virtual object. The method cannot intelligently reserve and utilize the key real operation area and props, so that virtual guidance and physical operation are cracked, a trainee is difficult to map a virtual instruction to a real action, and the conversion efficiency of training is reduced. Therefore, a mine safety training system capable of realizing stable and accurate large-space positioning, natural and real professional gesture interaction and intelligent and dynamic virtual and real scene depth fusion is needed. Disclosure of Invention Based on the technical problems, the invention aims to provide a mine safety training system and a method based on mixed reality and large space positioning, and the method can realize high-precision large space positioning, support natural interaction of professional gestures, and dynamically fuse virtual and real scenes according to training flows, so that immersive, efficient and large-scale popularization of mine safety training experience is provided. The specific technical scheme is as follows: A mine safety training system based on mixed reality and large space positioning comprises a cabin structure, a plurality of visual positioning markers, MR head-mounted equipment, a positioning module, a gesture interaction module, a scene fusion module and a training control module, wherein the cabin structure is used for providing a physical training space, the visual positioning markers are arranged in the cabin structure and each visual positioning marker is provided with a unique coding identifier, the MR head-mounted equipment comprises a high-precision camera and a display unit, the high-precision camera is used for collecting an environment image containing the visual positioning markers, the positioning module comprises an image processing unit, the image processing unit is used for recognizing the visual positioning markers in the environment image through an image recognition technology and calculating the real-time position and the gesture of the MR head-mounted equipment in a virtual scene, the gesture interaction module is used for recognizing professional gestures in mine operation, the scene fusion module is used for reserving a designated real image area in a real-time video captured by the high-precision camera and fusing virtual teaching content with the reserved real image area, and the training control module is used for integrating and scheduling real-time position data of the positioning module, recognition results of the gesture interaction module and gesture interaction parameters of the gesture interaction module in the large space running and free running and virtual gesture interaction and real-safety and training flow. In addition, the mine safety training system based on mixed reality and large space positioning in the technical schem