Search

CN-116310231-B - Engineering equipment real-time interaction system and motion simulation method based on mixed reality

CN116310231BCN 116310231 BCN116310231 BCN 116310231BCN-116310231-B

Abstract

The invention belongs to the technical field of man-machine interaction, and discloses a real-time interaction system and a motion simulation method of engineering equipment based on mixed reality, wherein the real-time interaction system comprises mixed reality hardware, a control handle, a simulation control module, a space mapping and collision detection module and an information storage module; by establishing a complex mechanical constraint relation in engineering equipment operation simulation, introducing simple constraint in a physical engine to simulate the motion process of each component of the engineering equipment, the problem of considering both real-time performance and simulation effect of motion simulation is solved, and an operator can perform virtual operation on the engineering equipment in a scene in a mixed virtual environment with real immersion and interactivity to simulate the construction process, so that a platform-independent virtual interactive simulation method capable of meeting the interactivity and real-time performance is provided.

Inventors

  • DAI MINGYUAN
  • LU YILIN
  • WANG WEI

Assignees

  • 江苏徐工工程机械研究院有限公司
  • 燕山大学

Dates

Publication Date
20260505
Application Date
20220909

Claims (15)

  1. 1. The engineering equipment real-time interaction system based on mixed reality is characterized by comprising mixed reality hardware, a control handle, a simulation control module, a space mapping and collision detection module and an information storage module; the mixed reality hardware is respectively in communication connection with the simulation control module, the space mapping and collision detection module and is used for acquiring and displaying a three-dimensional model of engineering equipment, identifying and acquiring construction site environment information and acquiring operator position information; the control handle is in communication connection with the simulation control module and is used for transmitting the digital signal of operation to the simulation control module; The simulation control module is respectively in communication connection with the mixed reality hardware, the control handle, the space mapping and collision detection module and the information storage module, and is used for receiving virtual interaction information, analyzing the virtual interaction information into instruction information corresponding to the movement of the virtual three-dimensional model, and simulating the movement of the virtual three-dimensional model in real time according to the instruction; The space mapping and collision detection module is respectively in communication connection with the mixed reality hardware and the simulation control module, and is used for displaying the construction site environment information acquired by the mixed reality hardware and the virtual three-dimensional model in the simulation control module in a real construction site environment through a display screen of the mixed reality hardware in a virtual-real fusion mode, and simultaneously carrying out collision detection and real-time motion interference simulation between a virtual object and a real scene object; the information storage module is respectively in communication connection with the simulation control module, the space mapping and the collision detection module and is used for storing motion state information and collision detection information records during virtual training and virtual verification so as to realize the reproduction of training and verification results.
  2. 2. The real-time interactive system of engineering equipment based on mixed reality of claim 1, further comprising a server, wherein the server is a hardware carrier of the simulation control module, the space mapping and collision detection module and the information storage module.
  3. 3. The real-time interactive system of engineering equipment based on mixed reality of claim 1, wherein said simulation control module comprises an interactive instruction acquisition module, a dynamics simulation calculation module and a scene management and graphic image real-time rendering module; The interactive instruction acquisition module is used for receiving operation information of an operator wearing the mixed reality hardware, wherein the operation information is virtual interactive information, and the virtual interactive information comprises a handle operation digital signal from a control handle, handle operation of the mixed reality hardware, operator gestures, interactive instruction information of operator voices, operator position information of the operator wearing the mixed reality hardware and real construction site environment information; the dynamics simulation calculation module is used for carrying out real-time dynamics simulation calculation according to the operation information of an operator and outputting a simulation calculation result to the scene management and graphic image real-time rendering module; the scene management and graphic image real-time rendering module is used for updating the motion state of the three-dimensional model of the engineering equipment in the virtual scene in real time according to the simulation calculation result of the dynamic simulation calculation module, and outputting the operation result to the mixed reality hardware to be displayed to an operator.
  4. 4. The real-time interactive system of engineering equipment based on mixed reality of claim 1, wherein the control handle comprises a handle driver, the control handle transmits an operation digital signal to the simulation control module through the handle driver, and the handle driver analyzes an operation control flow input by the control handle into an operation digital signal and transmits the operation digital signal to the simulation control module.
  5. 5. A method for simulating the motion of engineering equipment based on mixed reality is characterized in that based on the real-time interaction system of claim 3, the dynamics simulation calculation module establishes corresponding kinematic constraints among rigid body components of the engineering equipment by carrying out multi-rigid body dynamics modeling on the engineering equipment, and the kinematic constraints comprise a multi-rigid body dynamics system and a motion constraint system.
  6. 6. The method for simulating the motion of engineering equipment based on mixed reality according to claim 5, wherein the method for simulating the motion comprises the following steps: Step one, configuring mixed reality hardware, a control handle and a server; step two, loading a virtual model to the mixed reality hardware; If the user operation input is correct, dynamics simulation calculation is carried out, and a simulation calculation result is output to a scene management and graphic image real-time rendering module, wherein the scene management and graphic image real-time rendering module updates the motion state of the three-dimensional model of the engineering equipment in the virtual scene in real time according to the simulation calculation result of the dynamics simulation calculation module, and outputs the operation result to the mixed reality hardware for display to an operator; Displaying the engineering equipment three-dimensional model in the real-time updated virtual scene in the step three in a real construction site environment, displaying in a virtual-real fusion mode, and simultaneously performing collision detection and real-time motion interference simulation between a virtual object and a real scene object; writing the operation process into an information storage module, and recording and storing the operation process information in the process to finish the task.
  7. 7. The method for simulating the motion of engineering equipment based on mixed reality according to claim 5, wherein said multi-rigid-body dynamics system comprises a multi-rigid-body system physical model and a multi-rigid-body system mathematical model, said multi-rigid-body system physical model is a physical model for expressing mechanical characteristics of the system by performing physical modeling on the engineering equipment geometric model, said physical modeling comprises the following steps: Step S1, assembling an engineering equipment geometric model according to kinematic constraint and an initial position condition of a virtual three-dimensional model, setting a parent-child nesting relationship between virtual three-dimensional model components by analyzing motion dependency relationship among the components of the virtual three-dimensional model, and finally completing construction of a whole engineering equipment model tree; step S2, regarding each geometric model member as a rigid member and setting a Cartesian generalized coordinate vector thereof Wherein N represents a multi-rigid-body system having n points, and any adjacent rigid bodies connected by hinges are regarded as a unit, namely n represents the number of units of the multi-rigid-body system; the position of another rigid body relative to the rigid body is described by generalized coordinates using the single rigid body as reference For the vector of the centroid of each rigid body member in the absolute coordinate system, x represents the x-axis in the absolute coordinate system, y represents the y-axis in the absolute coordinate system, z represents the z-axis in the absolute coordinate system, Three euler angles for the rigid body member relative to the coordinate base itself, The angle of precession is indicated as being indicative of the angle of precession, The nutation angle is indicated as being the angle of nutation, Representing the self-rotation angle, the configuration vector matrix of the whole multi-rigid-body dynamics system is as follows: (1); The expression of the kinematic constraint equation set of the whole multi-rigid-body dynamic system is that (2); In the formula (2), the amino acid sequence of the compound, The number of the constraint pairs is equal to the number of the constraint pairs, Representing the motion constraint equation set of the whole multi-rigid-body dynamics system as a whole, Representing the motion constraint equation for the corresponding member alone, wherein, V is a variable arbitrarily taken as a distinction from the driving constraint equation; When the total degree of freedom of the whole multi-rigid-body dynamic system is zero, the motion is determined, so that the expression of the driving constraint number equation required by the multi-rigid-body dynamic system is as follows: (3); Wherein, the For a driving constraint number equation, the driving constraint is a time function about generalized coordinates; the motion constraint equation is a vector form of a driving constraint equation, H is a variable which is taken at will and used for distinguishing the motion constraint equation from the motion constraint equation, t represents the motion time of a multi-rigid-body system; All constraints imposed by the multi-rigid-body dynamics system are combined by the kinematic constraint of the formula (2), the driving constraint of the formula (3) and the Euler parameter constraint: (4); The equation (4) forms n nonlinear position equation sets of the whole multi-rigid-body dynamic system under generalized coordinates; driving constraint equations for the whole system, wherein the driving constraint is a time function related to generalized coordinates; the expression is Euler parameter constraint equation; And S3, analyzing the engineering equipment geometric model, and establishing a corresponding constraint type for the geometric model component by using a physical engine according to the constraint equation type among the geometric model components to finish the physical modeling from the engineering equipment geometric model to the multi-rigid-body system physical model.
  8. 8. The method for simulating the motion of engineering equipment based on mixed reality of claim 7, wherein the mathematical model of the multi-rigid-body system is mathematical modeling performed after a physical model of the multi-rigid-body system is obtained, so that a mathematical model of the speed and the acceleration of the multi-rigid-body system is obtained, and a dynamic model of the multi-rigid-body system is obtained.
  9. 9. The method for simulating the motion of engineering equipment based on mixed reality according to claim 8, wherein said mathematical model of the multi-rigid-body system comprises a mathematical model of the motion of a rigid body member and a mathematical model of the rotation of the rigid body member.
  10. 10. The method for simulating the motion of engineering equipment based on mixed reality according to claim 9, wherein the modeling process of the rigid body component motion mathematical model comprises the following steps: Step A, deriving the formula (4), and then obtaining a multi-rigid-body system speed constraint equation: (5); the whole formula (5) is the result of deriving the formula (4), wherein For the Jacobian matrix of the system, i.e ; For constraining the time derivative of equation, i.e. , The generalized speed of the system is provided; and B, secondarily deriving the formula (4), wherein the acceleration constraint equation of the multi-rigid-body system is as follows: (6); In the middle of Is a Jacobian matrix; For the derivative of the Jacobian matrix with respect to time, i.e. ; Quadratic derivation of time for constraint equations, i.e. ; The generalized acceleration of the system is adopted; Step C, obtaining a first type of pull-type multiplier form of a multi-rigid-body system motion equation by a multi-rigid-body system position constraint equation of the formula (4), a multi-rigid-body system speed constraint equation of the formula (5) and a multi-rigid-body system acceleration constraint equation of the formula (6): (7); Wherein, the Respectively a system generalized coordinate, a speed and an acceleration vector, , , Is Lagrangian multiplier column vector, is the internal force and internal moment between rigid body components connected by kinematic pair, It is the time that is required for the device to be in contact with the substrate, Representing a quality matrix of the system, Is a generalized external force column vector, comprising external force and external moment, N is n-dimensional real number vector space, and the numerical value is consistent with the unit number of the multi-rigid body system; m is m-dimensional real number vector space, and the number of m is consistent with that of constraint pairs.
  11. 11. The method for simulating the movement of engineering equipment based on mixed reality as set forth in claim 9 or 10, wherein modeling of the rigid body member rotation mathematical model requires that each mechanism of the engineering equipment in the three-dimensional space adopts a satellite coordinate system fixedly connected to the rigid body member to determine the movement of a multi-rigid body system, and specifically, three gesture variables of a direction cosine matrix, euler angles and Euler quaternions are used for describing the rigid body to construct fixed-point rotation; The Euler angle is as follows: (8); the direction cosine matrix is as follows: (9); The Euler quaternion is: (10); The Euler quaternion is used for representing a direction cosine matrix as follows: (11); euler parameter variables in the direction cosine matrix and the Euler quaternion need to satisfy constraint equations: (12); wherein a 1 ,a 2 ,a 3 is a vector of Euler quaternions, a 4 is a Euler quaternion scalar, A constraint vector constituted by euler parameters of the rigid body member.
  12. 12. The method for simulating the movement of engineering equipment based on mixed reality according to claim 5, wherein the movement constraint system calculates Jacobian matrixes of two connected rigid body member objects according to constraint types, calculates moment of inertia in cooperation with the shapes of the rigid body members, and updates positions and speeds of the interconnected rigid body member objects so as to simulate the stress effect of the connected objects, and the constraint types among the rigid body members comprise hinge constraint and sliding rod constraint.
  13. 13. The method for simulating the movement of the engineering equipment based on mixed reality according to claim 12, wherein the implementation steps of the hinge constraint are as follows: step I: obtaining mass centers of a first rigid body component and a second rigid body component, and obtaining a mass center formula of the rigid body components: (13); Wherein, the Representing a rigid body member centroid position vector; 、 、 respectively representing the coordinates x, y and z of the mass center of the rigid body, wherein i, j and k respectively represent vector bases; Step II, calculating the position of an anchor point relative to the first rigid body component, and acquiring a hinge shaft of the first rigid body component; step III, calculating the position of an anchor point relative to the second rigid body component, and acquiring a hinge shaft of the second rigid body component; And IV, carrying out hinge constraint on the two rigid body components.
  14. 14. The method for simulating the movement of the engineering equipment based on mixed reality according to claim 13, wherein the implementation steps of the sliding rod constraint are as follows: step a, acquiring mass centers of the first rigid body component and the second rigid body component by utilizing a formula (13); calculating the position of the second rigid body component relative to the first rigid body component, and acquiring a sliding shaft of the first rigid body component; Step c, calculating the position of the first rigid body component relative to the second rigid body component, and obtaining the sliding shaft of the second rigid body component; And d, restraining the sliding rods of the two rigid body members.
  15. 15. The method for simulating the movement of engineering equipment based on mixed reality as set forth in claim 7, wherein the function in the physical engine can be successfully used only by adjusting the spatial posture or the constraint axis of the rigid body member through a movement constraint system established by the physical engine, and the adjusting method comprises the steps of applying a Rodrigues rotation formula around a unit vector Rotate and the rotation angle is The rotation matrix of (2) is: (14); After the spatial pose of the rigid member is acquired, the rigid member is adjusted by equation (14).

Description

Engineering equipment real-time interaction system and motion simulation method based on mixed reality Technical Field The invention belongs to the technical field of man-machine interaction, and particularly relates to a real-time interaction system and a motion simulation method of engineering equipment based on mixed reality. Background Engineering equipment such as high-altitude operation equipment, pump trucks, drilling trolleys and the like is complex in structure, high in control difficulty and high in danger coefficient, technical level requirements on operators are high, in order to reduce construction risks, improve working quality and efficiency, ensure construction safety, and simulate corresponding movement behaviors generated by engineering equipment control of operators, a simulation control simulation training system related to engineering machinery is used, interactivity cannot be met, mixed reality is a computer virtual technology enabling real world and virtual objects to be displayed and interacted in the same visual space, a brand-new visual environment is created, and rapid development and application are gradually provided in the fields of industry, education training, entertainment, medical treatment and the like. In order to make the development process independent of a software platform, the traditional method calculates the position coordinates of the movement of different parts of a working device through a conversion matrix and then displays the position coordinates, the calculation result is relatively accurate, the calculation is complex, the real-time requirement is difficult to meet, the prior art simulates the movement of the emergency engineering machinery based on a virtual reality modeling language, the movement control can be simplified through the setting of DOF nodes, and the development of a control interface corresponding to the platform is not perfect. However, in the field of virtual simulation control of engineering equipment, no technology can simultaneously meet the requirements of real-time construction flow verification and training on a construction site, real-time operation simulation of equipment motion and visual display in a virtual visualization mode. Disclosure of Invention Aiming at the defects of the prior art, the invention provides a real-time interaction system and a motion simulation method of engineering equipment based on mixed reality. In order to achieve the above purpose, the present invention provides the following technical solutions: In a first aspect, the invention provides a real-time interaction system of engineering equipment based on mixed reality, which comprises mixed reality hardware, a manipulation control handle, a simulation control module, a space mapping and collision detection module and an information storage module; the mixed reality hardware is respectively in communication connection with the simulation control module, the space mapping and collision detection module and is used for acquiring and displaying a three-dimensional model of engineering equipment, identifying and acquiring construction site environment information and acquiring operator position information; the control handle is in communication connection with the simulation control module and is used for transmitting an operating digital signal to the simulation control module through handle driving; the simulation control module is respectively in communication connection with the mixed reality hardware, the control handle, the space mapping and collision detection module and the information storage module, and is used for receiving virtual interaction information, analyzing the virtual interaction information into instruction information corresponding to the movement of the virtual three-dimensional model, and simulating the movement of the virtual three-dimensional model according to the instruction; The space mapping and collision detection module is respectively in communication connection with the mixed reality hardware and the simulation control module, and is used for displaying the construction site environment information acquired by the mixed reality hardware and the virtual three-dimensional model in the simulation control module in a real construction site environment through a display screen of the mixed reality hardware in a virtual-real fusion mode, and simultaneously carrying out accurate collision detection and real-time motion interference simulation between a virtual object and a real scene object; the information storage module is respectively in communication connection with the simulation control module, the space mapping and the collision detection module and is used for storing motion state information and collision detection information records during virtual training and virtual verification so as to realize the reproduction of training and verification results. With reference to the first aspect, the real-time interactive system