Search

CN-115481537-B - Handheld tool applicability verification method and system based on augmented reality technology

CN115481537BCN 115481537 BCN115481537 BCN 115481537BCN-115481537-B

Abstract

The invention discloses a handheld tool applicability verification method and system based on an augmented reality technology, and belongs to the field of handheld tools. The invention specifically and systematically constructs the scoring index of the object to be identified according to the existing three-dimensional real-time tracking registration result and experimental effect analysis, identifies the object to be identified by a natural characteristic point matching method, superimposes a digital model of the product to be detected on a verification environment, realizes virtual-real fusion and synchronous operation of a virtual model and a physical model of the handheld tool, and realizes the applicability verification of the handheld tool based on an augmented reality technology. The invention can improve the user interactivity, reduce the production cost, improve the production quality and shorten the production period. In addition, when the applicability verification environment of the handheld tool is constructed, the method for compressing the stl model is utilized, so that the number of vertices of the product digital model to be detected and the virtual model information of the handheld tool after compression is obviously reduced, the storage amount occupied by the product digital model to be detected and the virtual model information of the handheld tool in AR equipment is saved, and the verification efficiency is further improved.

Inventors

  • YANG XIAONAN
  • FANG HAONAN
  • HU YAOGUANG
  • Tang Boliang
  • Mao Wanting

Assignees

  • 北京理工大学

Dates

Publication Date
20260505
Application Date
20220921

Claims (8)

  1. 1. The handheld tool applicability verification method based on the augmented reality technology is characterized by comprising the following steps of: When the handheld tool applicability verification environment is built, the number of vertexes of the product digital model to be detected and the handheld tool virtual model information after compression is greatly reduced by using a stl model compression processing method, so that the importing time of the product digital model to be detected and the handheld tool virtual model is shortened, and the memory space occupied by the product digital model to be detected and the handheld tool virtual model in AR equipment is saved; Step two, constructing scoring indexes of objects to be identified, selecting objects with higher scores, identifying the objects to be identified with higher scores by a natural characteristic point matching method in a three-dimensional real-time tracking registration module, and superposing a digital model of a product to be detected in a verification environment; Thirdly, carrying out virtual-real fusion and synchronous operation of the handheld tool virtual model and the human hand in a verification environment; step four, adding box type collision body attribute to the handheld tool virtual model, and reducing the load of the AR equipment while guaranteeing the collision accuracy; The box type collision body attribute is added to the virtual model of the handheld tool, the virtual model is in a cuboid state, the length, the width and the height of the cuboid are equal to the maximum value of the whole handheld tool in three dimensions, and the load of hardware can be reduced while the collision accuracy is ensured; If the demand of the collision detection precision of the digital model of the product to be detected is higher, virtual grid points of the digital model of the product to be detected are constructed according to the characteristics of the digital model of the product to be detected, virtual grid points fitting the surface of the digital model of the product to be detected are generated based on a graphics algorithm, grid point layout information is corrected by fusing the information of the digital model of the product to be detected on the basis of visual perception information of AR equipment, and the construction precision and the perception precision of the collision body of the digital model of the product to be detected are improved; And step six, adopting a trigger detection method to perform collision detection on the virtual grid of the digital model of the product to be detected obtained in the step five and the box-type collision body of the virtual model of the handheld tool obtained in the step four, and further realizing the suitability verification of the handheld tool based on the augmented reality technology.
  2. 2. The method for verifying the applicability of a handheld tool based on an augmented reality technology as set forth in claim 1, wherein the first implementation method is, 1.1, Carrying out light weight treatment on stl models of a digital model of a product to be detected and a handheld tool virtual model, wherein a physical engine does not support direct import of stl format files, so that a single stl format model is required to be split into a plurality of objects, vertexes with the same positions are removed, and then the stl format model is imported into the physical engine; step 1.1.1, reading all vertex information of the stl model, and dividing the vertices into a plurality of sub-networks according to a certain number; Step 1.1.2, screening out vertexes with the same positions from each sub-network, setting the vertexes as the same vertex, and storing information of only one vertex to realize the lightweight treatment of the model; Reading vertex information of each model triangular surface one by one, establishing an array for storing vertexes and an array for storing triangular surfaces, judging whether the vertex positions of each surface are the same or not, if so, only storing information of one vertex, and not adding information in the vertex array like any more, but adding repeated triangular surfaces in the triangular surface array; step 1.2, importing the mixed reality tool kit MRTK into a physical engine to realize cross-platform AR application development, and realizing rotation, grabbing and scaling operations on objects through MRTK.
  3. 3. The method for verifying the applicability of the handheld tool based on the augmented reality technology as set forth in claim 2, wherein the second implementation method is as follows, Firstly, modifying the format and the size of a recognized picture, secondly, importing the picture into a cloud platform for recognition, after importing the picture into the platform, scoring the uploaded picture by the platform, taking the content richness, the contrast and the physical size of the image as scoring standards, and selecting the picture scored above four stars as the recognized picture; Step 2.2, the server carries out gray processing on the uploaded picture, the picture is changed into a black-and-white image, the characteristic points of the black-and-white image are extracted based on a natural characteristic point matching method, And 2.3, identifying the object to be identified, and superposing the predefined digital model in the physical engine on the verification environment to realize superposition of the digital model of the product to be detected on the verification environment.
  4. 4. The method for verifying the applicability of a handheld tool based on the augmented reality technology as set forth in claim 3, wherein in the third step, Selecting an implementation method according to the existence of a handheld tool entity model, wherein the virtual-real fusion and synchronous operation of the handheld tool virtual model and the human hand comprise the following two implementation methods; the method comprises the steps that when a handheld tool entity model exists, the natural characteristic point matching method in the three-dimensional real-time tracking registration module in the second step is repeated, objects to be identified are identified, the handheld tool virtual model is overlapped on the handheld tool entity model, and virtual-real fusion and synchronous operation of the handheld tool virtual model and the entity model are achieved; When the hand-held tool virtual model is not available, constructing a 'grip pair' condition according to the characteristics of a grip physical process of the hand-held tool physical model and an augmented reality environment, constructing a grip intention recognition algorithm based on the 'grip pair', judging the grip condition between the human hand and the hand-held tool virtual model, and realizing virtual-real fusion and synchronous operation of the hand-held tool virtual model and the human hand; The gripping pair consists of two contact points, and if more than one gripping pair exists, the gripped virtual model is judged to be in a gripping state; Step 3.1, acquiring images of human hands of a current frame, determining position and posture data of key nodes of the human hands relative to AR equipment based on a convolutional neural network two-hand pose estimation algorithm, and superposing a virtual hand model at the key nodes of the two hands identified by the current frame; the virtual hand model consists of a plurality of virtual joint models, wherein each virtual joint model is a cylinder and approximately simulates finger joints of a real double hand, a topological relation exists between the virtual hand joint models, namely, a high-level virtual joint model comprises a low-level virtual joint model, when the high-level virtual joint model moves, the motion of the low-level virtual joint model is driven, and the virtual hand model is represented by the following parameterization: Wherein, the For the ith virtual joint model, For the position of the virtual joint model, a group of vectors in an augmented reality environment coordinate system , And The representation is made of a combination of a first and a second color, For the pose of the virtual joint model, a group of vectors in an augmented reality environment coordinate system , And The representation is made of a combination of a first and a second color, Is a parameter of the virtual joint model, Indicating the length of the cylinder and, The diameter of the cylinder is indicated, Representing the driven sub-joint model of the virtual joint model, Representing a kth virtual joint model; Each virtual joint model in the virtual hand model corresponds to a key node identified by gesture tracking, the position and gesture data of each identified hand key node are used for updating the position and gesture of the virtual joint model of the current frame, the position and gesture of the virtual hand model are determined according to the position and gesture of the key node, and mapping of the real hands in a virtual space is realized, wherein the formula is as follows: Wherein, the For the position vector of the ith virtual joint model, Is a rotation matrix of the virtual joint model, the conversion relation between the rotation matrix and Euler angles is shown as a formula, wherein Representing rotation about the z-axis A degree; The transformation matrix is used for representing an augmented reality environment coordinate system and a camera coordinate system of the virtual joint model; is the position vector of the key node corresponding to the ith virtual joint model, A rotation matrix of a key node corresponding to the ith virtual joint model; Step 3.2, based on a virtual joint model of the human hand, combining a collision detection algorithm to determine whether the human hand is in contact with a handheld tool virtual model, if the collision detection algorithm detects that the human hand is in contact with the handheld tool virtual model, calculating whether a 'grip pair' can be formed between a plurality of contact points of the human hand and the handheld tool virtual model according to a grip intention recognition algorithm, judging whether a grip condition exists between the human hand and the handheld tool virtual model, if the human hand is in contact with different fingers, determining that the gripping is successful, converting into a gripping state, and realizing virtual-real fusion and synchronous operation of the handheld tool virtual model and the human hand; The grasping pair consists of two contact points, if more than one grasping pair exists, the grasped virtual model is judged to be in a grasping state, and whether grasping is completed or not is judged without contact calculation based on a plurality of contact points, so that grasping intention judgment is more flexible, the grasping intention is more approximate to a real three-dimensional gesture operation condition, a complex gesture interaction scene is more suitable, visual interaction feeling of a user is more met, and meanwhile, if a plurality of pairs of grasping pairs exist, a plurality of contact points forming the grasping pair participate in interaction intention recognition, and robustness, flexibility, efficiency and immersion feeling of gesture interaction intention recognition are improved; the physical process of grasping the real object is characterized in that the basic rule of Newton rigid body mechanics is applied to judge whether the object can be grasped by grasping whether the object is stressed and the friction force between the contact surfaces of the virtual hand model and the manipulated model, and the implementation principle is that a simplified coulomb friction force model is utilized to analyze the stressed state of the object; The said "grip pair" is composed of two contact points of virtual hand model and gripped model, and is characterized by that the angle between the connecting line of two contact points and normal line of contact surface is not greater than a fixed angle The two contact points will form a stable grip pair g (a, b), the fixed angle Namely a friction angle; The grasping intention recognition algorithm is established according to the condition of a grasping pair, and circularly judges whether all the current contact points and the other contact point form a pair of grasping pairs, wherein for any two contact points a and b of a virtual hand and a virtual object in one-time circular judgment, the angle between the connecting line of the two contact points and the normal line of the contact surface of each contact point is not more than a fixed angle Said two contact points will form a stable gripping pair g (a, b), this fixed angle I.e. the friction angle, i.e. the grip co-g (a, b) should be such that Wherein, the And The normal vector is the normal vector of the contact point a and the contact point b, and is the normal vector of the cylindrical surface of the joint virtual model at the contact point; A connecting line for the contact points a and b; for the friction angle, the value of the friction angle needs to be tested for the specific manipulated model to meet the stable, natural grip of the virtual component.
  5. 5. The method for verifying the applicability of the handheld tool based on the augmented reality technology as set forth in claim 4, wherein the fifth implementation method is as follows, Step 5.1, selecting a method for constructing a collision body of the digital model of the product to be detected according to the characteristics and the accuracy requirements of the digital model of the product to be detected and the verification environment; Step 5.2, if the requirement on the collision detection precision of the digital model of the product to be detected is low, fitting the digital model of the product to be detected by adopting a grid collision body, and respectively adding collision body attributes to split sub-models to obtain the collision body of the digital model of the product to be detected, so as to improve the collision detection precision; If the collision detection precision requirement of the digital model of the product to be detected is high, constructing virtual grid points of the digital model of the product to be detected according to the characteristics of the digital model of the product to be detected, manually arranging a grid top point set, generating virtual grids fitting the surface of the digital model of the product to be detected based on a graphic algorithm, fusing the digital model information of the product to be detected on the basis of the visual perception information of AR equipment to correct the layout information of the grid points, and improving the construction precision and the perception precision of a collision body of the digital model of the product to be detected; The method comprises the steps of fitting a virtual grid on the surface of a digital model of a product to be detected to obtain a triangular vertex index set, wherein the triangular processing is to decompose a polygon into a plurality of triangles, the polygons are formed by the vertexes, the triangular vertex index set is established according to an adjacency principle because a manually defined vertex sequence is a required direct modeling sequence, then generating a triangle combination through a graphics algorithm to generate the virtual grid adapting to the surface of the digital model of the product to be detected, and the grid vertex set and the triangular vertex index set are represented by the following parameterizations: (8) (9) Wherein the method comprises the steps of Representing a set of virtual grid vertices, V i representing the ith grid point; representing the ith triangle forming the virtual grid, ("a") ) Representing the three vertex index sets of this triangle.
  6. 6. The method for verifying the applicability of a handheld tool based on an augmented reality technology as set forth in claim 5, wherein in step six, The mathematical principle of the trigger detection algorithm is shown by the following formula: (10) 、 A collection of points on a collision body (a product to be detected digital model virtual grid, a handheld tool virtual model box-type collision body), 、 Is that 、 Is a dot in (2); (11) formula (11) is referred to as the Minkowski difference, which indicates when a collision body 、 When overlapping or intersecting, the difference set { The method comprises the steps of determining whether an origin is included, and triggering and detecting a product digital model virtual grid to be detected and a handheld tool virtual model box type collision body, namely judging whether a difference set of a product digital model virtual grid point set to be detected and a handheld tool virtual model box type collision body point set includes origin to verify whether interference occurs or not, wherein the occurrence of collision is indicated by the inclusion of the origin.
  7. 7. The method for verifying the applicability of a handheld tool based on an augmented reality technology as set forth in claim 6, wherein the physical engine is a Unity3D physical engine for achieving a more accurate verification effect.
  8. 8. The handheld tool applicability verification system based on the augmented reality technology is used for realizing the handheld tool applicability verification method based on the augmented reality technology as set forth in claims 1,2, 3, 4, 5, 6 or 7, and is characterized by comprising a data acquisition and processing module, a handheld tool applicability verification environment construction module, a three-dimensional real-time tracking registration module and a collision detection module; The data acquisition and processing module is used for acquiring RGB images and depth images of the current frame, acquiring natural characteristic point information of an object to be identified according to the RGB images based on a natural characteristic point matching method in the three-dimensional real-time tracking registration module, and acquiring position and posture information of hand key nodes of the current frame according to the RGB images and the depth images based on a two-hand posture estimation algorithm of a convolutional neural network; The handheld tool applicability verification environment construction module is used for constructing a handheld tool applicability verification environment based on the augmented reality equipment and the physical engine, wherein the handheld tool applicability verification environment comprises a to-be-detected product digital model and a handheld tool virtual model; The three-dimensional real-time tracking registration module comprises a product digital model superposition sub-module to be detected and a handheld tool virtual model superposition sub-module; the hand-held tool virtual model superposition sub-module is used for carrying out virtual-real fusion and synchronous operation of a hand-held tool virtual model and a human hand in the verification environment, selecting an implementation method according to the existence of the hand-held tool physical model, and when the hand-held tool physical model exists, carrying out recognition on the object to be recognized by the natural characteristic point matching method to realize superposition of the hand-held tool virtual model on the hand-held tool physical model; The collision detection module comprises a handheld tool virtual model collision body construction submodule, a to-be-detected product digital model collision body construction submodule and a trigger detection submodule, wherein the handheld tool virtual model collision body construction submodule is used for adding box-type collision body attributes to a handheld tool, presenting a cuboid state, enabling the length, the width and the height of the cuboid to be equal to the maximum value of the whole handheld tool in three dimensions, reducing the load of hardware while guaranteeing the collision accuracy, constructing a to-be-detected product digital model collision body virtual grid according to the characteristics and the accuracy requirements of a to-be-detected product digital model and verification environment, fitting the to-be-detected product digital model with the grid collision body, enabling the to-be-detected product digital model to be fitted with the grid collision body if the to-be-detected product digital model collision detection accuracy requirements are low, enabling the collision body attributes to be respectively added to be obtained by adopting a splitting submodule, improving the collision detection accuracy, constructing a to-be-detected product digital model virtual grid point according to the characteristics of the to-be-detected product digital model collision detection accuracy requirements, generating a virtual grid fitting the surface of the to-be-detected product digital model on the basis of the visual equipment, enabling the to be-detected product digital model to be-detected to be based on the graphics algorithm, and enabling the virtual grid point to be applied to the virtual model to be detected to be subjected to the actual, and the virtual model to be detected to be based on the virtual model to be detected to realize the actual, and the actual model to be applied to the digital model, and the device to be applied to the digital model detection model.

Description

Handheld tool applicability verification method and system based on augmented reality technology Technical Field The invention relates to a handheld tool applicability verification method and system based on an augmented reality technology, and belongs to the field of handheld tools. Background With the development of industry 4.0, the manufacturing industry needs new technologies to reduce the time and cost of product development. The augmented reality technology is the hottest and foremost technology in the information technology field, and has wide development prospect in industry, for example, in the assembly links of large-scale industrial products such as airplanes, automobiles and the like. The assembly procedure is used as a ring which is most important and time-consuming in the production link of mechanical products, and often faces the problems of higher skill threshold, higher cost of training staff, excessively complex assembly steps, various parts required by assembly, high similarity and the like, so that assembly workers with high technology, strong capability and skilled operation are required, and the problem can be solved by applying AR technology. The traditional process verification technology is complicated and is easy to be interfered by external factors, such as tools and tools, parts are in shortage, working environments and the like, and the steps of BOM assembly, assembly follow-up, signature follow-up of process files and the like are needed, while the AR technology can greatly simplify the complicated operation steps, the traditional paper BOM is replaced by an informationized electronic list, and the training cost of assembly workers is reduced by a virtual environment with high reduction degree and a reasonable man-machine interaction method. The assembly work of parts such as cables, an environmental control system, a hydraulic system, large parts and the like of manufacturing enterprises such as an airplane is a task completed in the final assembly stage of the airplane, and the airplane has the characteristics of high assembly quality requirement and less production quantity, so that the assembly work of most parts and components in the final assembly process of the airplane is completed manually by workers. Therefore, in the product verification link, the analysis of the applicability of the handheld tool is particularly important. Through the augmented reality technology, a manufacturer can quickly manufacture a virtual prototype of a product and verify the feasibility of the virtual prototype to obtain quick and direct design feedback, the defects of a process scheme or the product design scheme are found in advance, the rationality of the product design scheme and the feasibility of the process scheme are greatly improved, the research and development cost of enterprises is greatly reduced while the time is saved, in addition, compared with the virtual reality technology, the virtual reality technology is characterized in that the virtual reality technology is integrated, the virtual reality technology can restore the production scene more truly and more directly, the most realistic operation feedback is given to an operator while the cost is saved, and the virtual reality integration effect is realized by virtue of a high-precision collision detection technology, a high-precision three-dimensional tracking registration technology and a high-reduction virtual scene. Disclosure of Invention Aiming at the problems that the production quality is reduced due to unreasonable design and change of working environment in the traditional manufacturing process, virtual simulation modeling difficulty is high, human modeling is unreal, system interactivity is poor and the like in the existing tool applicability analysis process, the main purpose of the invention is to provide a handheld tool applicability verification method and system based on an augmented reality technology, to construct a handheld tool applicability verification environment based on augmented reality equipment and a physical engine, to construct a scoring index of an object to be identified, to select an object with higher scoring, to identify the object to be identified through a natural characteristic point matching method in a three-dimensional real-time tracking registration module, to superimpose a digital model of the product to be detected on the verification environment, to realize virtual-real fusion and synchronous operation of a handheld tool virtual model and a physical model, and to realize handheld tool applicability verification based on the augmented reality technology. The invention can improve the user interactivity, reduce the production cost, improve the production quality and shorten the production quality. In addition, when the applicability verification environment of the handheld tool is constructed, the method for compressing the stl model is utilized, so that the number of vertexes of th