CN-121981704-A - AR (augmented reality) glasses and large model-based auxiliary equipment maintenance method and system

CN121981704ACN 121981704 ACN121981704 ACN 121981704ACN-121981704-A

Abstract

The invention discloses an equipment auxiliary maintenance method and system based on AR glasses and a large model, wherein the equipment auxiliary maintenance method and system comprises the steps of collecting dialogue information, visual image information and equipment maintenance information which are input, constructing dialogue context data, visual context data and equipment maintenance context data based on historical maintenance data, carrying out feature fusion on the dialogue context data, the visual context data and the equipment maintenance context data to obtain multi-mode context fusion feature vectors, inputting the multi-mode context fusion feature vectors into the large model for training to obtain a multi-mode perception maintenance guidance large model, carrying out recognition and positioning of current maintenance equipment and maintenance steps according to the latest input information comprising voice information and visual image information, and calling corresponding maintenance equipment models and maintenance steps from an equipment maintenance database to carry out AR superimposed visual display based on recognition and positioning results of the current maintenance equipment and the maintenance steps.

Inventors

LIN FENGXU
ZHANG FAN
CHEN LIANGLIANG
WU WEIHAO
CHEN CHEN
XIN YUFENG

Assignees

杭州中港科技有限公司

Dates

Publication Date: 20260505
Application Date: 20260122

Claims (10)

1. An AR glasses and large model based equipment auxiliary maintenance method, comprising: Collecting dialogue information, visual image information and equipment maintenance information which are input, and constructing dialogue context data, visual context data and equipment maintenance context data based on historical maintenance data; feature fusion is carried out on the dialogue context data, the visual context data and the equipment maintenance context data to obtain a multi-mode context fusion feature vector; inputting the multi-mode context fusion characteristics into a large model for training to obtain a multi-mode perceived maintenance guidance large model, and identifying and positioning current maintenance equipment and maintenance steps according to the latest input information comprising voice information and visual image information; And based on the identification and positioning results of the current maintenance equipment and maintenance steps, calling corresponding maintenance equipment models and maintenance steps from an equipment maintenance database to perform AR superimposed visual display.
2. The AR glasses and large model based equipment auxiliary maintenance method according to claim 1, wherein the dialogue context data construction method comprises constructing an input text sequence of a dialogue based on time sequence, wherein the input text sequence of the dialogue is: wherein Input text representing a t-th sequence; obtaining system response data of each dialogue based on the dialogue input text sequence Wherein Input text representing the t-th text sequence Corresponding system response data; Encoding the dialogue input text and the system response data by using an encoding function to obtain the following dialogue context data: wherein Historical response data representing the last time the text was entered for the corresponding dialog, The dialog coding function is indicated and the superscript D indicates the dialog token.
3. The AR glasses and large model based equipment auxiliary maintenance method according to claim 1, wherein the visual context data construction method comprises the steps of acquiring an image sequence of a camera for equipment Wherein The length of the historical time window is indicated, Performing equipment component recognition on each image sequence by using a component detection model to obtain the following component recognition result , Wherein Wherein Representing the recognition result of the ith component of the corresponding device at the corresponding point in time t, wherein A category label representing the i-th component, A bounding box representing the i-th component, Indicating the confidence in the detection of the ith component.
4. The AR glasses and large model based equipment auxiliary maintenance method according to claim 1, wherein the visual context data construction method comprises the steps of obtaining historical gazing behaviors of a user on an equipment image sequence and obtaining corresponding historical gazing point data And acquiring a device part identification result based on the historical gaze point by adopting an image identification model Further calculate the component recognition result And device component recognition results based on historical gaze points Cross-over ratio between Further, the following visual context data was obtained The superscript V refers to the device visual image label.
5. The AR glasses and large model-based equipment auxiliary maintenance method according to claim 1, wherein the equipment maintenance context data construction method comprises the steps of scanning and acquiring a unique identifier ID of equipment, acquiring equipment maintenance parameters through the unique identifier, and inquiring a maintenance database, wherein the maintenance parameters comprise a historical maintenance record H (ID), maintenance technical parameters T (ID), an equipment component set P (ID) and a current fault state F t (ID), and obtaining the following equipment maintenance context data Wherein Representing the results of the equipment servicing database search, The superscript E of (a) refers to the equipment servicing mark.
6. The method for auxiliary maintenance of equipment based on AR glasses and large models according to claim 1, wherein said method for constructing context fusion feature vectors comprises the steps of respectively generating said dialogue context data Device maintenance context data And visual context data Unified coding is carried out to obtain the following feature vectors, namely the dialogue context feature vectors Device maintenance context feature vector And visual context feature vectors 。
7. The AR glasses and large model based equipment auxiliary maintenance method according to claim 1, wherein the context fusion feature vector construction method comprises constructing a query vector based on an input dialogue text: wherein The query vector is represented as a result of which, Representing the query transformation weight matrix, The offset vector representing the query is presented, Representing uniformly coded dialogue context feature vectors, further constructing key transformation weight matrix and value transformation weight matrix into key value pairs by using feature vectors of different contexts, calculating attention weights according to different context sources, and calculating final context fusion feature vector according to the attention weights : Wherein The context source type is indicated and the context source type is indicated, Represents the attention weight of the corresponding context source i, The key corresponding to context source i.
8. The AR glasses and large model-based equipment auxiliary maintenance method according to claim 1, wherein the AR overlapping visual display method comprises the steps of detecting key points of visual image frames through a cv small model, calculating correction parameters, projecting corresponding 3D information in a maintenance database onto equipment according to correction parameter transformation, and displaying text information remarks on a corresponding display interface.
9. An AR glasses and large model based equipment auxiliary maintenance system, characterized in that the system performs an AR glasses and large model based equipment auxiliary maintenance method according to any one of claims 1-8.
10. A computer readable storage medium, characterized in that it stores a computer program that is executed by a processor to implement an AR glasses and large model based device auxiliary maintenance method according to any one of claims 1-8.

Description

AR (augmented reality) glasses and large model-based auxiliary equipment maintenance method and system Technical Field The invention relates to the technical field of maintenance, in particular to an equipment auxiliary maintenance method and system based on AR (augmented reality) glasses and a large model Background The auxiliary maintenance method in the prior art is mainly realized by a maintenance staff frequently carrying out on-site inquiry of a maintenance manual and the like based on own experience. The prior art has the technical problems that 1, on-site inquiry is time-consuming and labor-consuming, and depending on the level of a maintainer, larger deviation possibly occurs to a new-hand inquiry result, so that maintenance failure is easy to occur, 2, when a related maintenance model is simply called based on-site inquiry, complete monitoring and reasoning of a maintenance process cannot be solved, so that guidance on the maintenance process is still based on personal experience, and thus, a maintenance guidance effect is poor, 3, the prior art cannot accurately record maintenance records of target equipment with a context relation, so that the guidance effect based on continuity of the maintenance process is poor, and in the prior art, after maintenance is stopped halfway, the completed maintenance step is required to be browsed again, so that the current maintenance step can be accurately positioned. Disclosure of Invention One of the purposes of the invention is to provide an equipment auxiliary maintenance method and system based on AR glasses and a large model, wherein the method and the system provide multi-mode context data as a fused core feature to conduct large model reasoning and are used for accurately positioning a current part to be maintained and a maintenance step where the part to be maintained is located in a maintenance process, so that the invention can realize management and control and guidance of a full-step chain in the maintenance process based on the large model, thereby greatly improving maintenance efficiency and greatly reducing maintenance threshold. The invention further aims to provide an AR glasses and large model-based equipment auxiliary maintenance method and system, and the method and the system provide multi-mode fusion characteristics constructed based on equipment maintenance context, visual context and voice context as reasoning data input by a large model, so that the invention can obtain the characteristics associated with the visual fixation characteristics and the equipment maintenance context according to the voice input, and further can accurately identify the positioning before and after the maintenance step of equipment corresponding to the current fixation position and the automatic output of the maintenance parameters and steps of subsequent equipment. The invention further aims to provide an equipment auxiliary maintenance method and system based on AR glasses and a large model, wherein the method and the system are based on understanding of the large model on multi-mode contexts, so that a current input sentence of a maintenance person can be accurately identified to a target prompt word, and based on the understanding capability of the large model, the input sentence of the maintenance person can be enabled to identify a core prompt word with a context without a very accurate keyword, and the core prompt word based on the understanding of the context is obtained to carry out visual presentation of corresponding maintenance equipment and maintenance steps. Another object of the present invention is to provide a method and a system for auxiliary maintenance of equipment based on AR glasses and a large model, where the method and the system provide a visual model based on a specific maintenance step, and when the visual model of the specific equipment is found based on the output result of the large model for understanding the multi-modal context, and the visual model of the equipment corresponds to the specific maintenance step, the visual presentation is performed by means of AR superposition, so that the effect of maintenance guidance is greatly improved. In order to achieve at least one of the above objects, the present invention further provides an apparatus auxiliary maintenance method based on AR glasses and a large model, the method comprising: Collecting dialogue information, visual image information and equipment maintenance information which are input, and constructing dialogue context data, visual context data and equipment maintenance context data based on historical maintenance data; feature fusion is carried out on the dialogue context data, the visual context data and the equipment maintenance context data to obtain a multi-mode context fusion feature vector; inputting the multi-mode context fusion characteristics into a large model for training to obtain a multi-mode perceived maintenance guidance large model, and identifying and