Search

CN-122024174-A - Escalator step detection method integrating YOLO algorithm and digital twin technology

CN122024174ACN 122024174 ACN122024174 ACN 122024174ACN-122024174-A

Abstract

The invention discloses an escalator step detection method integrating a YOLO algorithm and a digital twin technology, which comprises the steps of firstly collecting sensor and video stream data, constructing a step data set, then modifying YOLOv n, introducing a CCA attention mechanism, reconstructing an EL-Head detection Head, obtaining a lightweight TJ-YOLOv8 model through pruning and INT8 quantization, training, deploying the model to an edge end to realize step real-time detection, constructing a topological relation, marking suspected missing steps, generating abnormal metadata with three-dimensional coordinates, constructing an escalator digital twin body, realizing data injection, virtual-real synchronization and sensor calibration, fusing multi-source heterogeneous characteristics, generating a decision signal, and combining gradient accumulation and history backtracking to form a three-level filtering system to judge abnormality. The invention has high detection precision and quick response, realizes closed loop control of step missing, and improves the safety guarantee and the operation and maintenance intellectualization level of the escalator.

Inventors

  • LU CHENGLONG
  • QING GUANGWEI
  • NI MINMIN
  • WU XIAOYUE
  • WANG SHUANG
  • ZHOU QIANFEI
  • JIANG MING

Assignees

  • 南京市特种设备安全监督检验研究院

Dates

Publication Date
20260512
Application Date
20260410

Claims (9)

  1. 1. An escalator step detection method integrating a YOLO algorithm and a digital twin technology is characterized by comprising the following steps: step S1, synchronously acquiring and preprocessing multi-source heterogeneous data, namely acquiring escalator step operation sensing data and video information, preprocessing, and constructing step state labeling data based on the preprocessed data; Step S2, improving YOLOv n lightweight detection model construction and training: the method comprises the steps of taking YOLOv n as a basic framework, completing network transformation through main network re-parameterization transformation, neck network insertion RepCCA attention mechanism and detection Head reconstruction into an EL-Head structure, and realizing light weight through channel pruning and INT8 quantization, and then training and verifying a model after transformation by utilizing a data set constructed in the step S1 to obtain a TJ-YOLOv8 model meeting the requirement of step detection precision; Step 3, edge end real-time detection and abnormal metadata generation, namely deploying a TJ-YOLOv model to an edge computing node, performing step detection on an escalator step operation video stream which is collected and preprocessed in real time on site, outputting a detection result, dynamically constructing a step chain topology relationship, marking suspected missing steps and generating abnormal event metadata with step three-dimensional space coordinates; S4, dynamic bidirectional driving of a digital twin model, namely constructing an escalator digital twin body comprising a step three-dimensional model, a kinematic model and a physical parameter database, injecting sensor data acquired in real time and abnormal event metadata into the digital twin body in real time through a stream processing engine, driving a virtual model to synchronize with the physical entity state, and reversely calibrating sensor data drift by utilizing the digital twin body kinematic model to correct reference coordinates of a step topological relation; Step 5, multi-mode data fusion decision is carried out, namely a cross-domain feature fusion module is designed, three types of heterogeneous features of step height abrupt change quantity, confidence degree attenuation quantity of a TJ-YOLOv model and topological chain breaking length after digital twin body calibration are collected by a sensor, and an instant fusion decision signal is generated through normalization and weighted fusion; Then, the historical operation parameters of suspected abnormal steps are queried through digital twin body backtracking for verification, a three-level decision filtering system of instantaneous characteristics, short-term accumulation and long-term backtracking is formed, and whether the abnormality exists continuously or not is judged; step S6, millisecond alarm feedback and model self-updating: If the abnormal condition is verified to exist continuously, an emergency braking signal is sent to the escalator control system within 50ms, the missing position is highlighted on a human-computer interface, meanwhile, abnormal event related data are packed into training samples, incremental learning is conducted on the TJ-YOLOv model based on the federal learning framework, and a digital twin body synchronous generation simulation fault case library optimization topological chain fracture prediction algorithm is conducted.
  2. 2. The escalator step detection method based on the YOLO algorithm and the digital twin technology, according to claim 1, is characterized in that in the step S1, the escalator operation sensing data and video information under different conditions are collected through an infrared ranging sensor, a photoelectric sensor array and an industrial camera group which are arranged on an upper cabin and a lower cabin of the escalator; preprocessing comprises microsecond time alignment of sensor data and video streams through a hardware synchronization module, performing GPU (graphic processing unit) acceleration inter-frame differential processing on the video streams, extracting a step motion region and compressing resolution; Marking three types of labels, namely a normal step, a missing step and a shielding step, according to a YOLO format, marking a category ID, a boundary frame center point coordinate and a boundary frame width and height according to a YOLO standard format, and dividing a marked data set into a training set, a verification set and a test set according to a ratio of 8:1:1.
  3. 3. The escalator step detection method according to claim 1, wherein in step S2, specifically, the method comprises: step S2.1, finishing network transformation by taking YOLOv n as a basic framework, and specifically comprising the following steps: The main network is transformed, namely a C2f module basic structure of YOLOv n is reserved, and all 3X 3 standard convolutions in the module are replaced by RepVGG type reparameterized convolutions, wherein a three-branch parallel structure of 3X 3 convolutions, 1X 1 convolutions and residual connection is adopted in a training stage, and a reasoning stage is fused into a single 3X 3 convolution; inserting RepCCA attention mechanism module after YOLOv n neck SPPF module, wherein RepCCA attention mechanism firstly carries out global, height and width pooling treatment and weighted fusion on input characteristics through CCA module to obtain CCA characteristic diagram, and then introduces a three-branch parallel structure to carry out heavy parameterization optimization; The detection Head reconstruction comprises the steps of replacing YOLOv n original decoupling heads with EL-Head high-efficiency lightweight detection heads, combining two parallel 3X 3 convolutions in the original decoupling heads into a single 3X 3 convolution, introducing a lightweight substructure consisting of 1X 1 convolutions and RepCCA modules, wherein the 1X 1 convolutions are responsible for channel dimension reduction and information integration, and the RepCCA modules focus on space feature extraction; S2.2, performing light weight processing, namely performing channel pruning and INT8 quantization on the network model after transformation, and reducing the quantity and calculated quantity of model parameters; Step S2.3, training the modified network model, namely training the lightweight network model by utilizing a cascade state labeling data set and adopting a customized training strategy; And step S2.4, verifying the trained network model, namely, verifying the accuracy of the trained network model by taking the step missing detection mAP@0.5, the omission factor and the false detection rate as evaluation indexes, and determining the model as a TJ-YOLOv model if the accuracy meets a preset threshold.
  4. 4. The escalator step detection method according to claim 3, wherein in step S2, the CCA module processes the input feature map X CCA ∈R C×H×W , and specifically includes: 1) Splitting an input feature map X CCA according to channels, and respectively performing horizontal direction pooling, vertical direction pooling and global direction pooling on the features of a C-th channel to obtain corresponding feature tensors, wherein C is more than or equal to 0 and less than C, and specifically comprises the following steps: Horizontal pooling, namely traversing and summing the rows with the height fixed as h along the width direction and averaging to obtain a horizontal pooling characteristic tensor : ; Wherein H is a height direction position index, H < H is more than or equal to 0, i is a width direction traversal index, i is more than or equal to 0 and is less than or equal to W, x c (H, i) is a pixel characteristic value at the height H and width i of the c-th channel; pooling in the vertical direction, namely traversing and summing the columns with the width of w fixed along the height direction and averaging to obtain the pooling characteristic tensor in the vertical direction : ; Wherein W is a width direction position index, W is more than or equal to 0 and less than or equal to W < W, j is a height direction traversal index, and x c (j, W) is a pixel characteristic value at the height j and width W in the c-th channel; Global direction pooling, namely traversing and summing the whole feature space and averaging to obtain a global direction pooling feature tensor: ; z c (W, H) is global pooling characteristic value at width W and height H in the C-th channel, C is total number of characteristic diagram channels, H is total number of pixels of the characteristic diagram height, and W is total number of pixels of the characteristic diagram width; 2) And generating global feature weights, namely compressing channels by adopting a1 multiplied by 1 convolution transformation function F 1 on a global pooling feature tensor z c (w, h), and obtaining the global feature weights g c by a nonlinear activation function Sigmoid function, wherein the formula is as follows: ; 3) Generating an intermediate mapping feature map, and pooling the horizontal direction feature tensors And vertical pooling feature tensors Splicing in the space dimension, compressing channels by a convolution transformation function F 1 with the size of 1 multiplied by 1, carrying out batch normalization and nonlinear activation to obtain an intermediate mapping characteristic diagram F, wherein the formula is as follows: ; Wherein, the Meaning that the Concat splice operation is performed in both directions, Representing a nonlinear activation function, r representing a channel compression rate; 4) Generating horizontal and vertical feature weights, namely performing Split segmentation on the intermediate mapping feature map f along the space dimension to obtain independent feature tensors And Then, the characteristic weights g hor and g ver in the horizontal direction and the vertical direction are obtained by respectively operating through a 1X 1 convolution F h 、F w and matching with a Sigmoid activation function, and the formula is as follows: ; ; 5) The CCA features are obtained through weighted fusion, namely, the global feature weight g c , the horizontal feature weight g hor and the vertical feature weight g ver are subjected to element-by-element weighted fusion, and an output feature diagram Y CCA (w, h) of the CCA module is obtained, wherein the formula is as follows: ; the input feature map dimension is the channel number C×the height H×the width W, W is the current position in the width dimension, i is the traversal index in the width direction, H is the current position in the height dimension, and j is the traversal index in the height direction.
  5. 5. The escalator step detection method based on the YOLO algorithm and the digital twin technique according to claim 3, wherein in the step S2, the RepCCA attention mechanism is a three-branch parallel structure, the training phase adopts 3×3 convolution, 1×1 convolution and residual connection to process the output characteristics of the CCA module in parallel, and the output formula is ; The reasoning stage combines the three branches into a single 3×3 convolution, and the output formula is Wherein X RepCCA is RepCCA module input feature map, W 1 、W 2 is convolution weight, b 1 、b 2 is convolution offset, For convolution operation, f CCA () is the CCA module mapping function.
  6. 6. The escalator step detection method according to claim 1, wherein in step S3, the edge end real-time detection and the abnormal metadata generation specifically include: Step S3.1, model deployment, namely converting a TJ-YOLOv8 model into a ONNX/TensorRT lightweight format and deploying the model to an edge computing node; S3.2, real-time video stream input and detection reasoning, namely, an edge computing node receives a step operation video stream preprocessed by a front end, and performs format conversion, normalization and dimension adjustment on video frames to obtain adaptive model input single frame data; Invoking a TJ-YOLOv8 model to infer single frame data, and outputting diagonal bounding box coordinates (x n,1 ,y n,1 )、(x n,2 ,y n,2 ), object confidence and class probability distribution of normal/missing/shielding steps of each step instance; Step S3.3, constructing and updating a step chain type topological relation diagram, namely extracting the center point coordinates P center (x n ,y n )=((x n,1 +x n,2 )/2,(y n,1 +y n,2 /2 of each frame of step boundary frame, calculating the relative displacement vectors (delta x, delta y) of the adjacent step center points, and dynamically constructing and updating the step chain type topological relation diagram; The step chain topological relation is a difference value set of the central point coordinates of adjacent steps in the same frame in the directions of the x axis and the y axis of the image plane, wherein Deltax=x n -x n-1 ,Δy=y n -y n-1 , wherein (x n ,y n ) is the nth step central point coordinate, and (x n-1 ,y n-1 ) is the nth-1 step central coordinate; Step S3.4, a suspected missing step mark is set, wherein a confidence threshold T conf is set, and when the relative displacement vector of the step S norm <T conf or the adjacent step exceeds a normal range, the corresponding step is marked as suspected missing; the suspected missing step marking rule is that unique ID is allocated to the suspected missing step, the format is 'staircase ID-timestamp-step serial number', and mark triggering reasons including insufficient confidence level, topological chain fracture or both are recorded; S3.5, generating abnormal event metadata, namely fusing the pixel coordinates of the step center point and the height data acquired by the infrared ranging sensor, converting the camera internal parameters and external parameters to obtain three-dimensional space coordinates P 3D (X word ,Y word ,Z word under a digital twin body world coordinate system, and generating the abnormal event metadata comprising the coordinates, the detection time stamp and the topological chain fracture information; the abnormal event metadata is in a JSON format and comprises an abnormal unique ID, an escalator number, a microsecond time stamp, a step sequence number, three-dimensional space coordinates, a model detection result, topology chain information, a marking reason and a synchronous sensor data field.
  7. 7. The escalator step detection method according to claim 6, wherein in step S4, the digital twin model dynamic bidirectional driving specifically comprises: S4.1, constructing a digital twin body of the escalator, importing a three-dimensional model of a step by a geometric layer, embedding a step kinematic formula L k (t)=L 0 +v×t+1/2a×t 2 into the kinematic layer, and storing rated parameters of the escalator and sensor calibration parameters by a data layer, wherein L step (t) is the displacement/position of a kth step at a moment t, L 0 is the initial position of the step, v is the rated running line speed of the step, a is the running acceleration of the step, and t is the current time; S4.2, injecting sensor data and abnormal event metadata into a digital twin body through a Kafka stream processing engine, and performing hiding/deformation operation on normal steps kept displayed and suspected missing steps in the virtual model; Step S4.3, calculating a step theoretical height value H theory (t) based on a kinematic model, calculating a deviation delta H=H real (t)-H theory (t) from a sensor actual height value H real (t), and correcting sensor data to be H cal (t)=H real (t) -delta H; s4.4, correcting the reference coordinates of the step chain topology relationship by taking the calibrated height data as a reference, wherein the formula is as follows P 3D-cal (X cal ,Y cal ,Z cal )= P 3D (X word ,Y word ,Z word )+(0,0,ΔH).
  8. 8. The escalator step detection method according to claim 7, wherein in step S5, the multi-modal data fusion decision is specifically included: Step S5.1, three types of heterogeneous characteristics are extracted, namely a gradient height mutation quantity F h (t)=∣H cal (t)-H cal (t-1) I, a confidence coefficient attenuation quantity F c (t)=S norm (t-1)-S norm (t) and a topological chain breaking length are extracted ; Step S5.2, performing min-max normalization on the three types of features to obtain N h (t)、N c (t)、N t (t), and performing weighted fusion to generate an instant decision signal S (t) =alpha N h (t)+βN c (t)+γN t (t), wherein alpha, beta and gamma are weight coefficients adjusted according to the actual application environment; step S5.3, calculating the sum of the continuous 3-frame real-time decision signals as the accumulated decision signal Setting an accumulation threshold T A , and triggering history backtracking when A (T) > T A ; s5.4, inquiring the operation parameters of suspected abnormal steps for 5 seconds through the digital twin body, and calculating a historical backtracking evaluation value at the moment t Setting a backtracking threshold T R , and judging that the abnormality continuously exists when S (T) is greater than 0.5, A (T) is greater than 1.5 and R (T) is greater than T R , wherein P (T-g) is a single-frame abnormal characteristic quantification value of a corresponding step at the time of T-g, and g=0, 1,2,3 and 4 are obtained after the instant decision signal S (T-i) is normalized.
  9. 9. The escalator step detection method according to claim 1, wherein in step S6, millisecond alarm feedback and model self-updating are specifically included: s6.1, if the abnormality is judged to exist continuously, an emergency braking signal is sent through a Profinet/EtherCAT industrial bus within 50ms, and three-dimensional space coordinates of suspected missing steps are highlighted on a human-computer interface; s6.2, packaging the abnormal video segments, the sensor time sequence data, the decision signals and the decision labels into a training sample set; Step S6.3, freezing a TJ-YOLOv model backbone network based on a federal learning framework, and only trimming a neck network and an EL-Head detection Head, wherein trimming parameters are learning rate 0.001, batch size 16 and training round 10, and after completion, issuing updated model weights to edge nodes; And S6.4, the digital twin body generates a simulation fault case library based on the abnormal metadata, and optimizes the judging threshold value of the topological chain fracture prediction algorithm.

Description

Escalator step detection method integrating YOLO algorithm and digital twin technology Technical Field The invention belongs to the technical field of escalator detection, and particularly relates to a real-time detection method for escalator step missing by combining a YOLO algorithm with digital twinning. Background The escalator is used as special equipment in the public transportation field, the characteristic of continuous high-load operation of the escalator brings stringent requirements to safety detection, the steps are used as a passenger direct bearing body, and the missing faults of the escalator easily cause serious safety accidents such as falling of passengers and entanglement of limbs, so the escalator is one of the most serious accident hidden trouble of the escalator, and the escalator is very important to the real-time and reliable detection of the escalator. The existing escalator step missing detection means has the defect that ① carries out missing detection on an escalator which is in current standard commonly by adopting a photoelectric detection device. Although the method can realize basic ladder stopping protection, the photoelectric detection setting position is in the upper and lower machine bins, the detection cannot be carried out when the step is in an inclined area, and meanwhile, the method is easily interfered by ambient light, dust and oil stains, and the risk of false alarm or missing alarm exists. ② For a large number of escalators still in service and executing the old standard, the detection function is completely not provided, and the safety of the escalator completely depends on manual inspection during regular maintenance. The manual inspection has the problems of low efficiency, high cost, strong subjectivity, incapability of realizing 24-hour uninterrupted monitoring and the like, is highly dependent on experience and responsibility of maintenance personnel, and is difficult to form a stable and reliable safety guarantee closed loop. In recent years, a deep learning target detection algorithm (such as a YOLO series) and a digital twin technology are gradually applied to the field of escalator detection, but the prior art still has obvious defects that the YOLO algorithm is applied to a 'technology island' phenomenon and only stays in a visual identification and simple alarm link, a detection result cannot be deeply related with an escalator physical entity and a real-time running state, the decision support value of massive detection data is not mined, the digital twin technology is applied to a primary stage and is mainly in static or simple dynamic three-dimensional display, the digital twin technology cannot be deeply fused with a high-precision real-time abnormal detection algorithm, the decision support capability of a closed loop is lacked, and operation and maintenance decisions still depend on offline experience and on-site judgment. Disclosure of Invention The invention aims to provide a method for detecting steps of an escalator by fusing a YOLO algorithm with digital twinning, which solves the problems in the background art. In order to solve the technical problems, the invention adopts the following scheme: an escalator step detection method integrating a YOLO algorithm and a digital twin technology comprises the following steps: step S1, synchronously acquiring and preprocessing multi-source heterogeneous data, namely acquiring escalator step operation sensing data and video information, preprocessing, constructing a step state labeling data set based on the preprocessed data, and providing a data base for model training; Step S2, improving YOLOv n lightweight detection model construction and training: performing network transformation by taking YOLOv n as a basic framework, performing heavy parameterization transformation on a main network, inserting a Rep CCA attention mechanism into a neck network, reconstructing a detection Head into an EL-Head structure, performing channel pruning and INT8 quantization to realize light weight, and training and verifying a model after transformation by using a data set constructed in the step S1 to obtain a TJ-YOLOv model meeting the requirement of step detection precision; Step 3, edge end real-time detection and abnormal metadata generation, namely deploying a TJ-YOLOv model to an edge computing node, performing step detection on an escalator step operation video stream which is collected and preprocessed in real time on site, outputting a detection result, dynamically constructing a step chain topology relationship, marking suspected missing steps and generating abnormal event metadata with step three-dimensional space coordinates; S4, dynamic bidirectional driving of a digital twin model, namely constructing an escalator digital twin body comprising a step three-dimensional model, a kinematic model and a physical parameter database, injecting sensor data acquired in real time and abnormal event metadata into t