Search

CN-121982383-A - Multi-target tracking method of photoelectric detector

CN121982383ACN 121982383 ACN121982383 ACN 121982383ACN-121982383-A

Abstract

The invention discloses a multi-target tracking method of a photoelectric detector, which comprises the steps of assembling double-light integrated detectors in a plurality of terminal devices, synchronously collecting resolution images and thermal imaging images of long-time sequences, dynamically fusing the visible light images and the infrared thermal imaging images by utilizing perspective transformation, generating bimodal images based on transformation matrixes, marking targets in the bimodal images by detecting frames according to a target detection algorithm, extracting appearance characteristics and motion characteristics of the framed targets to construct a multi-target characteristic set, establishing a dynamic image network model, analyzing space-time motion tracks of the multi-target characteristic set, tracking and matching multi-target space information, and outputting target motion tracks, so that the accuracy of track tracking in challenging scenes such as shielding, crossing and the like is improved, and the whole system is balanced between calculation efficiency and accuracy.

Inventors

  • LI KEHUI

Assignees

  • 北京中科飞鸿科技股份有限公司

Dates

Publication Date
20260505
Application Date
20260115

Claims (10)

  1. 1. A method for multi-target tracking of a photodetector, comprising: step S1, assembling double-light integrated detectors in a plurality of terminal devices, and synchronously collecting visible light images and infrared thermal imaging images in a long time sequence; step S2, dynamically fusing the visible light image and the infrared thermal imaging image by utilizing perspective transformation, and generating a bimodal image based on a transformation matrix; S3, marking a detection frame of the target in the bimodal image according to a target detection algorithm; S4, extracting appearance features and motion features of the framed object, and constructing a multi-object feature set; And S5, establishing a dynamic graph network model, analyzing the space-time motion trail of the multi-target feature set, tracking and matching multi-target space information, and outputting a target motion trail.
  2. 2. The method for multi-target tracking of a photodetector according to claim 1, wherein step S1 comprises: step S11, assembling double-light integrated detectors in N terminal devices; step S12, setting a main control FPGA, and sending a signal instruction to N terminal devices; Step S13, the N terminal devices synchronously receive signal instructions through N interfaces, and synchronously acquire visible light images and infrared thermal imaging images in unit time by using a double-light integrated detector; Step S14, uploading the visible light image and the infrared thermal imaging image to a master control FPGA; The double-light integrated detector comprises a visible light sensor and an infrared sensor; the visible light sensor comprises CMOS, CCD, CIS and a quantum dot image sensor; the infrared sensor comprises a thermal sensor, a quantum sensor, a passive infrared sensor, an active infrared sensor and an infrared focal plane detector.
  3. 3. The method for multi-target tracking of a photodetector according to claim 2, wherein step S2 comprises: Step S21, taking the width of the visible light image as a horizontal coordinate axis, the height of the visible light image as a vertical coordinate axis, establishing a first coordinate system based on the visible light image, setting a chessboard grid according to the size of the first coordinate system, and dividing the chessboard grid into three coordinate axes Mapping the chessboard grid into the first coordinate system by using OpenCV, detecting image target corner points in the chessboard grid, acquiring the positions of the image target corner points detected by each partition, finding out corner point pixel coordinates of the image target corner points in the first coordinate system, and storing the corner point pixel coordinates as first corner point coordinates; Step S22, carrying out histogram equalization on the infrared thermal imaging image, establishing a second coordinate system according to the width and the height of the infrared thermal imaging image, mapping the chessboard grid into the second coordinate system, detecting infrared imaging target angular points in the chessboard grid, acquiring the positions of the infrared imaging target angular points detected by each subarea, finding out infrared angular point coordinates of the infrared imaging target angular points in the second coordinate system, and storing the infrared imaging target angular points as second angular point coordinates; Step S23, setting z=0 by taking the chessboard grid as an xy plane, establishing 3D physical coordinates, and inputting the first angular point coordinates and the second angular point coordinates into a three-dimensional correction function to obtain a first projection matrix And a second projection matrix Calculating a homography transformation matrix H from a visible light image to an infrared thermal imaging image, aligning the second coordinate system to the first coordinate system according to the homography transformation matrix, and generating a bimodal image, wherein the homography transformation matrix H has the following calculation expression: ; And step S24, repeating the steps S21, S22 and S23 at the same time point according to different tenses of the visible light image and the infrared thermal imaging image, and dynamically calculating a homography transformation matrix until the second coordinate system and the first coordinate system are aligned to obtain a bimodal image.
  4. 4. The method for multi-target tracking of a photodetector according to claim 3, wherein step S3 comprises: And identifying the target in the bimodal image based on a target detection algorithm of deep learning image style migration, obtaining a preliminary detection frame, manually marking a category label of the preliminary detection frame by using a marking tool, obtaining detection frame data, and enabling the detection frame data to correspond to the bimodal image one by one, wherein the detection frame data comprises detection category labels and detection frame coordinates.
  5. 5. The multi-target tracking method of a photodetector as defined in claim 4, wherein: The step S4 specifically comprises the following steps: Step S41, according to the bimodal image and the corresponding detection frame coordinates, a target area which is detected by a detection frame in the multiscale clipping image is adjusted, the pixel value of the target area is scaled to be within a fixed threshold value, the average value of the pixel value of the target area is calculated, the difference value between the current average value of the pixel and the target average value of the previous frame is calculated, and the difference value is stored as an appearance characteristic according to a time sequence; Step S42, calculating the center coordinates of the target area, obtaining a detection frame sequence of the target in continuous frames, and calculating the displacement vector of the target, wherein the calculation expression of the displacement vector of the target is as follows: ; Wherein, the As a displacement vector of the object, For the current target region center coordinates, For the center coordinates of the target area of the previous frame, s is the number of the target area of the s-th frame, and the moving speed and the direction angle of the target are calculated, wherein the calculating expression of the moving speed and the direction angle is as follows: movement speed = ; ; Wherein, the For the horizontal displacement vector to be used, Calculating the average speed, acceleration and direction conversion rate of the target moving speed according to the moving speed and the direction angle for the vertical displacement vector; And step S43, storing the average speed, the acceleration and the direction conversion rate of the target as motion characteristics, and forming a multi-target characteristic set by corresponding the appearance characteristics and the motion characteristics one by one.
  6. 6. The method for multi-target tracking of a photodetector of claim 5, wherein step S5 comprises: Step S51, setting appearance characteristics and motion characteristics of the multi-target characteristic set as dynamic graph network nodes, defining space edges and time edges, and establishing a dynamic graph network model according to the characteristics of the space edges and the time edges; Step S52, according to the time sequence, the target node state of the previous frame is transferred to the current frame through a time edge, GRU is used for fusing the current observed target node and the history node state, and k times are repeated, so that the update of the dynamic graph network node is completed; Step S53, using the bipartite graph cost matrix, performing optimal target matching on the updated dynamic graph network node and the initial dynamic graph network node to obtain a target node successfully matched; step S54, the motion characteristics of the successfully matched target nodes are updated, the action track is prolonged, and the target nodes which are not successfully matched are divided into new targets and lost targets: initializing a new target track, and repeatedly executing steps S51, S52 and S53; The lost target starts prediction compensation for the lost target by using Kalman filtering.
  7. 7. The method for multi-target tracking of a photodetector of claim 6, wherein said histogram equalization is: Counting the pixel occurrence frequency of each gray level in the infrared thermal imaging graph, sorting the pixel occurrence frequency from small to large, performing one-to-one correspondence with the induction temperature of the infrared thermal imaging graph, and intercepting an effective temperature interval, wherein the effective temperature interval setting range comprises: Setting a minimum threshold value < an effective temperature interval < a maximum threshold value; pixels below the effective temperature interval are set as pixels corresponding to the set minimum threshold, and pixels above the effective temperature interval are set as pixels corresponding to the set maximum threshold.
  8. 8. The method for multi-target tracking of a photodetector of claim 7, wherein said method for multi-scale cropping comprises: and comparing the temperature difference of the detection frame area with the corresponding detection frame coordinates of the bimodal image, and cutting the area with obvious temperature difference to be used as a target of multi-scale cutting.
  9. 9. The method for multi-target tracking of a photodetector of claim 8, wherein said specific method for enabling predictive compensation of said lost target using Kalman filtering is: setting a prediction compensation condition of Kalman filtering: When the continuous lost frame number of the lost target is larger than a threshold value, triggering prediction compensation, performing covariance matrix prediction on the lost target by using a uniform model, generating a lost target prediction track, drawing a track visualization frame, stopping prediction when the lost target is successfully detected, and removing the lost target prediction track corresponding to the lost target when the prediction exceeds the maximum set frame number.
  10. 10. A multi-target tracking method for a photodetector as defined in claim 9, wherein: the dynamic graph network node comprises timestamp information, moving speed, direction angle and displacement vector of the target; the space edge comprises the distance between different targets in the same frame, the moving speed and the contrast of the direction angle; The temporal edge includes a position offset connection between adjacent frames of the same target in context time.

Description

Multi-target tracking method of photoelectric detector Technical Field The invention relates to the technical field of target learning, in particular to a multi-target tracking method of a photoelectric detector. Background With the development of sensor and computer technology, the photoelectric detector has shown a development trend of high intelligence, biochemical imitation and multidimensional, and aiming at complex scenes of shielding, low illumination and dynamic background, the multi-target tracking method of the photoelectric detector can be effectively adapted to various environments, and the technical bottleneck of traditional photoelectric tracking is solved. The invention in China with the application number 202411702591.7 discloses a multi-target photoelectric laser detection system and a method, and mainly comprises a laser measuring device, a photoelectric detector and a scanning swing mirror, wherein the photoelectric detector comprises a visible light detector and an infrared detector, the scanning swing mirror is arranged at the front end of the laser measuring device, position information of a target detected by the photoelectric detector is sent to the laser measuring device and the scanning swing mirror, and the scanning swing mirror is shifted according to the position information to adjust the optical axis of the laser measuring device, and meanwhile auxiliary lighting, tracking ranging and searching ranging functions are considered. The invention does not consider multi-target tracking obstacle under complex environment, can not predict the possibility track of target tracking in real time, and does not combine the dynamic graph network with space-time combined reasoning. Disclosure of Invention The method solves the technical problems that the multi-target tracking obstacle in the complex environment is not considered, the possible track of target tracking cannot be predicted in real time, and a dynamic graph network is not combined with space-time combined reasoning. In order to solve the technical problems, the invention provides the following technical scheme: A multi-target tracking method of a photodetector, wherein: step S1, assembling double-light integrated detectors in a plurality of terminal devices, and synchronously collecting visible light images and infrared thermal imaging images in a long time sequence; step S2, dynamically fusing the visible light image and the infrared thermal imaging image by utilizing perspective transformation, and generating a bimodal image based on a transformation matrix; S3, marking a detection frame of the target in the bimodal image according to a target detection algorithm; S4, extracting appearance features and motion features of the framed object, and constructing a multi-object feature set; And S5, establishing a dynamic graph network model, analyzing the space-time motion trail of the multi-target feature set, tracking and matching multi-target space information, and outputting a target motion trail. Preferably, step S1 specifically includes: step S11, assembling double-light integrated detectors in N terminal devices; step S12, setting a main control FPGA, and sending a signal instruction to N terminal devices; Step S13, the N terminal devices synchronously receive signal instructions through N interfaces, and synchronously acquire visible light images and infrared thermal imaging images in unit time by using a double-light integrated detector; Step S14, uploading the visible light image and the infrared thermal imaging image to a master control FPGA; The double-light integrated detector comprises a visible light sensor and an infrared sensor; the visible light sensor comprises CMOS, CCD, CIS and a quantum dot image sensor; the infrared sensor comprises a thermal sensor, a quantum sensor, a passive infrared sensor, an active infrared sensor and an infrared focal plane detector. Preferably, step S2 specifically includes: Step S21, taking the width of the visible light image as a horizontal coordinate axis, the height of the visible light image as a vertical coordinate axis, establishing a first coordinate system based on the visible light image, setting a chessboard grid according to the size of the first coordinate system, and dividing the chessboard grid into three coordinate axes Mapping the chessboard grid into the first coordinate system by using OpenCV, detecting image target corner points in the chessboard grid, acquiring the positions of the image target corner points detected by each partition, finding out corner point pixel coordinates of the image target corner points in the first coordinate system, and storing the corner point pixel coordinates as first corner point coordinates; Step S22, carrying out histogram equalization on the infrared thermal imaging image, establishing a second coordinate system according to the width and the height of the infrared thermal imaging image, mapping the chessboard grid into the second c