CN-121999016-A - Photoelectric turntable signal processing system and method based on multi-core heterogeneous architecture

CN121999016ACN 121999016 ACN121999016 ACN 121999016ACN-121999016-A

Abstract

The invention relates to a photoelectric turntable signal processing system and method based on a multi-core heterogeneous architecture. The system comprises an FPGA, a multi-core DSP and an ARM processor integrating a neural network reasoning engine, wherein the FPGA acquires visible light and infrared video, the preprocessing image is distributed to the DSP and the ARM in parallel through a SRIO and LVDS special passage, the ARM operation improves YOLOv a 4-tiny model to perform target recognition and sends position information to the DSP for guiding tracking initialization, the method comprises the steps that the FPGA acquires the distributed image in parallel, the ARM recognizes a target and guides the DSP to track, the DSP calculates the miss distance and integrates a memory tracking module, and Kalman prediction is kept output when the target is lost. The scheme is used for identifying and tracking depth cooperation, the end-to-end delay is less than or equal to 20ms, the recovery rate in 2 seconds of loss is more than or equal to 95%, the volume power consumption is reduced by more than 30%, and the method is suitable for military/police photoelectric reconnaissance, security monitoring, unmanned aerial vehicle load and other scenes.

Inventors

ZHOU ZHI
LI LEI
PEI JIANHUA
ZHANG BIWU
ZHOU YANG
SUN RONGKANG

Assignees

中国电子科技集团公司第五十八研究所

Dates

Publication Date: 20260508
Application Date: 20260320

Claims (8)

1. An optoelectronic turntable signal processing system based on a multi-core heterogeneous architecture is characterized by comprising: The FPGA is configured to collect at least two paths of original video data streams from the photoelectric sensor and preprocess the original video data, wherein the preprocessing at least comprises decoding, YUV format conversion and buffering; the FPGA is further configured to distribute the preprocessed first path of image data to the multi-core DSP through a first special data path and distribute the preprocessed second path of image data to the ARM in parallel through a second special data path, wherein the first special data path is a serial rapidIO interface, and the second special data path is an LVDS interface; The multi-core DSP is configured to receive the first path of image data through the serial rapidIO interface, execute a target tracking algorithm in parallel by the multi-core to calculate the off-target quantity and output the off-target quantity to a servo control system through the FPGA, wherein the multi-core DSP is internally configured to execute target tracking tasks in parallel by the multi-core, and the multi-core DSP comprises the following components: A first kernel for extracting HOG features of the image data; A second core for performing discrete fourier transform/inverse transform and correlation calculation; A third core for performing a tracking template update and average peak correlation energy calculation; The multi-core DSP further comprises a memory tracking module, wherein the memory tracking module is configured to predict the position based on the motion state before the target is lost and maintain the output of the off-target quantity in a preset time window when the average peak correlation energy value is lower than a preset threshold value to judge that the target is lost; The ARM is integrated with a neural network reasoning engine, is configured to receive the second path of image data through the LVDS interface, runs a light-weight target detection model after being subjected to INT8 quantization conversion, is of a network structure based on YOLOv-tiny improvement, comprises a newly added stride=64 depth feature layer, is used for carrying out target detection and classification on the image data and generating target position information, and is further configured to send the target position information to the multi-core DSP so as to guide or update target tracking; The dual-state interaction channel is arranged between the multi-core DSP and the ARM and is used for transmitting tracking state information and an identification control instruction, wherein the tracking state information comprises the average peak correlation energy value, and the ARM dynamically adjusts the identification strategy according to the received average peak correlation energy value to form a dual-direction closed loop for identification and tracking.
2. The optoelectronic turntable signal processing system of claim 1, wherein the multi-core DSP is further configured to calculate an average peak correlation energy value in real time and compare the average peak correlation energy value with a preset threshold to determine the tracking state, wherein the preset threshold is 12, and the target is determined to be lost when the average peak correlation energy value is lower than 12.
3. The system for processing the signals of the photoelectric turntable according to claim 1, wherein a preset time window of the memory tracking module is 0.5 to 2 seconds, a predicted miss distance is continuously output in the time window, if a target is captured again in the time window, a normal tracking mode is restored, and otherwise, a tracking failure state is reported.
4. The optoelectronic turntable signal processing system of claim 1, wherein the ARM dynamically adjusts the recognition strategy according to the received average peak correlation energy value, wherein the ARM reduces the recognition area to the periphery of the tracking window to reduce the computational effort consumption when the average peak correlation energy value is higher than a preset threshold value, and wherein the ARM resumes the full-image detection to increase the target re-capture probability when the average peak correlation energy value is lower than the preset threshold value.
5. The optoelectronic turntable signal processing system of claim 1, wherein the system is further configured to enter a search mode or report a lost state after a target loss timeout, the timeout period being configurable to 2 seconds.
6. A method for processing signals of an optoelectronic turntable, which is applied to the system of any one of claims 1 to 5, and is characterized by comprising the following steps: the method comprises the steps that S1, multi-path video data are collected in parallel by an FPGA, after preprocessing, first path image data are distributed to a multi-core DSP through a serial rapidIO interface, and second path image data are distributed to an ARM in parallel through an LVDS interface; Step S2, performing target identification by ARM based on a neural network reasoning engine, performing target detection and classification on images by adopting a YOLOv-tiny improved model quantized by INT8, and restraining and outputting target position and category information with highest confidence coefficient through non-maximum values, and sending the target position information to a multi-core DSP; Step S3, performing target tracking by the multi-core DSP based on the target position information or the tracking state of the previous frame, and running an improved multi-scale core correlation filtering algorithm by the multi-core in parallel to calculate the off-target quantity in real time and output the off-target quantity; And S4, when the tracking loss of lock is judged to be triggered, starting a memory tracking module by the multi-core DSP, predicting the target position based on a Kalman filtering algorithm, continuously outputting predicted off-target quantity within a preset time window of 0.5 to 2 seconds, maintaining the directional tracking of the target by the servo system, and dynamically adjusting an identification strategy by the ARM according to the received average peak related energy value to form a bidirectional closed loop for identification and tracking.
7. The method of claim 6, wherein the YOLOv-tini modified model in step S2 includes a newly added stride=64 depth feature layer for extracting deep features of high semantic information to enhance small object recognition capability.
8. The method of claim 6, wherein the predetermined threshold in step S3 is 12, and tracking loss is determined when the average peak correlation energy value is lower than 12.

Description

Photoelectric turntable signal processing system and method based on multi-core heterogeneous architecture Technical Field The invention relates to the technical field of photoelectric information processing and intelligent control, in particular to a photoelectric turntable signal processing system and method based on a multi-core heterogeneous architecture. Background The traditional photoelectric turntable signal processing system generally adopts an acquisition-transmission-processing separation type architecture. The image acquisition unit, the target recognition unit, the tracking control unit and the servo driving unit are usually physically or logically independent modules, and perform data interaction through a standard bus (such as ethernet and PCIe). The architecture causes long data transmission paths among modules and many links, introduces significant communication delay and processing delay, and is difficult to meet the requirement of continuous and stable tracking of high-speed and high-maneuvering targets. Along with the development of deep learning technology, a target detection and recognition method based on a convolutional neural network has been applied in various scenes. However, when such techniques are applied to optoelectronic turntable embedded systems that have stringent requirements on real-time, robustness and power consumption, the existing solutions still have significant bottlenecks: First, there is a conflict between the high-precision recognition model and the embedded computing power and real-time requirements. In order to improve the recognition accuracy in complex scenes, the prior art mostly adopts a deep neural network model with huge parameter quantity and high calculation complexity. Such models are still viable on general purpose computing platforms (e.g., GPUs), but are difficult to deploy directly to the resource-constrained embedded processing units employed by the optoelectronic turntable (e.g., ARM application processors). Although there have been attempts to lightweight models such as YOLO series, the balance problem between accuracy loss, inference speed and power consumption has not been solved at all, and it has been difficult to satisfy the system requirements of both high recognition rate and high processing frame rate. Secondly, the identification function and the follow-up tracking control link are not coordinated enough. In the existing optoelectronic turntable system architecture, the target recognition module and the tracking algorithm module are usually designed as independent or serial functional units. The recognition result is only used for initial capture or periodic verification of the target, and a real-time and compact state feedback and closed loop cooperative mechanism is absent between the two modules. When the target is lost temporarily due to occlusion, rapid movement or background interference, the system cannot make a rapid and autonomous decision and recover according to the context information provided by the identification module or the state feedback of the tracking module, so that the overall intelligent level and continuous and stable tracking capability of the system are limited. In addition, the multi-sensor data fusion processing capability is weak. In order to improve environmental adaptability, modern photoelectric turntables often integrate sensors in multiple bands such as visible light, infrared and the like. In the prior art, the processing of multiple paths of video streams stays in the independent processing and simple superposition layers, and the efficient and real-time heterogeneous data fusion at a feature layer or a decision layer cannot be realized, so that the complementary advantages of multiple sources of information cannot be fully exerted, and the system performance is easy to be suddenly reduced when the imaging quality of a single spectrum is poor. Therefore, the scheme in the prior art is difficult to meet the integrated requirements of the high-end photoelectric turntable system on high real-time, high reliability and low power consumption intelligent processing, and an innovative system architecture and a collaborative processing method are needed. Disclosure of Invention In order to solve the technical problems, the invention provides a photoelectric turntable signal processing system based on a multi-core heterogeneous architecture, which comprises the following components: The FPGA is configured to collect at least two paths of original video data streams from the photoelectric sensor and preprocess the original video data, wherein the preprocessing at least comprises decoding, YUV format conversion and buffering; the FPGA is further configured to distribute the preprocessed first path of image data to the multi-core DSP through a first special data path and distribute the preprocessed second path of image data to the ARM in parallel through a second special data path, wherein the first special data