CN-122017880-A - SPAD three-dimensional imaging method and device based on photon behavior tracking

CN122017880ACN 122017880 ACN122017880 ACN 122017880ACN-122017880-A

Abstract

The invention provides a SPAD three-dimensional imaging method and device based on photon behavior tracking, and relates to the technical field of SPAD and computer vision. The method comprises the steps of initializing a plurality of parallel nerve trackers, configuring state parameters comprising depth positions and uncertainty widths by the nerve trackers, obtaining photon time stamp sequences and converting each photon into a distance observation value, predicting dynamics control parameters and residual correction parameters by utilizing a pre-trained double-branch neural network according to the current state parameters and the current distance observation values of the nerve trackers so as to control macroscopic motion trend and microscopically correct smooth jitter, calculating attention masks according to whether the distance observation values fall into the uncertainty widths, updating the state parameters according to the dynamics control parameters, the residual correction parameters and the attention masks, and preferentially outputting depth estimation values based on accumulated confidence of the attention masks after the photon time stamp sequences are processed. Thus, the noise immunity robustness is high and the prediction accuracy is high.

Inventors

LAI RUI
HAO WEIDONG
FAN RUI
WANG YIKAI
GUAN JUNTAO
LI DONG
MA RUI
ZHU ZHANGMING

Assignees

西安电子科技大学

Dates

Publication Date: 20260512
Application Date: 20260413

Claims (10)

1. A SPAD three-dimensional imaging method based on photon behavior tracking is characterized by comprising the following steps: Initializing a plurality of parallel nerve trackers in a preset detection depth range, wherein each nerve tracker is configured with a state parameter comprising a depth position and an uncertainty width; acquiring a photon time stamp sequence acquired by a single photon avalanche diode sensor, and converting each photon in the photon time stamp sequence into a corresponding distance observation value according to a flight time principle; According to the current state parameters and the current distance observation values of the nerve trackers, respectively predicting dynamics control parameters and residual error correction parameters by utilizing a pre-trained double-branch nerve network, wherein the dynamics control parameters are used for controlling macroscopic motion trend of the nerve trackers, and the residual error correction parameters are used for carrying out microscopic correction on the nerve trackers so as to smooth track jitter; Calculating an attention mask according to whether the distance observation value falls within the uncertainty width of the current nerve tracker, and updating state parameters according to the dynamics control parameters, the residual error correction parameters and the attention mask to obtain updated state parameters of the current nerve tracker, wherein the updated state parameters comprise final state parameters; And after the photon time stamp sequence is processed, selecting a depth position in the final state parameter corresponding to the nerve tracker with the highest confidence as a depth estimation value according to the accumulated confidence of the nerve tracker based on the attention mask in the updating process, and using the depth position as a three-dimensional imaging.
2. The SPAD three-dimensional imaging method based on photon behavior tracking according to claim 1, wherein said state parameters further comprise velocity momentum, hidden memory state, and residual feedback amount.
3. The SPAD three-dimensional imaging method based on photon behavior tracking according to claim 2, wherein said pre-trained dual-branch neural network comprises parallel dynamic motion branches and residual correction branches, said dynamic motion branch first neural network nonlinear mapping function and said residual correction branch second neural network nonlinear mapping function each employ a multi-layer perceptron structure; The method for respectively predicting the dynamics control parameter and the residual error correction parameter by utilizing the pre-trained double-branch neural network according to the current state parameter and the current distance observation value of each neural tracker comprises the following steps: Inputting the deviation between the current distance observation value and the corresponding depth position and the corresponding uncertainty width into the dynamic motion branch to be processed by the dynamic motion branch, and outputting the dynamic control parameters, wherein the dynamic control parameters comprise global driving force, kalman gain coefficient and width contraction factor; And inputting the normalized relative position between the current distance observation value and the corresponding depth position, the corresponding uncertainty width, the hidden memory state and the residual feedback quantity at the last moment into the residual correction branch so as to process by utilizing the residual correction branch and output the residual correction parameters, wherein the residual correction parameters comprise position residual errors, speed residual errors and memory updating characteristics.
4. The SPAD three-dimensional imaging method based on photon behavior tracking according to claim 1, wherein said attention mask is expressed as: ; Wherein, the Is the first Attention mask for a time step, The function is activated for Sigmoid, Is the first Distance observations of the individual time steps, Is the first Depth position of the time step, Is the first Uncertainty width of individual time steps.
5. A SPAD three-dimensional imaging method based on photon behavior tracking according to claim 3, wherein said updated state parameters comprise updated velocity momentum, updated depth position, updated uncertainty width, updated hidden memory state and updated residual feedback quantity; And updating the state parameter according to the dynamics control parameter, the residual error correction parameter and the attention mask to obtain an updated state parameter of the current nerve tracker, wherein the method comprises the following steps: According to the Kalman gain coefficient, the global driving force, the speed residual error and the attention mask, carrying out weighted fusion on the speed momentum to obtain the updated speed momentum; Updating the depth position according to the updated velocity momentum, the position residual error and a preset momentum leakage factor to obtain the updated depth position, wherein the preset momentum leakage factor is used for keeping the inertia drift of the nerve tracker when the attention mask is not activated; adaptively shrinking the uncertainty width according to the width shrinkage factor and the attention mask to obtain the updated uncertainty width; according to the memory updating characteristics, carrying out moving average updating on the hidden memory state to obtain the updated hidden memory state; and combining the position residual error and the speed residual error to obtain the updated residual error feedback quantity.
6. The SPAD three-dimensional imaging method based on photon behavior tracking according to claim 5, wherein said updated velocity momentum is expressed as: ; Wherein, the Is the first The updated velocity momentum of the individual time steps, Is the first The kalman gain factor for each time step, Is the first The momentum of the velocity of the individual time steps, Is the first The global driving force of the individual time steps, Is the first The speed residual of the one time step, Is the first Attention mask for each time step.
7. The SPAD three-dimensional imaging method based on photon behavior tracking according to claim 5, wherein said updated depth position is expressed as: ; Wherein, the Is the first Updated depth positions for the individual time steps, Is the first Depth position of the time step, Is the first The updated velocity momentum of the individual time steps, Is the first Attention mask for a time step, For the preset momentum leak factor, Is the first Position residuals for each time step.
8. The SPAD three-dimensional imaging method based on photon behavior tracking according to claim 1, wherein after processing the photon time stamp sequence, selecting a depth position in the final state parameter corresponding to a neural tracker with a highest confidence as a depth estimation value according to the accumulated confidence of the attention mask of each neural tracker in an updating process, wherein the depth estimation value is used for constructing three-dimensional imaging, and the method comprises: after the photon time stamp sequence is processed, accumulating the attention masks obtained by the nerve trackers in the updating process respectively to obtain accumulated confidence coefficients of the nerve trackers; comparing the accumulated confidence degrees of all the parallel nerve trackers, and selecting a target nerve tracker, wherein the target nerve tracker is the nerve tracker with the highest accumulated confidence degree; and taking the depth position in the final state parameter of the target nerve tracker as the depth estimated value for constructing the three-dimensional imaging.
9. A SPAD three-dimensional imaging method based on photon behavior tracking according to claim 3, wherein said expression of kinetic motion branching is: ; Wherein, the For the said kinetic control parameters, Is the first The global driving force of the individual time steps, Is the first The kalman gain factor for each time step, Is the first The width contraction factor of the individual time steps, A first neural network nonlinear mapping function that branches the dynamic motion, Is the first Distance observations of the individual time steps, Is the first Depth position of the time step, Is the first Uncertainty width of individual time steps; The residual correction branch has the expression: ; Wherein, the The parameters are modified for the said residual errors, Is the first The position residuals of the individual time steps, Is the first The speed residual of the one time step, Is the first The memory of the individual time steps updates the characteristics, A second neural network nonlinear mapping function that corrects branches for the residual, Is the first The hidden memory state of each time step, Is the first Residual feedback amount for each time step.
10. A SPAD three-dimensional imaging apparatus based on photon behavior tracking, comprising: the system comprises an initialization module, a detection module and a control module, wherein the initialization module is used for initializing a plurality of parallel nerve trackers in a preset detection depth range, and each nerve tracker is configured with state parameters comprising depth positions and uncertainty widths; the conversion module is used for acquiring a photon time stamp sequence acquired by the single photon avalanche diode sensor and converting each photon in the photon time stamp sequence into a corresponding distance observation value according to a flight time principle; The prediction module is used for respectively predicting dynamics control parameters and residual error correction parameters according to the current state parameters and the current distance observation values of the nerve trackers by utilizing a pre-trained double-branch nerve network, wherein the dynamics control parameters are used for controlling macroscopic motion trend of the nerve trackers, and the residual error correction parameters are used for carrying out microscopic correction on the nerve trackers so as to smooth track jitter; The updating module is used for calculating an attention mask according to whether the distance observation value falls within the uncertainty width of the current nerve tracker, and updating state parameters according to the dynamics control parameters, the residual error correction parameters and the attention mask to obtain updated state parameters of the current nerve tracker, wherein the updated state parameters comprise final state parameters; And the selection module is used for selecting the depth position in the final state parameter corresponding to the nerve tracker with the highest confidence as a depth estimation value according to the accumulated confidence of the attention mask in the updating process of each nerve tracker after the photon time stamp sequence is processed, so as to construct three-dimensional imaging.

Description

SPAD three-dimensional imaging method and device based on photon behavior tracking Technical Field The invention relates to the technical field of Single-photon avalanche diode (SPAD) three-dimensional imaging and computer vision, in particular to a SPAD three-dimensional imaging method and device based on photon behavior tracking. Background SPAD has become a key technology in the field of high-precision three-dimensional depth sensing imaging by virtue of single photon level detection sensitivity and picosecond level time resolution. By actively emitting laser pulses and measuring the Time-of-Flight (ToF), the SPAD sensor can realize three-dimensional scene reconstruction under extreme conditions of extremely low illumination, strong background light interference and other failure of the traditional sensor, and has important application value in the fields of automatic driving, robot navigation, augmented reality and the like. However, the highly sensitive nature of SPADs allows capture of signal photons while also introducing significant amounts of ambient background noise (e.g., sunlight). Especially in outdoor strong light scenes, signal photons are often submerged by random background noise, and how to extract signal photons from ambient light interference is a core technical challenge. The existing mainstream SPAD three-dimensional imaging method generally adopts a histogram accumulation strategy, namely photon time stamps acquired in a plurality of detection periods are constructed into a statistical histogram, and target distance information is acquired by detecting the peak position of the histogram. However, this histogram accumulation strategy presents a significant hardware resource bottleneck in that very fine time bins need to be set to ensure depth measurement accuracy, resulting in a dramatic expansion in the resource consumption of on-chip memory with increasing sensor resolution and probe depth. This large memory overhead makes it difficult for histogram accumulation strategies to implement on-chip integration on power-and area-limited edge-side hardware. Aiming at the bottleneck problem of hardware resources, the recent research is turned to a neural network processing method without a histogram, and the time sequence models such as a cyclic neural network or a pulse neural network are utilized to directly process an original photon time stamp sequence and regress to obtain a depth value, so that the storage overhead caused by the construction of the histogram is avoided. However, the existing neural network method generally simplifies depth estimation into a single black box regression task, lacks an explicit distinguishing mechanism for signal photons and background noise photons, and is easy to excessively fit short-time noise burst under the condition of strong background light interference or low signal to noise ratio, so that a depth estimation result is severely dithered, and the prediction accuracy and noise robustness of the prior art are lower. Disclosure of Invention The embodiment of the invention aims to provide a SPAD three-dimensional imaging method and device based on photon behavior tracking, which solve the problems of lower prediction accuracy and poor noise immunity robustness in the prior art. In order to solve the technical problems, the embodiment of the invention provides the following technical scheme: the first aspect of the invention provides a SPAD three-dimensional imaging method based on photon behavior tracking, which comprises the following steps: Initializing a plurality of parallel nerve trackers in a preset detection depth range, wherein each nerve tracker is configured with a state parameter comprising a depth position and an uncertainty width; Acquiring a photon time stamp sequence acquired by a single photon avalanche diode sensor, and converting each photon in the photon time stamp sequence into a corresponding distance observation value according to a flight time principle; according to the current state parameters and the current distance observation values of the nerve trackers, respectively predicting dynamics control parameters and residual error correction parameters by utilizing a pre-trained double-branch nerve network, wherein the dynamics control parameters are used for controlling macroscopic motion trend of the nerve trackers, and the residual error correction parameters are used for carrying out microscopic correction on the nerve trackers so as to smooth track jitter; calculating an attention mask according to whether the distance observation value falls within the uncertainty width of the current nerve tracker, and updating state parameters according to the dynamics control parameters, the residual error correction parameters and the attention mask to obtain updated state parameters of the current nerve tracker, wherein the updated state parameters comprise final state parameters; After the photon timestamp sequence is processed, acco