CN-122023970-A - Failure detection method, system, equipment and medium for parallax prediction

CN122023970ACN 122023970 ACN122023970 ACN 122023970ACN-122023970-A

Abstract

The invention provides a failure detection method, a system, equipment and a medium for parallax prediction, wherein the method comprises the steps of obtaining a left image and a right image of a target scene, inputting the left image and the right image into a matching cost calculation network, outputting corresponding matching cost results, respectively inputting the matching cost results into an uncertainty estimation network and a parallax estimation network, outputting corresponding uncertainty prediction results and parallax prediction results, inputting the uncertainty prediction results and the parallax prediction results into a fusion network, and outputting corresponding failure detection results, wherein the failure detection results comprise failure, scene environment failure, camera calibration parameter failure and lens physical failure. According to the invention, by combining multiple networks to cooperatively work, the high precision and high reliability of parallax prediction are realized, and failure detection and processing are performed when necessary, so that the safety and robustness of unmanned aerial vehicle application are effectively improved.

Inventors

CHANG ZHIZHONG
WANG GUOHUI
ZHANG YU

Assignees

黑龙江惠达科技股份有限公司

Dates

Publication Date: 20260512
Application Date: 20251204

Claims (10)

1. The utility model provides a failure detection method of parallax prediction, which is characterized in that the method is applied to a convolutional neural network, wherein the convolutional neural network comprises a matching cost calculation network, an uncertainty estimation network, a parallax estimation network and a fusion network, and the method comprises the following steps: acquiring a left image and a right image of a target scene, inputting the left image and the right image into a matching cost calculation network, and outputting a corresponding matching cost result; Respectively inputting the matching cost result into an uncertainty estimation network and a parallax estimation network, and outputting a corresponding uncertainty prediction result and a corresponding parallax prediction result; And inputting the uncertainty prediction result and the parallax prediction result into a fusion network, and outputting a corresponding failure detection result, wherein the failure detection result comprises failure prevention, scene environment failure, camera calibration parameter failure and lens physical failure.
2. The method according to claim 1, wherein the steps of obtaining a left image and a right image of the target scene, inputting the left image and the right image into a matching cost calculation network, and outputting a corresponding matching cost node, further comprise: Freezing parameters of the fusion network, training an uncertainty estimation network and a parallax estimation network, and enabling the uncertainty estimation network and the parallax estimation network to independently learn and converge to a stable state; And thawing parameters of the fusion network, performing end-to-end joint training on the fusion network, the converged uncertainty estimation network and the parallax estimation network, and guiding the learning of the fusion network by utilizing the converged uncertainty estimation network and the parallax estimation network.
3. The method of claim 2, wherein the end-to-end joint training of the converged network with the converged uncertainty estimation network and the disparity estimation network and guiding learning of the converged network with the converged uncertainty estimation network and the disparity estimation network comprises: Constructing a sample set containing normal sample images and a plurality of failure sample images, and labeling classification labels for each sample image in the sample set to form a label set containing normal, scene environment failure, camera calibration parameter failure and lens physical failure, wherein the failure sample images comprise scene environment failure images, camera calibration parameter failure images and lens physical failure images which are generated in a simulation or are acquired in an actual mode; Dividing the sample set into a training set, a verification set and a test set according to a preset proportion, selecting a neural network architecture as a basis of a fusion network, and initializing network parameters of the neural network architecture; Inputting the training set and the corresponding labels into a fusion network, taking a cross entropy loss function as an optimization target, and iteratively updating network parameters through a back propagation algorithm and an optimizer to minimize the difference between a predicted label and a real label; after each training period is finished, evaluating the performance index of the fusion network by using the verification set, triggering early stop system to terminate training when the loss value of the verification set is stably converged or the accuracy reaches a preset threshold, and storing the model weight with the best performance on the verification set; and performing final performance evaluation on the trained fusion network by using the test set, comprehensively checking generalization capability and robustness of the fusion network on unknown data, and finally obtaining the deployable fusion network after the fusion network meets preset deployment standards and passes the evaluation.
4. The method according to claim 1, wherein after the step of inputting the uncertainty prediction result and the parallax prediction result into a fusion network and outputting a corresponding failure detection result, further comprising: When the failure detection result is one or more of scene environment failure, camera calibration parameter failure and lens physical failure, a corresponding response mechanism is automatically triggered, and a targeted correction or compensation strategy is executed.
5. The method of claim 4, wherein when the failure detection result is one or more of a scene environment failure, a camera calibration parameter failure, and a lens physical failure, the method automatically triggers a corresponding response mechanism to execute a targeted corrective measure or compensation policy, and comprises: when the failure detection result is that the scene environment fails, triggering an environment optimization strategy or starting a data re-acquisition process to acquire an effective image of a new view angle, prompting a user that the current scene environment is limited through voice broadcasting and visual interface characters, and adjusting illumination conditions or changing shooting positions to ensure data effectiveness.
6. The method of claim 4, wherein when the failure detection result is one or more of a scene environment failure, a camera calibration parameter failure, and a lens physical failure, the method automatically triggers a corresponding response mechanism to execute a targeted corrective measure or compensation policy, and comprises: When the failure detection result is that the camera calibration parameters fail, an online automatic camera parameter recalibration program is started immediately; If the error still exceeds a preset threshold after recalibration or the calibration process fails, starting an optical distortion correction program to perform software compensation; If the software correction is invalid, the camera is automatically switched to the standby camera, and a hardware state checking task is triggered.
7. The method of claim 4, wherein when the failure detection result is one or more of a scene environment failure, a camera calibration parameter failure, and a lens physical failure, the method automatically triggers a corresponding response mechanism to execute a targeted corrective measure or compensation policy, and comprises: When the failure detection result is that the lens is physically failed, triggering a hardware maintenance flow, and informing a user that the lens needs to be overhauled or replaced in a voice and text prompting dual mode.
8. A failure detection system for parallax prediction, applied to a convolutional neural network, the convolutional neural network comprising a matching cost calculation network, an uncertainty estimation network, a parallax estimation network and a fusion network, the system comprising: The computing module is used for acquiring a left image and a right image of the target scene, inputting the left image and the right image into a matching cost computing network and outputting a corresponding matching cost result; the prediction module is used for inputting the matching cost result into an uncertainty estimation network and a parallax estimation network respectively and outputting a corresponding uncertainty prediction result and a parallax prediction result; The detection module is used for inputting the uncertainty prediction result and the parallax prediction result into a fusion network and outputting a corresponding failure detection result, wherein the failure detection result comprises failure prevention, scene environment failure, camera calibration parameter failure and lens physical failure.
9. An electronic device comprising a memory for storing a computer program and a processor for executing the computer program stored in the memory to cause the processor to perform the steps of the method according to any one of claims 1 to 7.
10. A computer readable storage medium, characterized in that it has stored thereon a program which, when run, is adapted to carry out the steps of the method according to any of claims 1 to 7.

Description

Failure detection method, system, equipment and medium for parallax prediction Technical Field The invention relates to the technical field of unmanned aerial vehicle application, in particular to a parallax prediction failure detection method, a parallax prediction failure detection system, parallax prediction failure detection equipment and a parallax prediction failure detection medium. Background In the three-dimensional scene reconstruction method of the unmanned aerial vehicle scene, binocular vision is a key branch of a stereo matching technology. The traditional stereo matching method mainly comprises four stages of matching cost calculation, cost aggregation, parallax calculation and parallax optimization. In recent years, stereo matching algorithms based on deep learning are excellent in performance, but errors still exist in the predicted disparity map. To evaluate the quality of the predictions, the uncertainty of the network output is estimated by modeling the distribution of parallax errors, thereby scoring the confidence or uncertainty of each parallax value. However, the problem of domain migration between the real scene and the training scene and the imaging quality caused by the external environment (such as large-area sky interference) may cause the parallax prediction of the whole picture to be completely disabled. In this case, the parallax prediction of the entire image may be completely erroneous, resulting in failure to screen out reliable prediction results by a preset confidence threshold. Therefore, there is a need for an effective failure detection method for parallax prediction to solve the above problems. Disclosure of Invention The present invention has been made in view of the above problems, and provides a failure detection method, system, device and medium of parallax prediction that overcomes or at least partially solves the above problems. To achieve the above and other related objects, the present invention provides a failure detection method of parallax prediction, which is applied to a convolutional neural network, wherein the convolutional neural network includes a matching cost calculation network, an uncertainty estimation network, a parallax estimation network and a fusion network, and the method includes: acquiring a left image and a right image of a target scene, inputting the left image and the right image into a matching cost calculation network, and outputting a corresponding matching cost result; Respectively inputting the matching cost result into an uncertainty estimation network and a parallax estimation network, and outputting a corresponding uncertainty prediction result and a corresponding parallax prediction result; And inputting the uncertainty prediction result and the parallax prediction result into a fusion network, and outputting a corresponding failure detection result, wherein the failure detection result comprises failure prevention, scene environment failure, camera calibration parameter failure and lens physical failure. Optionally, before the steps of obtaining the left graph and the right graph of the target scene, inputting the left graph and the right graph into a matching cost calculation network, and outputting the corresponding matching cost node, the method further includes: Freezing parameters of the fusion network, training an uncertainty estimation network and a parallax estimation network, and enabling the uncertainty estimation network and the parallax estimation network to independently learn and converge to a stable state; And thawing parameters of the fusion network, performing end-to-end joint training on the fusion network, the converged uncertainty estimation network and the parallax estimation network, and guiding the learning of the fusion network by utilizing the converged uncertainty estimation network and the parallax estimation network. Optionally, the performing end-to-end joint training on the converged uncertainty estimation network and the parallax estimation network by the converged uncertainty estimation network and the parallax estimation network, and guiding learning of the converged network by using the converged uncertainty estimation network and the parallax estimation network, includes: Constructing a sample set containing normal sample images and a plurality of failure sample images, and labeling classification labels for each sample image in the sample set to form a label set containing normal, scene environment failure, camera calibration parameter failure and lens physical failure, wherein the failure sample images comprise scene environment failure images, camera calibration parameter failure images and lens physical failure images which are generated in a simulation or are acquired in an actual mode; Dividing the sample set into a training set, a verification set and a test set according to a preset proportion, selecting a neural network architecture as a basis of a fusion network, and initializing n