CN-116052116-B - Automatic parking method based on multi-source information perception and end-to-end deep learning

CN116052116BCN 116052116 BCN116052116 BCN 116052116BCN-116052116-B

Abstract

The invention belongs to the technical field of automatic parking, and discloses an automatic parking method based on multi-source information perception and end-to-end deep learning; the method comprises the steps of sampling four-way fish-eye image data, ultrasonic radar data, steering wheel rotation angle and vehicle speed data in a parking process in real time to construct an initial data set, preprocessing the four-way fish-eye image data in the initial data set to be all-round looking image information data to construct a training sample, constructing and optimizing a CNN-LSTM neural network, wherein CNN is used for processing all-round looking image information data, LSTM is used for processing ultrasonic obstacle distance information data and current driver driving information data, inputting the training sample to the CNN-LSTM neural network to train to obtain a trained end-to-end automatic parking model, and controlling a real vehicle by using the automatic parking model to realize end-to-end automatic parking. The invention solves the problems of low planning precision, low response speed and the like of the existing automatic parking method.

Inventors

JIANG HAOBIN
MA ZHENPENG
MA SHIDIAN

Assignees

江苏大学

Dates

Publication Date: 20260512
Application Date: 20230104

Claims (2)

1. An automatic parking method based on multi-source information perception and end-to-end deep learning is characterized by comprising the following steps: 1) Sampling a parking process at a sampling frequency f to construct an initial data set D, wherein the initial data set is recorded as D= { D 1 ,d 2 ……d i ……}, d i as sampling data of the ith time and is recorded as d i ={Pf i ,Pb i ,Pl i ,Pr i ,left i ,right i ,back i ,r i ,v i },, pf i 、Pb i 、Pl i 、Pr i is an image collected by four-way fisheye cameras arranged under an automobile front engine cover, an automobile tail and left and right rearview mirrors, left i 、right i 、back i is the distance between an automobile and an obstacle measured by ultrasonic radars arranged on the left side, the right side and the rear side of the automobile, r i is the steering wheel angle in the current sampling process, and v i is the automobile wheel speed in the current sampling process; 2) Constructing a training sample D' through the initial data set D; said step 2) comprises the steps of: 2.1 Calibrating the four vehicle-mounted fisheye cameras by Zhang Zhengyou calibration methods to obtain calibration parameters of the four vehicle-mounted fisheye cameras, wherein the calibration parameters comprise an inner parameter and an outer parameter; 2.2 Using the internal parameters and the external parameters to carry out distortion correction on the fish-eye image Pf, pb, pl, pr to obtain correction transformation graphs Pf ', pb', pl ', pr'; 2.3 The correction transformation maps Pf ', pb', pl ', pr' are changed into top views Pf ', pb', pl ', pr'; 2.4 Cutting and splicing the top views Pf ', pb', pl 'and Pr' to obtain a circular splice map P O ; 2.5 Downsampling the look-around mosaic P O to output a fixed-size image P T ; 2.6 Normalized processing is carried out on the image P T to obtain a training image P; 2.7 A training sample D ' is constructed, wherein the training sample D ' is marked as D ' = { D 1 ',d 2 '……d i '……},d i ' and contains ring image frame sequence data, ultrasonic obstacle distance information data and current driver driving information data, the training sample D ' is marked as D i '={P i ,left i ,right i ,back i ,r i ,v i }, and the training Label is marked as Label= { left i ,right i ,back i ,r i ,v i }; 3) Constructing and optimizing a CNN-LSTM neural network; said step 3) comprises the steps of: 3.1 The method comprises the steps of) building a CNN-LSTM neural network, wherein the neural network comprises CNN, LSTM and a characteristic fusion layer, the CNN part consists of 5 convolution layers, 5 pooling layers and 1 full-connection layer, the LSTM part consists of 2 full-connection layers, 1 pooling layer and 20 LSTM units, and the characteristic fusion layer part consists of 1 fusion layer and 2 full-connection layers; 3.2 Optimizing the neural network with an Adam optimizer; 4) Training a neural network to obtain an automatic parking driving model; 5) And (5) utilizing an automatic parking model to control the real vehicle so as to realize automatic parking.
2. The automatic parking method based on multi-source information sensing and end-to-end deep learning according to claim 1, wherein the step 4) comprises the steps of: 4.1 Inputting a training sample D'; 4.2 Calculating a mean square error MSE, wherein the calculation formula is as follows: pred is the prediction result in the training process, is a 2-dimensional tensor with the same size as a training Label Label, i, j is the coordinate of a row and a column, and n is the batch size; 4.3 If MSE > mean square error threshold a) go to step 4.1) to continue training, otherwise go to step 5).

Description

Automatic parking method based on multi-source information perception and end-to-end deep learning Technical Field The invention relates to the technical field of automatic parking, in particular to an automatic parking method based on multi-source information perception and end-to-end deep learning. Background With the progress of society, the living standard of residents in China is continuously improved, and automobiles become important vehicles which are indispensable for people, but due to the fact that the vehicles are continuously increased, the urban congestion is a big problem in life, and great inconvenience is caused to the traveling of people. The automatic parking automobile is also called an unmanned parking automobile, a computer parking automobile or a wheel type mobile robot, and is an intelligent automobile for realizing unmanned parking through a computer system. The parking motion strategy based on path planning and path tracking is combined with the kinematic constraint of the vehicle to plan a parking path, and then a control algorithm is used for path tracking. (the sensor is used for estimating the parking space and the vehicle body posture, and then the optimal parking path is planned). Although the conventional planning method (ex. circular arc method) can meet the requirements, the constraint conditions are correspondingly increased, the solving process becomes more complex, and the planning precision and response speed are reduced. Disclosure of Invention Aiming at the problems, in order to further improve the parking precision and response speed of automatic parking, the invention provides an automatic parking method based on multi-source information perception and end-to-end deep learning to realize end-to-end automatic parking. In order to achieve the purpose, the technical scheme of the invention is as follows, the automatic parking method based on multi-source information perception and end-to-end deep learning comprises the following steps: 1) Sampling a parking process at a sampling frequency f to construct an initial data set D, wherein the initial data set is recorded as D= { D 1,d2……di……},di as sampling data of the ith time and is recorded as di＝{Pfi,Pbi,Pli,Pri,lefti,righti,backi,ri,vi},, pf i、Pbi、Pli、Pri is an image collected by four-way fisheye cameras arranged under an automobile front engine cover, an automobile tail and left and right rearview mirrors, left i、righti、backi is the distance between an automobile and an obstacle measured by ultrasonic radars arranged on the left side, the right side and the rear side of the automobile, r i is the steering wheel angle in the current sampling process, and v i is the automobile wheel speed in the current sampling process; 2) Constructing a training sample D' through the initial data set D; 3) Constructing and optimizing a CNN-LSTM neural network; 4) Training a neural network to obtain an automatic parking driving model; 5) And (5) utilizing an automatic parking model to control the real vehicle so as to realize automatic parking. Further, the step 2) includes the steps of: 2.1 Calibrating the four vehicle-mounted fisheye cameras by Zhang Zhengyou calibration methods to obtain calibration parameters of the four vehicle-mounted fisheye cameras, wherein the calibration parameters comprise an inner parameter and an outer parameter; 2.2 Using the internal parameters and the external parameters to carry out distortion correction on the fish-eye image Pf, pb, pl, pr to obtain correction transformation graphs Pf ', pb', pl ', pr'; 2.3 The correction transformation maps Pf ', pb', pl ', pr' are changed to top views Pf ", pb", pl ", pr"; 2.4 Shearing and splicing the top views Pf ', pb', pl ', pr' to obtain a circular splice map P O; 2.5 Downsampling the look-around mosaic P O to output a fixed-size image P T; 2.6 Normalized processing is carried out on the image P T to obtain a training image P; 2.7 A training sample D ' is constructed, and is marked as D ' = { D 1',d2'……di'……},di ' which contains the cyclic image frame sequence data, the ultrasonic obstacle distance information data and the current driver driving information data, and is marked as D i'＝{Pi,lefti,righti,backi,ri,vi }, and a training Label is marked as Label= { left i,righti,backi,ri,vi }. Further, the step 3) includes the steps of: 3.1 The method comprises the steps of) building a CNN-LSTM neural network, wherein the neural network comprises CNN, LSTM and a characteristic fusion layer, the CNN part consists of 5 convolution layers, 5 pooling layers and 1 full-connection layer, the LSTM part consists of 2 full-connection layers, 1 pooling layer and 20 LSTM units, and the characteristic fusion layer part consists of 1 fusion layer and 2 full-connection layers; 3.2 Using Adam optimizer to optimize the neural network. Further, the step 4) includes the steps of: 4.1 Inputting a training sample D'; 4.2 Calculating a mean square error MSE, wherein the calculation formul