Search

CN-122009210-A - Auxiliary driving method and system based on machine vision

CN122009210ACN 122009210 ACN122009210 ACN 122009210ACN-122009210-A

Abstract

The invention provides an auxiliary driving method and system based on machine vision, which comprises the steps of constructing an original visual data set, establishing a three-dimensional space coordinate system, converting multi-view image data into a three-dimensional scene representation model, extracting semantic features from the three-dimensional scene representation model to construct a layered environment cognition map, identifying key interaction objects in a current driving scene according to interaction layer information in the layered environment cognition map, calculating risk influence weights of the key interaction objects on the driving of a vehicle according to each key interaction object to generate a dynamic risk distribution thermodynamic diagram, generating a candidate driving track set by adopting a multi-objective optimization algorithm based on the dynamic risk distribution thermodynamic diagram and the current driving state of the vehicle, verifying each candidate track, selecting a target track with optimal comprehensive score from the verified candidate tracks, and outputting a control instruction to a vehicle control system. The invention realizes more accurate, flexible and safe auxiliary driving effect and improves driving experience and driving safety.

Inventors

  • XIONG HUI
  • YANG XUESHI

Assignees

  • 深圳市领航者汽车智能技术开发有限公司

Dates

Publication Date
20260512
Application Date
20260203

Claims (10)

  1. 1. A machine vision-based driving assistance method, comprising: acquiring multi-view image data of the surrounding environment of the vehicle through a vehicle-mounted multi-view vision sensor array, synchronously acquiring pose parameters of each sensor, and constructing an original vision data set containing space coordinate information; Performing space-time fusion processing on the original visual data set, establishing a unified three-dimensional space coordinate system based on pose relation among sensors, and converting multi-view image data into a three-dimensional scene representation model with depth information, wherein the three-dimensional scene representation model comprises motion trail prediction information of a dynamic target; Extracting semantic features of road boundaries, lane lines, traffic participants and potential barriers from the three-dimensional scene representation model, and constructing a layered environment cognition map based on the semantic features, wherein the layered environment cognition map comprises a static layer, a dynamic layer and an interaction layer, and the interaction layer is used for representing potential interaction relations among different traffic participants; According to the interaction layer information in the layered environment cognition map, identifying key interaction objects in the current driving scene, calculating risk influence weights of the key interaction objects on the driving of the vehicle according to each key interaction object, and generating a dynamic risk distribution thermodynamic diagram; Generating a candidate running track set considering safety, comfort and efficiency by adopting a multi-objective optimization algorithm based on the dynamic risk distribution thermodynamic diagram and the current running state of the vehicle, and carrying out feasibility verification on each candidate track; And selecting a target track with the optimal comprehensive score from candidate tracks passing the feasibility verification, and outputting a control instruction comprising steering angle, acceleration and braking force distribution to a vehicle control system to realize auxiliary driving control.
  2. 2. The machine vision-based driving assistance method according to claim 1, wherein the step of performing a space-time fusion process on the original vision data set, establishing a unified three-dimensional space coordinate system based on a pose relationship between the sensors, and converting the multi-view image data into a three-dimensional scene representation model having depth information, comprises: selecting a central point of a vehicle body coordinate system as a global reference origin, calculating a space transformation matrix from each sensor coordinate system to the global coordinate system by using pose parameters of each sensor, and establishing a unified three-dimensional space coordinate system so that image data acquired by all sensors can be expressed under the same coordinate frame; Extracting feature points and calculating feature descriptors of images acquired by each vision sensor, performing feature matching between images of adjacent view angles, identifying feature point pairs corresponding to the same spatial physical points, and establishing pixel-level corresponding relations among the images of multiple view angles; Based on the feature point pairs and pose parameters of the corresponding sensors, calculating the space coordinates of each matched feature point in a unified three-dimensional space coordinate system by utilizing a triangulation principle, and carrying out accuracy verification on a depth estimation result according to the geometric consistency of multi-view observation to remove abnormal points which do not meet epipolar geometric constraint; Performing dense expansion on the sparse three-dimensional coordinates of the feature points, establishing parallax association in a dense pixel area of the image by adopting a semi-global stereo matching algorithm, spreading depth information to a continuous surface area with rich textures, and generating three-dimensional point cloud data containing dense depth values; And carrying out time sequence association analysis on the three-dimensional point cloud data, identifying dynamic targets in a scene by tracking point cloud motion modes among continuous frames, extracting motion speed and direction information of each dynamic target, carrying out extrapolation prediction based on short-time historical tracks, and integrating predicted future motion track information into a three-dimensional scene representation model to form a complete scene representation comprising a static structure, the dynamic targets and motion trends thereof.
  3. 3. The machine vision-based driving assistance method according to claim 2, wherein semantic features of road boundaries, lane lines, traffic participants and potential obstacles are extracted from the three-dimensional scene representation model, and a hierarchical environmental awareness map is constructed based on the semantic features, the hierarchical environmental awareness map including a static layer, a dynamic layer and an interaction layer, wherein the interaction layer is used for representing potential interaction relations between different traffic participants, comprising: Inputting the three-dimensional scene representation model into a pre-trained deep semantic segmentation network, marking each space region in the scene by category, identifying and extracting first semantic features of static traffic infrastructures such as road boundaries, lane lines and crosswalks, and second semantic features of traffic participants such as vehicles, pedestrians and bicycles and fixed obstacles, and distributing unique semantic tags and space position information for each identification object; Hierarchical organization is carried out according to attribute differences of the first semantic features and the second semantic features, static elements such as road boundaries, lane lines and traffic signs, which are unchanged in time, are classified into a static layer, traffic participants such as vehicles, pedestrians and bicycles with motion characteristics are classified into a dynamic layer, and current speed, motion direction and predicted track information obtained from a three-dimensional scene representation model are added for each object in the dynamic layer; For each traffic participant in the dynamic layer, analyzing the spatial relationship between the predicted track and the planning path of the vehicle, identifying traffic participants with the possible intersection, convergence or parallelization of the track, marking the identified objects as key interaction objects, and calculating the relative distance, the relative speed, the predicted intersection time and the intersection position between each pair of key interaction objects; Constructing a graph structure representation of an interaction layer based on the spatial relationship and the motion relationship between the key interaction objects, wherein each key interaction object is used as a node of the graph, a directed edge is established between corresponding nodes when a potential interaction relationship exists between the two objects, the directed edge carries an interaction type label and an interaction strength attribute, and the interaction type comprises competitive interaction, collaborative interaction and line-letting interaction; And integrating the static layer, the dynamic layer and the interaction layer to form a multi-level hierarchical environment cognitive map, and establishing an inter-layer association relationship in the map, so that traffic participants in the dynamic layer can be associated with a static layer road area where the traffic participants are located, and the interaction relationship in the interaction layer can be indexed to specific traffic participant objects in the dynamic layer, thereby realizing unified expression of the static environment, the dynamic target and the interaction relationship.
  4. 4. A machine vision based driving assistance method according to claim 3, wherein the step of identifying key interaction objects in the current driving scene according to the interaction layer information in the hierarchical environment cognitive map, calculating risk impact weights of the key interaction objects on the driving of the vehicle for each key interaction object, and generating a dynamic risk distribution thermodynamic diagram comprises: Extracting all traffic participant nodes with interaction relation with the vehicle from the interaction layer of the hierarchical environment cognition map, sorting in descending order according to the interaction intensity attribute of each node, selecting traffic participants with the interaction intensity exceeding a preset threshold as key interaction objects, and acquiring the current position, speed, predicted track and relative motion relation with the vehicle of each key interaction object; For each key interaction object, comprehensively evaluating a plurality of risk factors to calculate the risk influence weight of the key interaction object on the running of the vehicle, wherein the risk factors comprise the minimum predicted distance between the key interaction object and the vehicle, the relative speed, the track crossing angle, the uncertainty coefficient of the object type and the centrality index of the key interaction object in an interaction layer diagram structure, and the centrality index reflects the interaction relation degree of the key interaction object and a plurality of other traffic participants; Establishing a space risk influence area taking the current position of each key interaction object as a center, and determining the shape and the range of the risk influence area according to the risk influence weight, the movement speed and the movement direction of the key interaction object, so that the influence area range in front of the movement direction is larger than that of the side and the rear, and the influence area range of the high-speed movement object is larger than that of the low-speed movement object; building a two-dimensional rasterized space grid around the vehicle, traversing each grid unit in the grid, calculating the overlapping relation between the grid unit and the space risk influence areas of all key interaction objects, and accumulating risk values of a plurality of overlapped risk influence areas by adopting a nonlinear overlapping rule so as to enable the areas where a plurality of high-risk objects are close to generate a risk amplifying effect; And carrying out normalization processing on the calculated risk values of each grid unit, mapping the risk values to a preset color gradient scheme, generating a dynamic risk distribution thermodynamic diagram of a two-dimensional plane, wherein different colors in the dynamic risk distribution thermodynamic diagram represent different risk levels, and carrying out time sequence smoothing processing on the thermodynamic diagram to eliminate inter-frame jump and ensure the time continuity of the thermodynamic diagram.
  5. 5. The machine vision-based driving assistance method according to claim 4, wherein the step of generating a candidate travel track set considering safety, comfort and efficiency by using a multi-objective optimization algorithm based on the dynamic risk distribution thermodynamic diagram and the current travel state of the host vehicle, and performing feasibility verification on each candidate track comprises: Acquiring current running state parameters of the vehicle, including a current position, a running speed, a course angle, a steering wheel angle and acceleration, and determining an expected arrival position and an expected running speed in a planning time domain according to a target path provided by a navigation system, wherein the expected arrival position and the expected running speed are used as boundary conditions and reference targets for track generation; By changing key parameters such as a transverse offset, a longitudinal speed adjustment amount and a track curvature, a parameterized track generation method is adopted to generate a plurality of candidate running tracks on the premise of meeting the kinematic constraint of a vehicle, wherein the candidate running tracks cover various driving intentions including keeping a current lane, changing lanes leftwards and rightwards, accelerating and decelerating and avoiding; Respectively calculating the safety score, the comfort score and the efficiency score of each candidate driving track; The safety score is calculated by sampling discrete points on the track and inquiring risk values of corresponding positions in the dynamic risk distribution thermodynamic diagram, wherein the higher the risk value is, the lower the safety score is, the comfort score is calculated according to the curvature change smoothness degree and the acceleration change smoothness degree of the track, and the efficiency score is calculated according to the approach degree of the average speed and the expected speed of the track and the time efficiency of reaching a target position; The safety score, the comfort score and the efficiency score are weighted and summed to obtain a comprehensive score of each candidate track, the weight coefficient of each score is dynamically adjusted according to the characteristics of the current driving scene, the safety weight is increased in the high-risk scene, the efficiency weight is increased in the smooth road, and the comfort weight is increased in the passenger comfort priority scene; And carrying out feasibility verification on the candidate tracks with higher comprehensive scores, wherein the verification content comprises checking whether the tracks collide with static barriers, whether the traffic rules are violated, whether the steering angular speed and acceleration change rate of the vehicle exceed the physical limits of an executing mechanism, and whether the maximum risk value of the tracks on a dynamic risk distribution thermodynamic diagram exceeds a safety threshold, and only reserving the candidate tracks passing all the feasibility verification to form a final candidate running track set.
  6. 6. The machine vision-based assisted driving method according to claim 5, wherein the step of selecting a target trajectory with an optimal comprehensive score from candidate trajectories passing the feasibility verification and outputting a refined control instruction including a steering angle, acceleration and braking force distribution to the vehicle control system, and implementing the assisted driving control, comprises: sorting all tracks in the candidate running track set passing the feasibility verification according to the comprehensive scores, selecting the track with the highest comprehensive score as a preliminary target track, and comparing the preliminary target track with the historical track selected in the previous control period, if the difference between the preliminary target track and the historical track exceeds a preset stability threshold, introducing a track smooth transition mechanism in a time window, and avoiding frequent jump of track selection; performing time discretization sampling on the determined target track, extracting critical path points on the track according to fixed time intervals in a planning time domain, wherein each critical path point comprises target position coordinates, target course angle and target speed information, and calculating the required transverse displacement and longitudinal speed variation between adjacent critical path points; Based on a vehicle kinematic model, converting the transverse displacement into a front wheel steering angle control instruction, and adopting a control strategy combining feedforward and feedback, wherein the feedforward control calculates a basic steering angle according to the curvature of a target track, and the feedback control corrects the steering angle according to the transverse deviation and the course deviation of the current position of the vehicle and a target path point to generate an accurate steering angle instruction sequence; According to the longitudinal speed variation and the current speed of the vehicle, calculating longitudinal acceleration or deceleration required by realizing target speed, and respectively generating an acceleration control instruction or a braking control instruction according to positive and negative attributes of the acceleration, wherein the acceleration control instruction is realized by adjusting the throttle opening, and the braking control instruction is used for optimally distributing four-wheel braking force according to vehicle load distribution, road adhesion coefficient and anti-lock requirements by calculating required total braking force; And packaging the steering angle instruction sequence, the acceleration control instruction and the braking force distribution instruction to form a complete vehicle control instruction packet, adding priority identification, an execution time stamp and a safety constraint parameter of the instruction, sending the control instruction packet to a steering executing mechanism, a power system and a braking system of the vehicle through a vehicle-mounted communication bus, and realizing accurate control on the motion of the vehicle by the executing mechanisms according to the cooperative action of instruction requirements.
  7. 7. The machine vision-based driving assistance method according to claim 6, wherein when an interaction layer of a hierarchical environment cognitive map is constructed, a space-time interaction potential energy field model is adopted to quantify interaction relations among traffic participants, and an interaction potential energy calculation formula is as follows: Wherein, the The interaction potential energy intensity between the traffic participants i and j at the time t is represented; And Respectively representing the travel speeds of traffic participants i and j at time t; representing the spatial distance between traffic participants i and j; representing a distance regularization constant, and preventing numerical singular when the distance is too small; Representing the angle between the direction of the speed of traffic participant i and the vector connecting i to j; characterizing the alignment degree of the speed direction and the relative position as a direction correlation factor; Representing the latest intersection time of the predicted trajectories of traffic participants i and j; Representing a time attenuation coefficient, and controlling the influence weight of a future interaction event on a current decision; representing the interaction strength scaling factor.
  8. 8. The machine vision-based driving assistance method according to claim 7, wherein when generating the dynamic risk distribution thermodynamic diagram, a multi-source risk propagation model is adopted, and the comprehensive risk value calculation formula of the spatial points is as follows: Wherein, the Representing spatial position N represents the total number of key interaction objects in the current scene; Representing risk impact weights of the nth key interaction object; Representing the basic risk level of the nth key interaction object; representing the spatial position vector of the nth key interaction object at the time t; Representing spatial position Euclidean distance from the object location; the risk influence radius of the nth object is represented and is determined by the type of the object and the motion state; as a function of distance decay, such that risk decreases with increasing distance; A velocity vector representing an nth object; representing the directional risk adjustment function, the value range is [0,1], when the space point is The larger value is taken when the object is positioned at the front in the movement direction of the object, and the smaller value is taken when the object is positioned at the side or the rear.
  9. 9. The machine vision-based driving assistance method according to claim 8, wherein when the candidate travel track set is generated by adopting a multi-objective optimization algorithm, the comprehensive cost function calculation formula is as follows: wherein each sub-objective function is defined as: Wherein, the Representing a trajectory Is the overall cost of (1); representing candidate trajectories, defined as a mapping function of time t to spatial position; 、 、 Weight coefficients respectively representing safety, comfort and efficiency; Representing the safety cost, and obtaining the safety cost by carrying out time integration on the maximum risk value of the track passing region; representing comfort cost, characterized by square integral of track curvature and rate of change thereof; representing an efficiency cost characterized by a square integral of the actual speed deviation from a reference speed; And Respectively representing the starting time and the ending time of the track planning; Representing the spatial position of the track corresponding to the time t; Representing the curvature of the trajectory at time t; Representing the rate of change of curvature over time And Respectively representing a reference speed and an actual track speed; representing a track smoothness regularization term; representing regularization coefficients.
  10. 10. A machine vision-based driving assistance system for performing the machine vision-based driving assistance method according to any one of claims 1 to 9, comprising an in-vehicle multiview sensor and a server; The server is configured to: acquiring multi-view image data of the surrounding environment of the vehicle acquired by a vehicle-mounted multi-view vision sensor array and pose parameters of each sensor, and constructing an original vision data set containing space coordinate information; Performing space-time fusion processing on the original visual data set, establishing a unified three-dimensional space coordinate system based on pose relation among sensors, and converting multi-view image data into a three-dimensional scene representation model with depth information, wherein the three-dimensional scene representation model comprises motion trail prediction information of a dynamic target; Extracting semantic features of road boundaries, lane lines, traffic participants and potential barriers from the three-dimensional scene representation model, and constructing a layered environment cognition map based on the semantic features, wherein the layered environment cognition map comprises a static layer, a dynamic layer and an interaction layer, and the interaction layer is used for representing potential interaction relations among different traffic participants; According to the interaction layer information in the layered environment cognition map, identifying key interaction objects in the current driving scene, calculating risk influence weights of the key interaction objects on the driving of the vehicle according to each key interaction object, and generating a dynamic risk distribution thermodynamic diagram; Generating a candidate running track set considering safety, comfort and efficiency by adopting a multi-objective optimization algorithm based on the dynamic risk distribution thermodynamic diagram and the current running state of the vehicle, and carrying out feasibility verification on each candidate track; And selecting a target track with the optimal comprehensive score from candidate tracks passing the feasibility verification, and outputting a refined control instruction comprising steering angle, acceleration and braking force distribution to a vehicle control system to realize auxiliary driving control.

Description

Auxiliary driving method and system based on machine vision Technical Field The invention relates to the technical field of intelligent automobiles, in particular to a machine vision-based auxiliary driving method and system. Background Computer vision is intended to mimic the manner in which the human visual system works, enabling a computer to process, analyze, and understand image or video data. An intelligent driving system is an innovation of applying computer vision technology to vehicle driving, and aims to perceive and understand road environment through image data collected by a sensor such as a camera during the driving process of the vehicle so as to make decisions more accurately and intelligently, and such a system aims to improve the safety, accuracy and adaptability of driving the vehicle so that the vehicle can drive in a complex traffic environment. However, the existing intelligent driving system is not accurate enough for understanding the surrounding environment, so that the auxiliary driving effect is poor, and even safety risks occur. Disclosure of Invention The invention provides an auxiliary driving method and system based on machine vision based on the problems, and by the aid of the auxiliary driving method and system, more accurate, flexible and safe auxiliary driving effect is realized, and driving experience and driving safety are remarkably improved. In view of this, an aspect of the present invention proposes a machine vision-based driving assistance method, including: acquiring multi-view image data of the surrounding environment of the vehicle through a vehicle-mounted multi-view vision sensor array, synchronously acquiring pose parameters of each sensor, and constructing an original vision data set containing space coordinate information; Performing space-time fusion processing on the original visual data set, establishing a unified three-dimensional space coordinate system based on pose relation among sensors, and converting multi-view image data into a three-dimensional scene representation model with depth information, wherein the three-dimensional scene representation model comprises motion trail prediction information of a dynamic target; Extracting semantic features of road boundaries, lane lines, traffic participants and potential barriers from the three-dimensional scene representation model, and constructing a layered environment cognition map based on the semantic features, wherein the layered environment cognition map comprises a static layer, a dynamic layer and an interaction layer, and the interaction layer is used for representing potential interaction relations among different traffic participants; According to the interaction layer information in the layered environment cognition map, identifying key interaction objects in the current driving scene, calculating risk influence weights of the key interaction objects on the driving of the vehicle according to each key interaction object, and generating a dynamic risk distribution thermodynamic diagram; Generating a candidate running track set considering safety, comfort and efficiency by adopting a multi-objective optimization algorithm based on the dynamic risk distribution thermodynamic diagram and the current running state of the vehicle, and carrying out feasibility verification on each candidate track; And selecting a target track with the optimal comprehensive score from candidate tracks passing the feasibility verification, and outputting a control instruction comprising steering angle, acceleration and braking force distribution to a vehicle control system to realize auxiliary driving control. Optionally, performing space-time fusion processing on the original visual data set, establishing a unified three-dimensional space coordinate system based on pose relation among the sensors, and converting the multi-view image data into a three-dimensional scene representation model with depth information, including the steps of: selecting a central point of a vehicle body coordinate system as a global reference origin, calculating a space transformation matrix from each sensor coordinate system to the global coordinate system by using pose parameters of each sensor, and establishing a unified three-dimensional space coordinate system so that image data acquired by all sensors can be expressed under the same coordinate frame; Extracting feature points and calculating feature descriptors of images acquired by each vision sensor, performing feature matching between images of adjacent view angles, identifying feature point pairs corresponding to the same spatial physical points, and establishing pixel-level corresponding relations among the images of multiple view angles; Based on the feature point pairs and pose parameters of the corresponding sensors, calculating the space coordinates of each matched feature point in a unified three-dimensional space coordinate system by utilizing a triangulation principle, and c