US-12626518-B2 - Method to detect lane segments for creating high definition maps

US12626518B2US 12626518 B2US12626518 B2US 12626518B2US-12626518-B2

Abstract

Disclosed herein are system, method, and computer program product embodiments for detecting lane segments in an image for creating high-definition (HD) maps. A neural network can be used to classify a pixel of an image of a location as belonging to a driving lane. If the pixel belongs to a first boundary line of the driving lane, it is labeled accordingly. Based on the labeled pixel and one or more additional labeled pixels as part of the first boundary line of the driving lane, a first line drawing of the first boundary line of the driving lane is constructed. A lane segment based on a combination of the first line drawing of the first boundary line of the driving lane and a second line drawing of a second boundary line of the driving lane can be created.

Inventors

Willibald Brems
Francesco Ferroni

Assignees

FORD GLOBAL TECHNOLOGIES, LLC

Dates

Publication Date: 20260512
Application Date: 20221202

Claims (15)

1 . A method, comprising: classifying, by a neural network on one or more computing devices, a pixel of an image including a birds eye view (BEV) of a location as belonging to a driving lane; assigning, based on the pixel being determined to belong to a first boundary line of the driving lane by the neural network on the one or more computing devices, a label indicating the pixel as part of the first boundary line of the driving lane; constructing, by the one or more computing devices and based on the labeled pixel and one or more additional labeled pixels as part of the first boundary line of the driving lane, a first line drawing of the first boundary line of the driving lane; creating, using the one or more computing devices, a lane segment based on a combination of the first line drawing of the first boundary line of the driving lane and a second line drawing of a second boundary line of the driving lane; and determining, by a second neural network on the one or more computing devices, whether to connect the lane segment to a second lane segment in an intersection, wherein the second neural network is separated from the neural network, wherein the determining whether to connect the lane segment to the second lane segment in the intersection further includes, generating, by the second neural network on the one or more computing devices, an existence probability for an identified potential connection between the lane segment and the second lane segment; and verifying, by the second neural network on the one or more computing devices, the identified potential connection is a valid connection based on a comparison of the existence probability to a predetermined threshold.
2 . The method of claim 1 , further comprising: partitioning, using the one or more computing devices, the image into a plurality of parcels, wherein each parcel of the plurality of parcels comprises a same predetermined number of pixels, wherein the pixel and the one or more additional labeled pixels are selected from a parcel of the plurality of parcels.
3 . The method of claim 1 , further comprising: aligning, by the one or more computing devices, the first line drawing of the first boundary line and the second line drawing of the second boundary line of the driving lane with a detected lane stop.
4 . The method of claim 1 , further comprising: predicting, by the neural network on the one or more computing devices, a coarse driving direction for each of the labeled pixel and the one or more additional labeled pixels; and estimating, based on the predicted coarse driving directions of the labeled pixel and the one or more additional labeled pixels by the neural network on the one or more computing devices, a direction for each of the first line drawing of the first boundary line and the second line drawing of the second boundary line of the driving lane.
5 . The method of claim 4 , wherein predicting the coarse driving direction for each of the labeled pixel and the one or more additional labeled pixels further comprises: generating, by the neural network on the one or more computing devices, probabilities for a plurality of direction bins defining a plurality of directions separated by at least one increment; and identifying, by the neural network on the one or more computing devices, the coarse driving direction based on a comparison of two direction bins of the plurality of direction bins with the highest probabilities.
6 . The method of claim 1 , wherein the second neural network is configured to operate outside of an image space and treat lanes and roads as a graph.
7 . A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, causes the one or more processors to perform operations comprising: classifying, by a neural network, a pixel of an image including a birds eye view (BEV) of a location as belonging to a driving lane; assigning, based on the pixel being determined to belong to a first boundary line of the driving lane by the neural network, a label indicating the pixel as part of the first boundary line of the driving lane; constructing, based on the labeled pixel and one or more additional labeled pixels as part of the first boundary line of the driving lane, a first line drawing of the first boundary line of the driving lane; creating a lane segment based on a combination of the first line drawing of the first boundary line of the driving lane and a second line drawing of a second boundary line of the driving lane; and determining, by a second neural network, whether to connect the lane segment to a second lane segment in an intersection, wherein the second neural network is separated from the neural network, wherein the determining whether to connect the lane segment to the second lane segment in the intersection further includes, generating, by the second neural network, an existence probability for an identified potential connection between the lane segment and the second lane segment; and verifying, by the second neural network, the identified potential connection is a valid connection based on a comparison of the existence probability to a predetermined threshold.
8 . The non-transitory computer-readable medium of claim 7 , wherein the operations further comprise: partitioning the image into a plurality of parcels, wherein each parcel of the plurality of parcels comprises a same predetermined number of pixels, wherein the pixel and the one or more additional labeled pixels are selected from a parcel of the plurality of parcels.
9 . The non-transitory computer-readable medium of claim 7 , wherein the operations further comprise: aligning the first line drawing of the first boundary line and the second line drawing of the second boundary line of the driving lane with a detected lane stop.
10 . The non-transitory computer-readable medium of claim 7 , wherein the operations further comprise: predicting, by the neural network, a coarse driving direction for each of the labeled pixel and the one or more additional labeled pixels; and estimating, based on the predicted coarse driving directions of the labeled pixel and the one or more additional labeled pixels by the neural network, a direction for each of the first line drawing of the first boundary line and the second line drawing of the second boundary line of the driving lane.
11 . The non-transitory computer-readable medium of claim 10 , wherein predicting a coarse driving direction for each of the labeled pixel and the one or more additional labeled pixels further comprises: generating, by the neural network, probabilities for a plurality of direction bins defining a plurality of directions separated by at least one increment; and identifying, by the neural network, the coarse driving direction based on a comparison of two direction bins of the plurality of direction bins with the highest probabilities.
12 . A system, comprising: one or more processors; and a memory communicatively coupled to the one or more processors, wherein the memory stores instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: classifying, by a neural network, a pixel of an image including a birds eye view (BEV) of a location as belonging to a driving lane, assigning, based on the pixel being determined to belong to a first boundary line of the driving lane by the neural network, a label indicating the pixel as part of the first boundary line of the driving lane, constructing, based on the labeled pixel and one or more additional labeled pixels as part of the first boundary line of the driving lane, a first line drawing of the first boundary line of the driving lane, creating a lane segment based on a combination of the first line drawing of the first boundary line of the driving lane and a second line drawing of a second boundary line of the driving lane, and determining, by a second neural network, whether to connect the lane segment to a second lane segment in an intersection, wherein the second neural network is separated from the neural network, wherein the determining whether to connect the lane segment to the second lane segment in the intersection further includes, generating, by the second neural network, an existence probability for an identified potential connection between the lane segment and the second lane segment; and verifying, by the second neural network, the identified potential connection is a valid connection based on a comparison of the existence probability to a predetermined threshold.
13 . The system of claim 12 , wherein the operations further comprise: aligning the first line drawing of the first boundary line and the second line drawing of the second boundary line of the driving lane with a detected lane stop.
14 . The system of claim 12 , wherein the operations further comprise: predicting, by the neural network, a coarse driving direction for each of the labeled pixel and the one or more additional labeled pixels; and estimating, based on the predicted coarse driving directions of the labeled pixel and the one or more additional labeled pixels by the neural network, a direction for each of the first line drawing of the first boundary line and the second line drawing of the second boundary line of the driving lane.
15 . The system of claim 14 , wherein predicting a coarse driving direction for each of the labeled pixel and the one or more additional labeled pixels further comprises: generating, by the neural network, probabilities for a plurality of direction bins defining a plurality of directions separated by at least one increment; and identifying, by the neural network, the coarse driving direction based on a comparison of two direction bins of the plurality of direction bins with the highest probabilities.

Description

BACKGROUND Autonomous vehicles (AVs) can be configured to operate autonomously and navigate their environment with little or no human input. In order to do so safely, AVs may use high definition (HD) maps in combination with various sensors that collect real-time data about roadways, surrounding vehicles, and other objects or actors they may encounter. HD maps are typically highly precise roadmaps that may contain information about lane boundaries, lane stops, intersections, pedestrian crossings, stop signs, and traffic signs. AVs use the information in HD maps to assist with localization and to navigate their environment on a lane-level basis. The time and costs to generate HD maps remain a significant challenge. Current solutions for automating HD map creation often rely on neural network models that operate in the image space to detect lane segments in images. Although these models can predict straight lane segments, they often fail to detect lane segments in an intersection. Intersections often include different lanes that can be in the same location or overlap. However, for models that operate in the image space (e.g., classical object detection and segmentation), there can be no overlap between lanes or other objects. Accordingly, what is needed are improved approaches for detecting lane segments for creating HD maps. SUMMARY Aspects disclosed herein generally relate to methods, systems, and computer program products for detecting lane segments used to create high-definition (HD) maps. Aspects related to classifying, by a neural network on one or more computing devices, a pixel of an image including a birds eye view (BEV) of a location as belonging to a driving lane. Based on the pixel being determined to belong to a first boundary line of the driving lane by the neural network on the one or more computing devices, a label is assigned indicating the pixel as part of the first boundary line of the driving lane. Based on the labeled pixel and one or more additional labeled pixels as part of the first boundary line of the driving lane, a first line drawing of the first boundary line of the driving lane is constructed. A lane segment based on a combination of the first line drawing of the first boundary line of the driving lane and a second line drawing of a second boundary line of the driving lane is created. BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings are incorporated herein and form a part of the specification. FIG. 1 illustrates an exemplary autonomous vehicle system, in accordance with aspects of the disclosure. FIG. 2 illustrates an exemplary architecture for a vehicle, in accordance with aspects of the disclosure. FIG. 3 illustrates an exemplary architecture for a Light Detection and Ranging (“lidar”) system, in accordance with aspects of the disclosure. FIG. 4 illustrates a flow chart of an example method for detecting lane segments used to create HD maps, in accordance with aspects of the disclosure. FIG. 5 illustrates an exemplary lane detection model, in accordance with aspects of the disclosure. FIG. 6 illustrates example input channels for the exemplary lane detection model, in accordance with aspects of the disclosure. FIG. 7 illustrates the basic model architecture for the exemplary lane detection model, in accordance with aspects of the disclosure. FIG. 8 illustrates the model heads of the exemplary lane detection model, in accordance with aspects of the disclosure. FIG. 9 illustrates a flow chart of an example method for obtaining feature information in a BEV image, in accordance with aspects of the disclosure. FIG. 10 illustrates an example method for estimating the driving direction of the boundary lines of a driving lane based on outputs produced by the model heads of the exemplary lane detection model, in accordance with aspects of the disclosure. FIG. 11 illustrates an example method for creating lane segments based on outputs produced by the model heads of the exemplary lane detection model, in accordance with aspects of the disclosure. FIG. 12 illustrates a flow chart of an example method for predicting lane segments in an intersection using a lane graph prediction model, in accordance with aspects of the disclosure. FIG. 13 illustrates an exemplary architecture for the lane graph prediction model, in accordance with aspects of the disclosure. FIG. 14 illustrates an example output of the lane graph prediction model, in accordance with aspects of the disclosure. FIG. 15 illustrates an example output of the lane graph prediction model for one exemplary lane segment in an intersection, in accordance with aspects of the disclosure. FIG. 16 illustrates an exemplary result of the example method for detecting lane segments used to create HD maps, in accordance with aspects of the disclosure. FIG. 17 illustrates an example computer system useful for implementing various embodiments. In the drawings, like reference numbers generally indicate identical or similar elements. Ad