US-20260127895-A1 - LANE LINE MARKING TYPE ESTIMATION AND MARKING TYPE CHANGE DETECTION USING TEMPORAL SEMANTIC SEGMENTATION INFORMATION

US20260127895A1US 20260127895 A1US20260127895 A1US 20260127895A1US-20260127895-A1

Abstract

An apparatus includes a memory for storing image data; and processing circuitry in communication with the memory. The processing circuitry is configured to obtain a current set of one or more camera images from a current time, calculate lane marking confidence values for two or more lane marking types at various positions in a scene captured by the images, and determine the lane marking type for each position by comparing these confidence values with previously stored confidence values associated with the same positions. The apparatus then outputs the lane marking type for each position in the scene.

Inventors

Markus Petersson
Adam Aili

Assignees

QUALCOMM INCORPORATED

Dates

Publication Date: 20260507
Application Date: 20241101

Claims (20)

1 . An apparatus for processing image data, the apparatus comprising: a memory for storing the image data; and processing circuitry in communication with the memory, wherein the processing circuitry is configured to: obtain a current set of one or more camera images of the image data from a current time; calculate respective lane marking confidence values for two or more lane marking types at a plurality of positions in a scene captured by the current set of one or more camera images; determine a lane marking type for each of the plurality of positions based on a comparison of the respective lane marking confidence values with previously-stored lane marking confidence values associated with the plurality of positions in the scene; and output the lane marking type for each of the plurality of positions in the scene.
2 . The apparatus of claim 1 , wherein the processing circuitry is further configured to: determine a location of a lane marking change from a first lane marking type to a second lane marking type; and output the location of the lane marking change.
3 . The apparatus of claim 2 , wherein to output the location of the lane marking change, the processing circuitry is configured to: output the location of the lane marking change location indicating a first change type from solid lane markings to dashed lane markings; or output the location of the lane marking change indicating a second change type from dashed lane markings to solid lane markings.
4 . The apparatus of claim 1 , wherein to calculate the respective lane marking confidence values for the two or more lane marking types at the plurality of positions in the scene captured by the current set of one or more camera images, the processing circuitry is further configured to: apply semantic segmentation to a single current image corresponding to the current set of one or more camera images from the current time to generate: a lane marking object located at each of the plurality of positions in the scene captured by the current set of one or more camera images; a first confidence value indicating a probability the lane marking object located at each of the plurality of positions in the scene corresponds to a first one of the two or more lane marking types; and a second confidence value indicating the probability the lane marking object located at each of the plurality of positions in the scene corresponds to a second one of the two or more lane marking types; and wherein to determine the lane marking type for each of the plurality of positions, the processing circuitry is further configured to: determine the lane marking type for each of the plurality of positions based on a comparison of the first confidence value and the second confidence value with the previously-stored lane marking confidence values associated with the plurality of positions in the scene.
5 . The apparatus of claim 1 , wherein the processing circuitry is further configured to: generate camera features from the current set of one or more camera images corresponding to the two or more lane marking types at the plurality of positions in the scene captured by the current set of one or more camera images; and project the camera features into a birds-eye-view (BEV) image space; and wherein to calculate the respective lane marking confidence values for the two or more lane marking types at the plurality of positions in the scene captured by the current set of one or more camera images, the processing circuitry is configured to: apply semantic segmentation to the BEV image space to generate the respective lane marking confidence values for the two or more lane marking types.
6 . The apparatus of claim 1 : wherein the plurality of positions in the scene captured by the current set of one or more camera images includes one or more occluded lane markings; and wherein the processing circuitry is further configured to: output a predicted lane marking type for the one or more occluded lane markings based on the comparison of the respective lane marking confidence values corresponding to the one or more occluded lane markings with previously-stored lane marking confidence values associated with the plurality of positions in the scene.
7 . The apparatus of claim 1 , wherein the processing circuitry is further configured to: update position information of a vehicle relative to the lane marking type output for each of the plurality of positions in the scene; and discard previously-stored lane marking confidence values associated with the plurality of positions in the scene determined to be located behind the vehicle based on the position information as updated for the vehicle.
8 . The apparatus of claim 1 , wherein the processing circuitry is further configured to: match the respective lane marking confidence values calculated for the two or more lane marking types at the plurality of positions in the scene captured by the current set of one or more camera images with the previously-stored lane marking confidence values associated with the plurality of positions in the scene using ego motion of a vehicle that captured the set of one or more camera images.
9 . The apparatus of claim 1 , wherein to output the lane marking type for each of the plurality of positions in the scene includes the processing circuitry configured to output one of: the lane marking type indicating a first change type to attention markings preceding a toll gate; the lane marking type indicating a second change type to crosswalk markings preceding a crosswalk; the lane marking type indicating a third change type to construction markings preceding a construction zone; or the lane marking type indicating a fourth change type to tunnel markings preceding a tunnel.
10 . The apparatus of claim 1 , wherein the processing circuitry is further configured to: detect a vehicle initiating a maneuver to change lanes or overtake; and output a determination whether the maneuver is permissible based on the lane marking type output for at least one of the plurality of positions in the scene.
11 . The apparatus of claim 1 , wherein the processing circuitry and the memory are part of an advanced driver assistance system (ADAS).
12 . The apparatus of claim 1 , wherein the processing circuitry is configured to use the lane marking type output for each of the plurality of positions in the scene to control a vehicle.
13 . The apparatus of claim 1 , wherein the apparatus further comprises: one or more cameras affixed to a vehicle configured to capture the current set of one or more camera images from the current time; and wherein the one or more cameras affixed to the vehicle capture a forward view of an environment surrounding the vehicle.
14 . A method of processing image data comprising: obtaining a current set of one or more camera images of the image data from a current time; calculating respective lane marking confidence values for two or more lane marking types at a plurality of positions in a scene captured by the current set of one or more camera images; determining a lane marking type for each of the plurality of positions based on a comparison of the respective lane marking confidence values with previously-stored lane marking confidence values associated with the plurality of positions in the scene; and outputting the lane marking type for each of the plurality of positions in the scene.
15 . The method of claim 14 : determining a location of a lane marking change from a first lane marking type to a second lane marking type; and outputting the location of the lane marking change.
16 . The method of claim 14 , further comprising: outputting a location of the lane marking change location indicating a first change type from solid lane markings to dashed lane markings; or outputting the location of the lane marking change indicating a second change type from dashed lane markings to solid lane markings.
17 . The method of claim 14 , wherein calculating the respective lane marking confidence values for the two or more lane marking types at the plurality of positions in the scene captured by the current set of one or more camera images, further comprises: applying semantic segmentation to a single current image corresponding to the current set of one or more camera images from the current time and generating: a lane marking object located at each of the plurality of positions in the scene captured by the current set of one or more camera images; a first confidence value indicating a probability the lane marking object located at each of the plurality of positions in the scene corresponds to a first one of the two or more lane marking types; and a second confidence value indicating the probability the lane marking object located at each of the plurality of positions in the scene corresponds to a second one of the two or more lane marking types; and wherein determining the lane marking type for each of the plurality of positions, further comprises: determining the lane marking type for each of the plurality of positions based on a comparison of the first confidence value and the second confidence value with the previously-stored lane marking confidence values associated with the plurality of positions in the scene.
18 . The method of claim 14 , further comprising: generating camera features from the current set of one or more camera images corresponding to the two or more lane marking types at the plurality of positions in the scene captured by the current set of one or more camera images; and projecting the camera features into a birds-eye-view (BEV) image space; and wherein calculating the respective lane marking confidence values for the two or more lane marking types at the plurality of positions in the scene captured by the current set of one or more camera images, includes: applying semantic segmentation to the BEV image space to generate the respective lane marking confidence values for the two or more lane marking types.
19 . The method of claim 14 : wherein the plurality of positions in the scene captured by the current set of one or more camera images includes one or more occluded lane markings; and wherein the method further comprises: outputting a predicted lane marking type for the one or more occluded lane markings based on the comparison of the respective lane marking confidence values corresponding to the one or more occluded lane markings with previously-stored lane marking confidence values associated with the plurality of positions in the scene.
20 . A non-transitory computer-readable medium storing instructions that, when executed, cause processing circuitry to: obtain a current set of one or more camera images from a current time; calculate respective lane marking confidence values for two or more lane marking types at a plurality of positions in a scene captured by the current set of one or more camera images; determine a lane marking type for each of the plurality of positions based on a comparison of the respective lane marking confidence values with previously-stored lane marking confidence values associated with the plurality of positions in the scene; and output the lane marking type for each of the plurality of positions in the scene.

Description

TECHNICAL FIELD This disclosure relates to sensor systems, including image projections for use in advanced driver-assistance systems (ADAS). BACKGROUND An autonomous driving vehicle is a vehicle that is configured to sense the environment around the vehicle, such as the existence and location of objects, and to operate without human control. An autonomous driving vehicle may include cameras that produce image data that may be analyzed to determine the existence and location of other objects around the autonomous driving vehicle. A vehicle having advanced driver-assistance systems (ADAS) is a vehicle that includes systems which may assist a driver in operating the vehicle, such as parking or driving the vehicle. SUMMARY The present disclosure generally relates to techniques and devices for detecting and estimating lane marking locations, lane marking types, and lane marking change locations using temporal semantic segmentation information. For example, aspects of the disclosure may obtain one or more current camera images capturing a forward view of a lane of travel and locate lane markings and calculate confidence values for the lane markings in relation to a subject vehicle within the lane of travel. For instance, a processing system may calculate current confidence values for solid lane markings and confidence values for dashed lane markings within the forward view along the lane of travel in relation to the subject vehicle. The current confidence values may be stored and subsequently referenced as prior frame confidence values. The current and prior confidence values may be compared to evaluate whether the type of lane markings and then also to determine whether a change in lane marking type has occurred (e.g., to determine whether the lane markings have transitioned from solid to dashed or dashed to solid), as well as determine a point at which the transition occurs in relation to the subject vehicle. Previously calculated confidence values may be referenced and utilized in conjunction with new lane marking information in real time, until such time that prior lane marking confidence values are no longer within the forward view along the lane of travel in relation to the subject vehicle, at which point the prior lane marking confidence values are no longer relevant and therefore no longer needed. Because prior lane marking confidence values are utilized as part of the determination of lane marking type, as well as a determination whether the lane markings type has transitioned from solid to dashed or dashed to solid and where such a transition occurs, the determinations may be made with high accuracy from the temporal semantic segmentation information associated with the prior lane marking confidence values, even when lane markings within the forward view become partially or fully occluded and are therefore indeterminate within a current frame of the forward view captured in relation to the subject vehicle. In one example, an apparatus for processing image data, the apparatus includes a memory for storing the image data; and processing circuitry in communication with the memory. According to such an example, the processing circuitry is configured to obtain a current set of one or more camera images of the image data from a current time. According to certain examples, the apparatus calculates respective lane marking confidence values for two or more lane marking types at a plurality of positions in a scene captured by the current set of one or more camera images. In at least one example, the apparatus determines a lane marking type for each of the plurality of positions based on a comparison of the respective lane marking confidence values with previously-stored lane marking confidence values associated with the plurality of positions in the scene. According to such examples, the apparatus outputs the lane marking type for each of the plurality of positions in the scene. In another example, a method includes obtaining a current set of one or more camera images of the image data from a current time. In one example, the method includes calculating respective lane marking confidence values for two or more lane marking types at a plurality of positions in a scene captured by the current set of one or more camera images. According to certain examples, the method includes determining a lane marking type for each of the plurality of positions based on a comparison of the respective lane marking confidence values with previously-stored lane marking confidence values associated with the plurality of positions in the scene. In at least one example, the method includes outputting the lane marking type for each of the plurality of positions in the scene. In another example, a non-transitory computer-readable medium stores instructions that, when executed, cause processing circuitry to obtain a current set of one or more camera images from a current time. In one example, the instructions cause the processing circuitry