US-12620222-B2 - Systems and methods for extracting surface markers for aircraft navigation

Abstract

A method comprises capturing, with a vehicle vision sensor, a color image of a landing site including landing surface markers; converting the color image to a gray scale image; and performing multi-scale-binarization to detect multiple edges of the gray scale image and produce binary images. The method determines contours of edges of the binary images having closed shapes, detects closed shapes of contours of edges having four corners, and verifies whether four-sided candidate contours are valid as potential landing surface markers. If more than one contour is associated with a valid ID within a surface marker library, then the contour within the smallest window size is selected. If multiple contours with the same window size can be associated with a valid ID, then a mean of corresponding corners of multiple contours is computed. The method then performs corner refinement of valid four-sided candidate contours identified as potential landing surface markers.

Inventors

• Vibhor L. Bageshwar
• Ashwin Ganapathy
• Vijay Venkataraman
• Romi Srivastava

Assignees

• HONEYWELL INTERNATIONAL INC.

Dates

Publication Date

20260505

Application Date

20231109

Priority Date

20230926

Claims (19)

1. 1 . A method comprising: capturing, with a vision sensor on a vehicle, a color image of a landing site that includes one or more landing surface markers; converting the color image to a gray scale image; performing multi-scale-binarization to detect multiple edges of the gray scale image and produce a plurality of binary images having differing sizes, based on the gray scale image; wherein multiple windows of different sizes are used to detect edges in the gray scale image, and within a given window, an adaptive threshold is used to slide the window across the gray scale image to produce a complete edge map for the given window; determining contours of edges of the binary images that have closed shapes; detecting any closed shapes of the contours of edges that have four corners by a process comprising: selecting the contours of edges that exceed a user-selected threshold for perimeter size; retaining the selected contours that have four sides; determining a minimum corner separation of the contours that have four sides; eliminating any contours that are within a user-selected threshold for distance from an edge of a respective binary image; and retaining any remaining contours as four-sided candidate contours; verifying whether the four-sided candidate contours are valid as potential landing surface markers by a process comprising: warping each four-sided candidate contour into a fixed size square contour that is tested for sufficient variance of an image intensity value by computing a standard deviation of the image intensity value, and if the standard deviation of the image intensity value is above a user-selected threshold, then converting the fixed size square contour into a binary image to determine a candidate bit pattern within the contour; wherein if an encoded bit error corresponding to a border of each candidate bit pattern is below a user-selected threshold, then accepting the candidate bit pattern for further processing; and determining if bits of the candidate bit pattern, other than its border bits, match any standard bit patterns within a surface marker library for a given landing site at a vertiport, and if there is a match within a user-selected bit error threshold, then accepting the candidate bit pattern as a valid ID; wherein if more than one contour is associated with a valid ID within the surface marker library, then selecting the contour within a smallest window size and ignoring all other contours; wherein if multiple contours within a same window size can be associated with a valid ID within the surface marker library, then computing a mean of corresponding corners of the multiple contours; and performing corner refinement of valid four-sided candidate contours identified as potential landing surface markers by a process comprising: using local gradients to move corners of each four-sided candidate contour to a sharpest transition point.
2. 2 . The method of claim 1 , further comprising: determining if a detected ID within a contour is in a list of valid IDs within the surface marker library for the given landing site; if the detected ID is in the list of valid IDs, accepting the contour with the detected ID as valid for further processing; if the detected ID is not in the list of valid IDs, ignoring the contour with the detected ID and proceeding to a next contour with a detected ID; wherein if a valid detected ID has a contour size above a first user-selected threshold, then the contour with the valid detected ID is accepted for further processing; wherein if a valid detected ID has a contour size below a second user-selected threshold, then the contour with this valid detected ID is rejected; wherein for any contours with valid detected IDs with a contour size between the first and second user-selected thresholds, the contour with the valid detected ID having a largest size is accepted and all other valid IDs are rejected.
3. 3 . The method of claim 2 , wherein a combination of contours with valid detected IDs are used to determine a correct landing site at a vertiport with multiple landing sites to avoid landing site ambiguity, when one or more nearby landing sites have common markers.
4. 4 . The method of claim 1 , wherein the vehicle comprises an unmanned aircraft systems (UAS) vehicle, an uncrewed aerial vehicle (UAV), or an urban air mobility (UAM) vehicle.
5. 5 . The method of claim 1 , wherein the one or more landing surface markers comprise one or more Aruco codes.
6. 6 . A system comprising: at least one vision sensor mounted on a vehicle; at least one processor operatively coupled to the at least one vision sensor; wherein the at least one processor includes program instructions, executable by the at least one processor, to perform a method comprising: capturing, with the at least one vision sensor, a color image of a landing site that includes one or more landing surface markers; converting the color image to a gray scale image; performing multi-scale-binarization to detect multiple edges of the gray scale image and produce a plurality of binary images having differing sizes, based on the gray scale image; wherein multiple windows of different sizes are used to detect edges in the gray scale image, and within a given window, an adaptive threshold is used to slide the window across the gray scale image to produce a complete edge map for the given window; determining contours of edges of the binary images that have closed shapes; detecting any closed shapes of the contours of edges that have four corners by a process comprising: selecting the contours of edges that exceed a user-selected threshold for perimeter size; retaining the selected contours that have four sides; determining a minimum corner separation of the contours that have four sides; eliminating any contours that are within a user-selected threshold for distance from an edge of a respective binary image; and retaining any remaining contours as four-sided candidate contours; verifying whether the four-sided candidate contours are valid as potential landing surface markers by a process comprising: warping each four-sided candidate contour into a fixed size square contour that is tested for sufficient variance of an image intensity value by computing a standard deviation of the image intensity value, and if the standard deviation of the image intensity value is above a user-selected threshold, then converting the fixed size square contour into a binary image to determine a candidate bit pattern within the contour; wherein if an encoded bit error corresponding to a border of each candidate bit pattern is below a user-selected threshold, then accepting the candidate bit pattern for further processing; and determining if bits of the candidate bit pattern, other than its border bits, match any standard bit patterns within a surface marker library for a given landing site at a vertiport, and if there is a match within a user-selected bit error threshold, then accepting the candidate bit pattern as a valid ID; wherein if more than one contour is associated with a valid ID within the surface marker library, then selecting the contour within a smallest window size and ignoring all other contours; wherein if multiple contours within a same window size can be associated with a valid ID within the surface marker library, then computing a mean of corresponding corners of the multiple contours; and performing corner refinement of valid four-sided candidate contours identified as potential landing surface markers by a process comprising: using local gradients to move corners of each four-sided candidate contour to a sharpest transition point.
7. 7 . The system of claim 6 , wherein the at least one processor includes further program instructions, executable by the at least one processor, to further perform a method comprising: determining if a detected ID within a contour is in a list of valid IDs within the surface marker library for the given landing site; if the detected ID is in the list of valid IDs, accepting the contour with the detected ID as valid for further processing; if the detected ID is not in the list of valid IDs, ignoring the contour with the detected ID and proceeding to a next contour with a detected ID; wherein if a valid detected ID has a contour size above a first user-selected threshold, then the contour with the valid detected ID is accepted for further processing; wherein if a valid detected ID has a contour size below a second user-selected threshold, then the contour with this valid detected ID is rejected; wherein for any contours with valid detected IDs with a contour size between the first and second user-selected thresholds, the contour with the valid detected ID having a largest size is accepted and all other valid IDs are rejected.
8. 8 . The system of claim 6 , wherein the vehicle comprises an unmanned aircraft systems (UAS) vehicle, an uncrewed aerial vehicle (UAV), or an urban air mobility (UAM) vehicle.
9. 9 . The system of claim 6 , wherein the one or more landing surface markers comprise one or more Aruco codes.
10. 10 . A system comprising: at least one vision sensor mounted on a vehicle; at least one processor onboard the vehicle and operatively coupled to the at least one vision sensor; and a navigation system onboard the vehicle and operatively coupled to the at least one processor; wherein the at least one processor hosts a set of program modules operative for extracting landing site surface marker information for use in navigation of the vehicle, the program modules comprising: a detect markers function module, which is operative to receive an input image captured by the at least one vision sensor and convert the input image to a gray scale image; a detect marker candidates function module, called from the detect markers function module, and operative to receive the gray scale image from the detect markers function module, wherein the detect marker candidates function module is operative to determine contours and corresponding corners from the gray scale image that satisfy user-selected criteria on number of corners and contour length, to detect potential candidates for markers; an identify marker candidates function module operative to receive the contours and corners detected from the detect marker candidates function module, check their validity for a potential marker valid ID within a surface marker library, return those contours and corners that correspond to a valid ID, and return a corresponding ID number; a filter detected markers function module operative to filter out multiple detections for a same marker, and retain a marker that is detected at a smallest window size for a given ID; and a corner refinement function module operative to refine a detected corner by aligning the detected corner with an actual marker corner as closely as possible; wherein extracted landing site surface marker information is sent to the navigation system for further processing to provide guidance for use during takeoff or landing of the vehicle.
11. 11 . The system of claim 10 , wherein the detect marker candidates function module is further operative to perform multi-scale-binarization to detect multiple edges of the gray scale image and produce a plurality of different edges within the same gray scale image.
12. 12 . The system of claim 11 , wherein multiple windows of different sizes are used to detect edges in the gray scale image, and within a given window, an adaptive threshold is used as the window slides across the gray scale image to produce a complete edge map of the gray scale image using the window.
13. 13 . The system of claim 11 , wherein the detect marker candidates function module is further operative to: determine contours of edges of binary images that have closed shapes; and detect any closed shapes of the contours of edges that have four corners by a process that comprises: selecting the contours of edges that exceed a user-selected threshold for perimeter size; retaining the selected contours that have four sides; determining a minimum corner separation of the contours that have four sides; eliminating any contours that are within a user-selected threshold for distance from an edge of a respective binary image; and retaining any remaining contours as four-sided candidate contours.
14. 14 . The system of claim 13 , wherein the identify marker candidates function module is further operative to verify whether the four-sided candidate contours are valid as potential landing surface markers by a process that comprises: warping each four-sided candidate contour into a fixed size square contour that is tested for sufficient variance of an image intensity value by computing a standard deviation of the image intensity value, and if the standard deviation of the image intensity value is above a user-selected threshold, then converting the fixed size square contour into a binary image to determine a candidate bit pattern within the contour; wherein if an encoded bit error corresponding to a border of each candidate bit pattern is below a user-selected threshold, then accepting the candidate bit pattern for further processing; and determining if bits of the candidate bit pattern, other than its border bits, match any standard bit patterns within a surface marker library for a given landing site at a vertiport, and if there is a match within a user-selected bit error threshold, then accepting the candidate bit pattern as a valid ID.
15. 15 . The system of claim 14 , wherein if more than one contour is associated with a valid ID within the surface marker library, the filter detected markers function module is further operative to select the contour within a smallest window size and ignore all other contours.
16. 16 . The system of claim 14 , wherein if multiple contours within a same window size can be associated with a valid ID within the surface marker library, then the filter detected markers function module is further operative to compute a mean of corresponding corners of the multiple contours.
17. 17 . The system of claim 14 , wherein the corner refinement function module is further operative to perform corner refinement of valid four-sided candidate contours identified as potential landing surface markers by using local gradients to move corners of each four-sided candidate contour to a sharpest transition point.
18. 18 . The system of claim 14 , wherein the potential landing surface markers comprise one or more Aruco codes.
19. 19 . The system of claim 10 , wherein the vehicle comprises an unmanned aircraft systems (UAS) vehicle, an uncrewed aerial vehicle (UAV), or an urban air mobility (UAM) vehicle.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of Indian Provisional Application No. 202311064467, filed on Sep. 26, 2023, same title, the contents of which are incorporated herein in their entirety. BACKGROUND Uncrewed aircraft systems (UAS) applications in the national airspace (NAS) include delivery, sampling, surveillance, and passenger transport. The UAS flights can be broken into various segments or phases including take-off, ascent to cruise, cruise, descent to landing, and landing. The cruise segment can include mission segments where the UAS loiters or performs other tasks. The UAS landing sites include allocated landing zones such as vertiports on the ground, on tops of buildings, or on moving vehicles such as trucks or ships. UAS navigation systems must satisfy stringent requirements to enable autonomous landing at sites such as vertiports to support these applications. These navigation systems must operate in both global navigation satellite system (GNSS) available and GNSS-denied environments. Further, certain UAS operators have requirements that vertiports are unpowered or minimally powered, so ground or surface infrastructure to support navigation systems cannot include powered features or GNSS ground stations. Hereafter, the term “surface” will refer to vertiports located on the ground, or tops of buildings, or on moving vehicles. As a result, UAS navigation systems require sensor alternatives to GNSS, to aid the navigation systems in landing zones that rely on unpowered features. A camera mounted on a UAS is one such sensor that can be used in combination with Aruco codes on the surface to aid the UAS navigation system during the landing segment. The Aruco codes are unpowered and can be located within the Final Approach and Takeoff (FATO) area of a vertiport. These codes provide onboard cameras with known surface features to compute measurements for the navigation system. SUMMARY A method comprises capturing, with a vision sensor on a vehicle, a color image of a landing site that includes one or more landing surface markers; converting the color image to a gray scale image; and performing multi-scale-binarization to detect multiple edges of the gray scale image and produce a plurality of binary images having differing sizes, based on the gray scale image. Multiple windows of different sizes are used to detect edges in the gray scale image, and within a given window, an adaptive threshold is used to slide the window across the gray scale image to produce a complete edge map for the given window. The method further comprises determining contours of edges of the binary images that have closed shapes; and detecting any closed shapes of the contours of edges that have four corners by a process comprising: selecting the contours of edges that exceed a user-selected threshold for perimeter size; retaining the selected contours that have four sides; determining a minimum corner separation of the contours that have four sides; eliminating any contours that are within a user-selected threshold for distance from an edge of a respective binary image; and retaining any remaining contours as four-sided candidate contours. The method also comprises verifying whether the four-sided candidate contours are valid as potential landing surface markers by a process that comprises warping each four-sided candidate contour into a fixed size square contour that is tested for sufficient variance of an image intensity value by computing a standard deviation of the image intensity value, and if the standard deviation of the image intensity value is above a user-selected threshold, then converting the fixed size square contour into a binary image to determine a candidate bit pattern within the contour. If an encoded bit error corresponding to a border of each candidate bit pattern is below a user-selected threshold, then accepting the candidate bit pattern for further processing. The method determines if bits of the candidate bit pattern, other than its border bits, match any standard bit patterns within a surface marker library for a given landing site at a vertiport, and if there is a match within a user-selected bit error threshold, then the candidate bit pattern is accepted as a valid ID. If more than one contour is associated with a valid ID within the surface marker library, then the method selects the contour within a smallest window size and ignores all other contours. If multiple contours within a same window size can be associated with a valid ID within the surface marker library, then the method computes a mean of corresponding corners of the multiple contours. The method also performs corner refinement of valid four-sided candidate contours identified as potential landing surface markers by a process that comprises using local gradients to move corners of each four-sided candidate contour to a sharpest transition point. BRIEF DESCRIPTION OF THE DRAWINGS Features of the present i