EP-4360064-B1 - SYSTEMS AND METHODS FOR CATEGORIZING IMAGE PIXELS

Inventors

• BRUMBY, Steven P.
• KONTGIS, Caitlin
• KARRA, KRISHNA

Dates

Publication Date

20260506

Application Date

20220623

Claims (16)

1. A method of training a machine learning model to categorize each pixel of an input overhead image, comprising: receiving, by processing circuitry, a plurality of overhead images, each comprising a respective plurality of pixels, wherein each pixel of each overhead image of the plurality of overhead images is designated as being of a particular mapping category of a plurality of mapping categories; and training, using the processing circuitry, a machine learning model to categorize each pixel of the input overhead image using the plurality of overhead images, wherein the training of the machine learning model comprises using boundary information that indicates where boundaries between regions of pixels of different categories are located.
2. The method of claim 1, wherein, for each pixel of each respective overhead image, the designation of the particular mapping category is appended to the respective overhead image; or wherein, for each pixel of each respective overhead image, the designation of the particular mapping category is stored at a database.
3. The method of claim 1, wherein the plurality of overhead images are annotated to comprise an indication of the boundary information.
4. The method of claim 1, wherein training the machine learning model further comprises biasing each of the respective plurality of mapping categories based on a frequency of occurrence of the mapping category in the plurality of overhead images.
5. The method of claim 1, wherein each overhead image of the plurality of overhead images comprises a satellite image or other aerial image that is of a predefined dimensioned area of the planet.
6. A method of generating a map by predicting to which mapping category of a plurality of mapping categories each pixel of a plurality of overhead images belongs using a machine learning model trained according to the method of any of claims 1-5, comprising: inputting the plurality of overhead images to a trained machine learning model; determining, using the trained machine learning model, for each pixel of the plurality of overhead images, to which mapping category of the plurality of mapping categories the pixel belongs; and generating, using processing circuitry, the map of a geographic area associated with the plurality of overhead images based on the plurality of overhead images and on the determined categories, wherein the machine learning model is trained by: receiving, by processing circuitry, a plurality of training overhead images, each comprising a respective plurality of pixels, wherein each pixel of each training overhead image of the plurality of training overhead images is designated as being of a particular mapping category of a plurality of mapping categories; and training, using the processing circuitry, a machine learning model to categorize each pixel of an input overhead image using the plurality of training overhead images.
7. The method of claim 6, wherein: the plurality of overhead images were captured at a plurality of times during a time period; generating the map of the geographic area comprises generating the map of the geographic area over the time period; and the map comprises a temporal axis.
8. The method of claim 7, wherein: generating the map comprises generating a composite image based on the plurality of overhead images together and the determined categories; the plurality of overhead images were captured at a plurality of times during a time period; the composite image comprises a plurality of pixels each of which respectively corresponds to a pixel included in each of the plurality of overhead images; and generating the composite image of the geographic area comprises: identifying, for each pixel of the plurality of overhead images of the geographic area, a particular mapping category that is determined most frequently, or determined to be associated with a highest confidence score, by the trained machine learning model over the time period; and for each respective pixel of the composite image, causing the pixel to be represented as being of the same particular mapping category that is identified for the pixel of the plurality of overhead images that corresponds to the pixel of the composite image.
9. The method of claim 6, wherein generating the map comprises formulating a class weighted mode parameter based on: a probability associated with each mapping category; and a desired weight of a particular mapping category of the plurality of mapping categories with respect to the other mapping categories of the plurality of mapping categories.
10. The method of claim 9, wherein: the plurality of mapping categories comprise at least grass, flooded vegetation, crops, bare ground, snow/ice and clouds; and formulating the class weighted mode comprises assigning higher weights to each of the grass and flooded vegetation categories as compared to the crops, bare ground, snow/ice, and clouds categories.
11. The method of claim 6, wherein generating the map comprises: generating a plurality of maps of a plurality of geographic areas associated with the plurality of overhead images based on the plurality of overhead images and on the determined categories; and compositing the plurality of maps together to generate a composite map comprising each of the plurality of geographic areas.
12. The method of claim 9, wherein formulating the class weighted mode parameter based on a desired weight of a particular mapping category of the plurality of mapping categories with respect to the other mapping categories of the plurality of mapping categories comprises assigning a first set of weights for the plurality of mapping categories to a particular geographic region as compared to other geographic regions.
13. The method of claim 6, further comprising: receiving feedback information from the trained machine learning model; determining, based on the feedback information, that one or more mapping category designations for respective pixels of at least one of the plurality of training overhead images need to be updated; updating the plurality of training overhead images by updating the one or more mapping category designations for the respective pixels of the at least one of the plurality of training overhead images; and updating the trained machine learning model using the updated plurality of training overhead images.
14. The method of claim 6, wherein: the trained machine learning model is implemented at particular equipment; the particular equipment comprises the processing circuitry that is configured to receive the feedback information from the trained machine learning model; the particular equipment comprises one or more sensors configured to capture the additional training overhead image; and the processing circuitry of the particular equipment is configured to perform the updating of the plurality of training overhead images with the additional training overhead image and the updating of the trained machine learning model using the updated plurality of training overhead images.
15. A system comprising the processing circuitry configured to execute the steps of the method of any of claims 1-5; processing circuitry configured to: receive a plurality of overhead images, each comprising a respective plurality of pixels, wherein each pixel of each overhead image of the plurality of overhead images is designated as being of a particular mapping category of a plurality of mapping categories; and train a machine learning model to categorize each pixel of the input overhead image using the plurality of overhead images.
16. A system of generating a map by predicting to which mapping category of a plurality of mapping categories each pixel of a plurality of overhead images belongs, comprising the processing circuitry configured to execute the steps of claims 6-14.

Description

Cross-Reference to Related Applications This application claims the benefit 35 U.S.C. §119(e) of U.S. Provisional Application No. 63/214,174 filed June 23, 2021. Background The present disclosure is directed towards categorizing image pixels. More particularly, the present disclosure is directed towards machine learning models for categorizing image pixels such as images of land use or land cover. Summary Over the last several decades, human-induced land use/land cover (LULC) change has affected ecosystems across the globe. LULC maps are foundational geospatial data products needed by analysts and decision makers across governments, civil society, industry, and finance to monitor global environmental change and measure risk to sustainable livelihoods and development. While there is an unprecedented amount of Earth observation imagery data available to track change, analyses must be automated in order to scale globally. In one approach, individual pixels of an image may be classified, e.g., based on a color of the pixel, and global maps having a resolution ranging from 30 meters per pixel to 500 meters per pixel are generated. However, there is a strong need for an improved high-level, automated geospatial analysis solution that can better convert the abundant pixel data into actionable insights for non-geospatial experts. The invention is defined by the appended claims. To overcome these problems, systems and methods are provided herein for training a machine learning model to categorize each pixel of an input overhead image by receiving, by processing circuitry, a plurality of overhead images, each comprising a respective plurality of pixels, where each pixel of each overhead image of the plurality of overhead images is designated as being of a particular mapping category of a plurality of mapping categories. The processing circuitry may be configured to train a machine learning model to categorize each pixel of the input overhead image using the plurality of overhead images. In addition, systems and methods are provided herein for generating a map by predicting to which mapping category of a plurality of mapping categories each pixel of a plurality of overhead images belongs. Such systems and method may include processing circuitry configured to input the plurality of overhead images to the trained machine learning model, and determine, using the trained machine learning model, for each pixel of the plurality of overhead images, to which mapping category of the plurality of mapping categories the pixel belongs. The processing circuitry may generate the map of a geographic area associated with the plurality of overhead images based on the plurality of overhead images and on the determined categories. In some embodiments, generating the map comprises generating the map at a resolution of 10 meters per pixel. In the provided systems and methods for training the machine learning model, for each pixel of each respective overhead image, the designation of the particular mapping category may be appended to the respective overhead image and/or the designation of the particular mapping category may be stored at a database. In some embodiments, the training of the machine learning model comprises using boundary information that indicates where boundaries between regions of pixels of different categories are located, where the overhead image may be annotated to comprise an indication of the boundary information (e.g., a dense labeling of pixels and neighboring pixels). Such boundary information may enable the machine learning model to learn patterns and characteristics of boundaries between certain regions of pixels of different categories, and explore spatial and spectral features to more accurately and efficiently categorize pixels. In some embodiments, training the machine learning model further comprises biasing each of the respective plurality of mapping categories based on, for example, a frequency of occurrence of the mapping category in the plurality of overhead images. In this way, certain under-represented mapping categories may be accounted for to correct any imbalances in the training dataset. In some embodiments, each overhead image of the plurality of overhead images comprises a satellite image or other aerial image that is of a predefined dimensioned area of the planet. In some aspects of this disclosure, training the machine learning model may further comprise using three-dimensional terrain information to train the machine learning model. In some embodiments, the machine learning model may be trained using any suitable types of imagery, e.g., visible or infrared optical images, synthetic aperture radar images, medium wave or long wave thermal infrared images, hyperspectral images, or any other suitable images, or any combination thereof. In the provided systems and methods for determining, using the trained machine learning model, to which mapping category of the plurality of mapping categories the