EP-4740113-A1 - GENERATION AND IMPLEMENTATION OF GEOSPATIAL WORKFLOWS

Abstract

Implementations are described herein for automatically generating multimodal geospatial workflows for accomplishing geospatial tasks. In various implementations, a natural language request may be processed based on generative model(s) such as LLM(s) to generate workflow output tokens that identify high-level actions for completing a geospatial task conveyed in the natural language request. First data indicative of the high-level actions may be processed using one or more of the generative models to generate dataset output tokens that identify responsive dataset(s) that likely contain data responsive to the geospatial task. Second data indicative of both the high-level actions and the responsive dataset(s) may be processed based on one or more of the generative models to generate data manipulation output tokens that identify data manipulation instructions for assembling data from the responsive dataset(s) into a response that fulfills the geospatial task.

Inventors

• GUPTA, ANANYA
• MURPHY, Gearoid
• GONCHARUK, ARTEM
• GUPTA, AKSHINA
• ZHANG, HAOYU
• WALKER, ADRIAN

Assignees

• X Development LLC

Dates

Publication Date

20260513

Application Date

20240823

Claims (20)

1. 1. A method implemented using one or more processors and comprising: processing a natural language request based on one or more generative models to generate workflow output tokens that identify high-level actions for completing a geospatial task conveyed in the natural language request; processing first data indicative of the high-level actions for completing the geospatial task based on one or more of the generative models to generate dataset output tokens that identify one or more responsive datasets that likely contain data responsive to the geospatial task conveyed in the natural language request; processing second data indicative of both the high-level actions for completing the geospatial task and the one or more responsive datasets based on one or more of the generative models to generate data manipulation output tokens that identify data manipulation instructions for assembling data from the one or more responsive datasets into a response that fulfills the geospatial task; causing the data manipulation instructions to be executed using the one or more responsive datasets to generate the response that fulfills the geospatial task; and causing the response that fulfills the geospatial task to be rendered at one or more computing devices.
2. 2. The method of claim 1, wherein the natural language request is processed based on a different generative model than the first data.
3. 3. The method of claim 1 or 2, wherein the natural language request is processed based on a different generative model than the second data.
4. 4. The method of any of the preceding claims, wherein the first data is processed based on a different generative model than the second data.
5. 5. The method of any of the preceding claims, wherein the first data further comprises metadata about a plurality of candidate datasets from which the one or more responsive datasets are identified.
6. 6. The method of claim 5, wherein the metadata includes human-curated content describing one or more of the candidate datasets.
7. 7. The method of claim 5, wherein the metadata includes data indicative of a database schema of one or more of the candidate datasets.
8. 8. The method of any of the preceding claims, wherein the dataset output tokens indicate, for a plurality candidate datasets from which the one or more responsive datasets are identified, respective measures of usefulness for performing the high-level actions for completing the geospatial task conveyed in the natural language request.
9. 9. The method of any of the preceding claims, wherein the one or more responsive datasets include a first overhead digital depiction of a geographic area that includes annotations identifying land-based features of the geographic area.
10. 10. The method of claim 9, wherein the first overhead digital depiction comprises a raster image.
11. 11. The method of claim 10, wherein the raster image is captured by a satellite or a drone.
12. 12. The method of claim 9, wherein the first overhead digital depiction comprises a vector-based map of the geographic area.
13. 13. The method of claim 9, wherein the one or more responsive datasets comprises a second overhead digital depiction of at least part of the geographic area.
14. 14. The method of claim 13, wherein the data manipulation instructions comprise instructions to join the first and second overhead digital depictions.
15. 15. The method of claim 13, wherein the data manipulation instructions comprise instructions to overlay one of the first and second overhead digital depictions in relation to the other.
16. 16. The method of any of the preceding claims, wherein the data manipulation instructions comprise source code composed in a high-level programming language, wherein the source code is configured to be executed to obtain data from the one or more responsive datasets and assemble the response to the natural language request.
17. 17. The method of any of the preceding claims, wherein the one or more responsive datasets comprise at least first and second responsive databases.
18. 18. The method of claim 17, wherein the data manipulation instructions comprise instructions to join data from first and second responsive databases.
19. 19. The method of claim 18, wherein the instructions to join data from the first and second responsive databases comprise structured query language (SQL) instructions.
20. 20. The method of claim 18, wherein the instructions to join data from the first and second responsive databases comprise source code composed in a high-level programming language.

Description

GENERATION AND IMPLEMENTATION OF GEOSPATIAL WORKFLOWS Background [0001] Generative models are types of machine learning models — often taking the form of and/or described as large language models (LLMs) — that can perform various tasks, such as language generation, machine translation, and question-answering, to name a few. These generative models are often trained on enormous amounts of diverse data including data from, but not limited to, webpages, electronic books, software code, electronic news articles, and so forth. Accordingly, these generative models leverage the underlying data on which they were trained in performing these various tasks. For instance, in performing a language generation task, generative models such as LLMs can process a natural language (NL) based input that is received from a client device, and generate output in NL or another form that is responsive to the NL based input and that is to be rendered at the client device. [0002] Geospatial datasets such as maps, high-elevation imagery, geographic database(s), etc., may be used in combination with each other to implement what will be referred to herein as “geospatial workflows" to accomplish a variety of different geospatial tasks, such as responding to geospatial queries from users. Data from two or more geospatial datasets can be used to determine, for instance, numbers of features such as trees in a particular geographic region, average tree canopy coverage for playgrounds in regions having different climates, methane leaks having manmade origins (e.g., in urban areas or areas in which oil or gas have been extracted), and real estate pricing trends in areas near large bodies of water, to name a few. SUMMARY [0003] Designing geospatial workflows can be challenging. A complex geospatial task may require a correspondingly complex geospatial workflow. An expert such as a geospatial data scientist or software engineer may need to spend significant time and resources designing such a complex geospatial workflow. Moreover, a complex geospatial workflow may not necessarily be scalable beyond its original purpose. This is especially true where the geospatial task the workflow is meant to accomplish is narrowly scoped. [0004] Implementations are described herein for automatically generating multimodal geospatial workflows for accomplishing geospatial tasks. More particularly, but not exclusively, implementations are described herein for processing various modalities of data using sequences of generative models (e.g., LLMs) to identify: (i) a geospatial task (e.g., from a natural language request); (ii) high-level actions needed to accomplish the geospatial task; (iii) dataset(s) that contain data responsive to the geospatial task; and (iv) data manipulation instructions for assembling data from the responsive dataset(s) into responsive data that fulfills the geospatial task. Once the data manipulation instructions are assembled, they may be executed to generate the responsive data, which in turn may be rendered at one or more output devices. [0005] In various implementations, a method may be implemented using one or more processors and may include: processing a natural language request based on one or more generative models, such as one or more single modal or multimodal large language models (LLMs), to generate workflow output tokens that identify high-level actions for completing a geospatial task conveyed in the natural language request; processing first data indicative of the high-level actions for completing the geospatial task based on one or more of the generative models to generate dataset output tokens that identify one or more responsive datasets that likely contain data responsive to the geospatial task conveyed in the natural language request; processing second data indicative of both the high-level actions for completing the geospatial task and the one or more responsive datasets based on one or more of the generative models to generate data manipulation output tokens that identify data manipulation instructions for assembling data from the one or more responsive datasets into a response that fulfills the geospatial task; causing the data manipulation instructions to be executed using the one or more responsive datasets to generate the response that fulfills the geospatial task; and causing the response that fulfills the geospatial task to be rendered at one or more computing devices. [0006] In various implementations, the natural language request may be processed based on a different generative model than the first data. In various implementations, the natural language request ma be processed based on a different generative model than the second data. In various implementations, the first data is processed based on a different generative model than the second data. [0007] In various implementations, the first data includes metadata about a plurality of candidate datasets from which the one or more responsive datasets are id