US-12625871-B1 - Efficient joins using a sliding join window

US12625871B1US 12625871 B1US12625871 B1US 12625871B1US-12625871-B1

Abstract

A system and method for performing relational style joins on large datasets is described. The system may include an observability platform configured to receive a request to perform a user query on a dataset. The query includes one or more filters to identify one or more spans in a trace. The dataset includes events and identifiers that may be retrieved based on the identified one or more spans in the trace, where the events have associated timestamps. A join window, dynamically sized based on the number of event identifiers, may be generated to process a list of generated data segments to perform one or more relational join operations on the dataset based on the one or more filters in the query, in memory. Relational join operations may be performed to generate a result set provided as a response to the request for display with data visualizations on the observability platform.

Inventors

Ian Wilkes

Assignees

Hound Technology, Inc.

Dates

Publication Date: 20260512
Application Date: 20241212

Claims (20)

1 . A computer-implemented method comprising: receiving a request to perform a query in association with a dataset on an observability platform, the request including user input defining the query, wherein the query includes one or more filters to identify one or more spans in a trace; retrieving the dataset including a number of event identifiers based on the one or more spans identified in the trace, the dataset comprising a plurality of events having associated timestamps, wherein the dataset is distributed across a plurality of nodes, each node having a data store; generating a join window that is dynamically sized based on the number of event identifiers in the dataset; generating a list of data segments based on the retrieved dataset, each data segment comprising a number of data rows retrieved from the data store of a node from the plurality of nodes; storing each data segment from the list of data segments in a single table: using the join window, processing the list of data segments stored in the single table by reading each data row of the single table into the join window to perform one or more relational join operations on the dataset based on the one or more filters in the query, wherein a join is completed whenever a match is found by comparing the data row being read to other data rows in the join window; generating a result set comprising completed joins based on the processed list of data segments; and providing the result set as a response to the request for display on the observability platform.
2 . The computer-implemented method of claim 1 , the method further comprising: sorting the list of data segments stored in the single table based on the associated timestamps of the plurality of events; and executing the one or more relational join operations on the dataset upon processing the sorted list of data segments to perform the query.
3 . The computer-implemented method of claim 2 , further comprising: determining one or more executed join operations includes particular data segments from a neighboring node; and retrieving the particular data segments from the neighboring node into memory as part of the join window.
4 . The computer-implemented method of claim 1 , wherein the one or more relational join operations are performed in memory on the list of the data segments for the dataset based on identified one or more join key values in the query using the join window.
5 . The computer-implemented method of claim 1 , wherein the one or more relational join operations are performed using an application programming interface call to a web service communicatively coupled to a data analysis engine.
6 . The computer-implemented method of claim 1 , wherein the result set of the query comprises a plurality of data values that detail the one or more spans in the trace based on the one or more filters in the query.
7 . The computer-implemented method of claim 5 , wherein providing the result set as the response to the query comprises: generating a user interface comprising a plurality of data visualizations of the result set, the user interface generated on the observability platform; receiving a selection of one of a plurality of data values in the user interface on the observability platform; and providing for display in the user interface the one or more spans in the trace based on the selection.
8 . A system comprising one or more processors and memory operably coupled with the one or more processors, wherein the memory stores instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform operations including: receiving a request to perform a query in association with a dataset on an observability platform, the request including user input defining the query, wherein the query includes one or more filters to identify one or more spans in a trace; retrieving the dataset including a number of event identifiers based on the one or more spans identified in the trace, the dataset comprising a plurality of events having associated timestamps, wherein the dataset is distributed across a plurality of nodes, each node having a data store; generating a join window that is dynamically sized based on the number of event identifiers in the dataset; generating a list of data segments based on the retrieved dataset, each data segment comprising a number of data rows retrieved from the data store of a node from the plurality of nodes; storing each data segment from the list of data segments in a single table: using the join window, processing the list of data segments stored in the single table by reading each data row of the single table into the join window to perform one or more relational join operations on the dataset based on the one or more filters in the query, wherein a join is completed whenever a match is found by comparing the data row being read to other data rows in the join window; generating a result set comprising completed joins based on the processed list of data segments; and providing the result set as a response to the request for display on the observability platform.
9 . The system of claim 8 , wherein the operations further comprise: sorting the list of data segments stored in the single table based on the associated timestamps of the plurality of events; and executing the one or more relational join operations on the dataset upon processing the sorted list of data segments to perform the query.
10 . The system of claim 9 , wherein the operations further comprise: determining one or more executed join operations includes particular data segments from a neighboring node; and retrieving the particular data segments from the neighboring node into memory as part of the join window.
11 . The system of claim 8 , wherein the one or more relational join operations are performed in memory on the list of the data segments for the dataset based on identified one or more join key values in the query using the join window.
12 . The system of claim 8 , wherein the one or more relational join operations are performed using an application programming interface call to a web service communicatively coupled to a data analysis engine.
13 . The system of claim 8 , wherein the result set of the query comprises a plurality of data values that detail the one or more spans in the trace based on the one or more filters in the query.
14 . The system of claim 13 , wherein providing the result set as the response to the query comprises: generating a user interface comprising a plurality of data visualizations of the result set, the user interface generated on the observability platform; receiving a selection of one of the plurality of data values in the user interface on the observability platform; and providing for display in the user interface the one or more spans in the trace based on the selection.
15 . A non-transitory computer-readable storage media storing one or more sequences of instructions which, when executed using one or more processors, cause the one or more processors to execute a method comprising: receiving a request to perform a query in association with a dataset on an observability platform, the request including user input defining the query, wherein the query includes one or more filters to identify one or more spans in a trace; retrieving the dataset including a number of event identifiers based on the one or more spans identified in the trace, the dataset comprising a plurality of events having associated timestamps, wherein the dataset is distributed across a plurality of nodes, each node having a data store; generating a join window that is dynamically sized based on the number of event identifiers in the dataset; generating a list of data segments based on the retrieved dataset, each data segment comprising a number of data rows retrieved from the data store of a node from the plurality of nodes; storing each data segment from the list of data segments in a single table: using the join window, processing the list of data segments stored in the single table by reading each data row of the single table into the join window to perform one or more relational join operations on the dataset based on the one or more filters in the query, wherein a join is completed whenever a match is found by comparing the data row being read to other data rows in the join window; generating a result set comprising completed joins based on the processed list of data segments; and providing the result set as a response to the request for display on the observability platform.
16 . The non-transitory computer-readable storage media of claim 15 , wherein the method further comprises: sorting the list of data segments stored in the single table based on the associated timestamps of the plurality of events; and executing the one or more relational join operations on the dataset upon processing the sorted list of data segments to perform the query.
17 . The non-transitory computer-readable storage media of claim 16 , wherein the method further comprises: determining one or more executed join operations includes particular data segments from a neighboring node; and retrieving the particular data segments from the neighboring node into memory as part of the join window.
18 . The non-transitory computer-readable storage media of claim 15 , wherein the one or more relational join operations are performed on the list of the data segments for the dataset based on identified one or more join key values in the query using the join window in-memory.
19 . The non-transitory computer-readable storage media of claim 18 , wherein the result set of the query comprises a plurality of data values that detail the one or more spans in the trace based on the one or more filters in the query.
20 . The non-transitory computer-readable storage media of claim 15 , providing the result set as the response to the query comprises: generating a user interface comprising a plurality of data visualizations of the result set, the user interface generated on the observability platform; receiving a selection of one of a plurality of data values in the user interface on the observability platform; and providing for display in the user interface the one or more spans in the trace based on the selection.

Description

BACKGROUND The present disclosure relates to systems and methods for analyzing large data sets to generate joins. In particular, the present disclosure relates to systems and methods for efficient joins using a sliding join window in an observability platform. In recent years, tracking and optimizing performance of Internet-enabled software applications has generated larger and larger data sets. Various application performance monitoring (APM) solutions have emerged to enable organizations to improve user experiences by tracking key software application performance metrics using monitoring software and telemetry data. With the abundance of data available, and with increasing complexity and structures, it is often difficult to identify the root causes that may affect the user experience of a software application, such as latency. Specifically, as software applications are updated to fix bugs, release new features, and/or deploy applications on more complex structures, such as distributed systems or cloud computing platforms, the user experience may be negatively affected for reasons that are harder to determine. As such, there is a persistent need to improve the tools used to investigate and analyze the large amounts of data used to observe the performance of software applications. It would be beneficial to enable users of application performance monitoring solutions to better analyze data through relational queries to identify the root causes of performance issues. SUMMARY The present disclosure relates to an observability platform that enables performance monitoring data to be analyzed using relational style queries to join data items. An observability platform receives user-supplied data via the Internet and stores it on behalf of the user. This data takes the form of a series of time-ordered “spans” representing discrete events in the user's system, collected in logical groups known as “traces” which represent a single, larger operation. Collections of spans may be grouped into larger time-ordered “segments”. An observability platform may be configured to receive a user request to perform a query in association with a dataset on an observability platform. The user request may include user input defining the query, where the query includes one or more filters to identify one or more spans in a trace. The filters may include criteria based on the relationship between two disparate spans in a trace, requiring a join in which the two spans are examined together. The observability platform's servers may use a “join window” to retain a limited number of spans in memory during query processing. A join window which is dynamically sized may be generated based on the number of event identifiers in the dataset. A list of data segments may be generated based on the retrieved dataset. A server may be configured to include a rolling join window module to process the list of data segments to perform (e.g., execute) one or more relational join operations on the dataset based on the one or more filters in the query using the join window. A result set may be generated based on the processed list of data segments and may be provided as a response to the request for display on the observability platform. Data visualizations may be provided on the observability platform based on the result data set of the processed query. Other implementations of one or more of these aspects and other aspects include corresponding systems, apparatus, and computer programs, configured to perform the various action and/or store various data described in association with these aspects. Numerous additional features may be included in these and various other implementations, as discussed throughout this disclosure. The features and advantages described herein are not all-inclusive and many additional features and advantages will be apparent in view of the figures and description. Moreover, it should be understood that the language used in the present disclosure has been principally selected for readability and instructional purposes, and not to limit the scope of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS The disclosure is illustrated by way of example, and not by way of limitation in the figures of the accompanying drawings in which like reference numerals are used to refer to similar elements. FIG. 1 is a high-level block diagram for relational joins illustrating a system including an observability platform in accordance with some implementations. FIG. 2 is a block diagram illustrating a computing device including software modules used in the system of FIG. 1, including the observability platform in accordance with some implementations. FIG. 3 is a block diagram illustrating a rolling join window module of the observability platform in accordance with some implementations. FIG. 4 is a block diagram illustrating a database of the observability platform in accordance with some implementations. FIG. 5A is a block diagra