US-12625894-B1 - Multi-tenancy retrieval-access generation ingestion versioning

Abstract

A system can store a group of checkpoints that comprises respective pairs comprising respective identifications of at least some respective data stored in a storage system and respective second generation identifiers that correspond to the at least some respective data. The system can, based on ingesting data from the storage system and to send the data to be ingested by a first retrieval-augmented generation system, identify a first checkpoint among the group of checkpoints, query a search system to identify a first portion of the data having respective third generation identifiers that are greater than the respective second generation identifiers in the first checkpoint, wherein the search system stores respective metadata of the at least some respective data from the storage system, and ingest the first portion of the data into the first retrieval-augmented generation system.

Inventors

• Gaurav Chawla
• Adam E. Brenner

Assignees

• DELL PRODUCTS L.P.

Dates

Publication Date

20260512

Application Date

20241113

Claims (20)

1. 1 . A system, comprising: at least one processor; and at least one memory that stores executable instructions that, when executed by the at least one processor, facilitate performance of operations, comprising: storing a group of checkpoints that comprises respective pairs, the respective pairs comprising respective identifications of at least some respective data stored in a storage system and respective second generation identifiers that correspond to the at least some respective data, wherein a first checkpoint of the group of checkpoints corresponds to a first retrieval-augmented generation system of a group of retrieval-augmented generation systems for which a retrieval-augmented generation process is configured to ingest data, and wherein the retrieval-augmented generation process is configured to ingest data via a communications protocol that omits tracking of previously-ingested data; and based on executing the retrieval-augmented generation process comprising performance of an iteration of ingesting data from the storage system and to send the data to be ingested by the first retrieval-augmented generation system, identifying the first checkpoint among the group of checkpoints based on the first checkpoint comprising an indication of the first retrieval-augmented generation system, querying a search system to identify a first portion of the data having respective third generation identifiers that are greater than the respective second generation identifiers in the first checkpoint, wherein the search system stores respective metadata of the at least some respective data from the storage system, and wherein the respective metadata comprises respective first generation identifiers that indicate respective updates to the respective data, ingesting the first portion of the data into the first retrieval-augmented generation system while refraining from ingesting a second portion of the data having corresponding fourth generation identifiers that are less than or equal to the corresponding second generation identifiers in the first checkpoint, and servicing queries to the first retrieval-augmented generation system based on the ingesting of the first portion of the data.
2. 2 . The system of claim 1 , wherein the first retrieval-augmented generation system comprises a first version of a group of versions, wherein the first checkpoint comprises respective parts that correspond to respective versions of the group of versions, and wherein the identifying of the first checkpoint among the group of checkpoints comprises: identifying a first part of the first checkpoint from the respective parts based on the first checkpoint associating an identification of the first version with the first part.
3. 3 . The system of claim 2 , wherein a second retrieval-augmented generation system of the group of retrieval-augmented generation systems comprises a second version of the group of versions, wherein the iteration of the ingesting of the data from the storage system is a first iteration of the ingesting of the data from the storage system, wherein the indication of the first retrieval-augmented generation system is a first indication of the first retrieval-augmented generation system, and wherein the operations further comprise: based on executing the retrieval-augmented generation process comprising performance of a second iteration of the ingesting of the data from the storage system and to the second retrieval-augmented generation system, identifying a second checkpoint among the group of checkpoints based on the second checkpoint comprising a second indication of the second retrieval-augmented generation system, and performing the second iteration of the ingesting of the data based on the second checkpoint.
4. 4 . The system of claim 1 , wherein the first retrieval-augmented generation system comprises a first version of a group of versions, wherein the indication of the first retrieval-augmented generation system is a first indication of the first retrieval-augmented generation system, and wherein the identifying of the first checkpoint among the group of checkpoints is based on a second identification of the first version.
5. 5 . The system of claim 1 , wherein the indication of the first retrieval-augmented generation system is determined based on a parameter to a call to invoke the performance of the iteration of the ingesting of the data from the storage system.
6. 6 . The system of claim 5 , wherein the indication of the first retrieval-augmented generation system is a first indication of the first retrieval-augmented generation system, wherein the first retrieval-augmented generation system comprises a first version of a group of versions, wherein the parameter is a first parameter, and wherein a second parameter of the call comprises a second indication of the first version of the group of versions of the first retrieval-augmented generation system.
7. 7 . The system of claim 1 , wherein the identifying of the first checkpoint among the group of checkpoints based on the first checkpoint comprising the indication of the first retrieval-augmented generation system comprises: determining whether the first checkpoint exists, to produce a result.
8. 8 . The system of claim 7 , wherein the result is a first result, wherein the first result indicates that the first checkpoint exists, and wherein the operations further comprise: determining whether a path on the storage system that is identified as a parameter to a call to invoke the performance of the iteration of the ingesting of the data from the storage system is identified in the first checkpoint, to produce a second result.
9. 9 . The system of claim 8 , wherein the querying of the search system to identify the first portion of the data that has corresponding third generation identifiers that are greater than the corresponding second generation identifiers in the first checkpoint is based on the second result indicating that the path is identified in the first checkpoint.
10. 10 . A method, comprising: storing, by a system comprising at least one processor, a group of checkpoints that comprise pairs, respective pairs of the pairs comprising respective identifications of respective data stored in a storage system and respective second generation identifiers that correspond to the respective data; and based on executing a retrieval-augmented generation process comprising performance of an iteration of ingesting the data from the storage system and to send the data to be ingested by a first retrieval-augmented generation system of a group of retrieval-augmented generation systems, identifying, by the system, a first checkpoint among the group of checkpoints based on the first checkpoint comprising an indication of the first retrieval-augmented generation system, querying, by the system, a search system to identify a first portion of the data having respective third generation identifiers that are greater than the respective second generation identifiers in the first checkpoint, wherein the search system stores respective metadata of the at least some respective data from the storage system, and wherein the respective metadata comprises respective first generation identifiers that indicate respective updates to the respective data, ingesting, by the system, the first portion of the data into the first retrieval-augmented generation system while refraining from ingesting a second portion of the data having corresponding fourth generation identifiers that are less than or equal to the corresponding second generation identifiers in the first checkpoint, and servicing, by the system, queries to the first retrieval-augmented generation system based on the ingesting of the first portion of the data.
11. 11 . The method of claim 10 , further comprising: based on the retrieval-augmented generation process performing a second iteration of the ingesting of the data from the storage system, determining, by the system, whether a second checkpoint exists, to produce a result, wherein the second checkpoint corresponds to the second iteration, and wherein the second checkpoint comprises the first checkpoint or another checkpoint other than the first checkpoint; and responsive to the result indicating that the second checkpoint does not exist, passing second data to the retrieval-augmented generation process independently of whether the second data has previously been passed to the retrieval-augmented generation process.
12. 12 . The method of claim 10 , further comprising: based on executing the retrieval-augmented generation process comprising performance a second iteration of the ingesting of the data from the storage system, determining, by the system, whether a second checkpoint exists, to produce a result, wherein the second checkpoint corresponds to the second iteration, and wherein the second checkpoint comprises the first checkpoint or another checkpoint other than the first checkpoint; and responsive to the result indicating that the second checkpoint exists and that a path for the second iteration is not identified by the first checkpoint, passing second data to the retrieval-augmented generation process independently of whether the second data has previously been passed to the retrieval-augmented generation process.
13. 13 . The method of claim 10 , wherein the first portion of the data comprises at least one file.
14. 14 . The method of claim 10 , wherein the first portion of the data comprises at least one directory.
15. 15 . A non-transitory computer-readable medium comprising instructions that, in response to execution, cause a system comprising at least one processor to perform operations, comprising: based on executing a retrieval-augmented generation framework comprising performance of an iteration of ingesting data from a storage system and to send the data to be ingested by a first retrieval-augmented generation system, identifying a first state file among a group of state files based on the first state file comprising an indication of the first retrieval-augmented generation system; querying a search system to identify a first portion of the data having respective third generation identifiers that are greater than the respective second generation identifiers in the first state file, wherein the search system stores respective metadata of the respective data from the storage system, wherein the respective metadata comprises respective first generation identifiers that indicate respective updates to the respective data; ingesting the first portion of the data into the first retrieval-augmented generation system while refraining from ingesting a second portion of the data having corresponding fourth generation identifiers that are less than or equal to the corresponding second generation identifiers in the first state file; and servicing queries to the first retrieval-augmented generation system based on the ingesting of the first portion of the data.
16. 16 . The non-transitory computer-readable medium of claim 15 , wherein the ingesting of the first portion of the data into the first retrieval-augmented generation system comprises: asynchronously passing the first portion of the data to the retrieval-augmented generation framework.
17. 17 . The non-transitory computer-readable medium of claim 15 , wherein the performance of the iteration of the ingesting of the data from the storage system comprises: determining that the first state file exists based on a name of the first retrieval-augmented generation system.
18. 18 . The non-transitory computer-readable medium of claim 17 , wherein the operations further comprise: after the determining that the first state file exists, determining that an identification of a version of the first retrieval-augmented generation system exists in the first state file.
19. 19 . The non-transitory computer-readable medium of claim 18 , wherein the operations further comprise: determining that a path on the storage system that corresponds to the performance of the iteration of the ingesting of the data is identified in the first state file for the identification of the version of the first retrieval-augmented generation system.
20. 20 . The non-transitory computer-readable medium of claim 15 , wherein a group of retrieval-augmented generation systems comprises the first retrieval-augmented generation system, and wherein the retrieval-augmented generation framework is configured to ingest the data for respective retrieval-augmented generation systems of the group of retrieval-augmented generation systems.

Description

RELATED APPLICATIONS The subject patent application is related by subject matter to, U.S. patent application Ser. No. 18/946,737, filed Nov. 13, 2024 and entitled “RETRIEVAL-ACCESS GENERATION INGESTION VERSIONING,” the entirety of which application is hereby incorporated by reference herein. BACKGROUND A retrieval-access generation (RAG) system can generally comprise a large language model (LLM) that operates on a specific information set (e.g., a set of documents) so that the LLM is configured to respond to queries based on that information set. A LLM can generally comprise a form of generative artificial intelligence (AI) that is configured to generative natural-language response outputs to natural-language query inputs. SUMMARY The following presents a simplified summary of the disclosed subject matter in order to provide a basic understanding of some of the various embodiments. This summary is not an extensive overview of the various embodiments. It is intended neither to identify key or critical elements of the various embodiments nor to delineate the scope of the various embodiments. Its sole purpose is to present some concepts of the disclosure in a streamlined form as a prelude to the more detailed description that is presented later. An example system can operate as follows. The system can store a group of checkpoints that comprises respective pairs, the respective pairs comprising respective identifications of at least some respective data stored in a storage system and respective second generation identifiers that correspond to the at least some respective data, wherein a first checkpoint of the group of checkpoints corresponds to a first retrieval-augmented generation system of a group of retrieval-augmented generation systems for which a retrieval-augmented generation process is configured to ingest data, and wherein the retrieval-augmented generation process is configured to ingest data via a communications protocol that omits tracking of previously-ingested data. The system can, based on executing the retrieval-augmented generation process comprising performance of an iteration of ingesting data from the storage system and to send the data to be ingested by the first retrieval-augmented generation system, identify the first checkpoint among the group of checkpoints based on the first checkpoint comprising an indication of the first retrieval-augmented generation system, query a search system to identify a first portion of the data having respective third generation identifiers that are greater than the respective second generation identifiers in the first checkpoint, wherein the search system stores respective metadata of the at least some respective data from the storage system, and wherein the respective metadata comprises respective first generation identifiers that indicate respective updates to the respective data, ingest the first portion of the data into the first retrieval-augmented generation system while refraining from ingesting a second portion of the data having corresponding fourth generation identifiers that are less than or equal to the corresponding second generation identifiers in the first checkpoint, and service queries to the first retrieval-augmented generation system based on the ingesting of the first portion of the data. An example method can comprise storing, by a system comprising at least one processor, a group of checkpoints that comprise pairs, respective pairs of the pairs comprising respective identifications of respective data stored in a storage system and respective second generation identifiers that correspond to the respective data. The method can further comprise, based on executing a retrieval-augmented generation process comprising performance of an iteration of ingesting the data from the storage system and to send the data to be ingested by a first retrieval-augmented generation system of a group of retrieval-augmented generation systems, identifying, by the system, a first checkpoint among the group of checkpoints based on the first checkpoint comprising an indication of the first retrieval-augmented generation system, querying, by the system, a search system to identify a first portion of the data having respective third generation identifiers that are greater than the respective second generation identifiers in the first checkpoint, wherein the search system stores respective metadata of the at least some respective data from the storage system, and wherein the respective metadata comprises respective first generation identifiers that indicate respective updates to the respective data, ingesting, by the system, the first portion of the data into the first retrieval-augmented generation system while refraining from ingesting a second portion of the data having corresponding fourth generation identifiers that are less than or equal to the corresponding second generation identifiers in the first checkpoint, and servicing, by the system, queries to the