US-20260127367-A1 - COMPUTER IMPLEMENTED METHODS FOR THE AUTOMATED ANALYSIS OR USE OF DATA, INCLUDING USE OF A LARGE LANGUAGE MODEL

Abstract

There is provided a method of improving the operation of a generative AI large language model (LLM)-based data processing system, by operating the LLM-based system in conjunction with a non-LLM data processing system; and in which (a) the LLM-based system sends a continuation as an input to the non-LLM system, and (b) the non-LLM system (i) uses symbolic representations to perform non-statistical reasoning on the input from the LLM-based system and (ii) generates a reasoned prompt or other context.

Inventors

• William Tunstall-Pedoe
• Robert Heywood
• Seth WARREN
• Paul BENN
• Duncan REYNOLDS
• Ayush Shah
• Luci KRNIC
• Ziyi Zhu

Assignees

• UNLIKELY ARTIFICIAL INTELLIGENCE LIMITED

Dates

Publication Date

20260507

Application Date

20251121

Priority Date

20220222

Claims (20)

1. 1 . A computer implemented method of improving the accuracy or reliability of an AI system including an LLM (large language model) based system, in which the LLM-based system uses a deep learning model capable of processing natural language and the AI system is capable of generating a sequence of reasoning steps; the method comprising: (i) retrieving data from a knowledge graph comprising semantic nodes representing entities and links or passages representing relationships; and (ii) representing the retrieved data in a machine readable language distinct from natural language text and using the retrieved data during LLM inference to produce an output.
2. 2 . The computer implemented method of claim 1 , in which retrieval extracts a subgraph that satisfies a class membership constraint.
3. 3 . The method of claim 1 , in which retrieval enforces a path constraint between semantic nodes.
4. 4 . The method of claim 1 , in which retrieved nodes and links are translated into the machine readable language with declared schemas.
5. 5 . The method of claim 1 , in which the machine readable representation identifies provenance and source identifiers for nodes or links.
6. 6 . The method of claim 1 , in which the AI system injects the machine readable representation into a prompt or context for inference.
7. 7 . The method of claim 1 , in which retrieved items are ranked using trust and uncertainty.
8. 8 . The method of claim 1 , in which the AI system prefers trusted entries when conflicting nodes or links are retrieved.
9. 9 . The method of claim 1 , in which retrieval is parallelized across graph partitions and merged according to a merging policy.
10. 10 . The method of claim 1 , in which the AI system de duplicates nodes or links and aggregates evidence across matching identifiers.
11. 11 . The method of claim 1 , in which the AI system displays a tree or graph citing the nodes and links used to support an output.
12. 12 . The method of claim 1 , in which resource budgets comprising a time budget or a compute budget constrain graph retrieval.
13. 13 . The method of claim 1 , in which the AI system cross validates retrieved items by answering questions represented in the machine readable language.
14. 14 . The method of claim 1 , in which the AI system caches retrieved subgraphs and reuses them across sessions.
15. 15 . The method of claim 1 , in which the machine readable representation conforms to a declared schema and identifies a schema version.
16. 16 . The method of claim 1 , in which the AI system validates the representation against the declared schema before injection.
17. 17 . The method of claim 1 , in which a validation failure returns a structured error identifying a field path and a violated constraint and the AI system repairs the representation and re validates prior to injection.
18. 18 . The method of claim 1 , in which the AI system stores machine readable graph items as training or re training items together with an uncertainty.
19. 19 . The method of claim 1 , in which retrieval includes links as nodes to represent higher order relationships.
20. 20 . The method of claim 1 , in which retrieval filters items use tenets expressed in the machine readable language.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This is a continuation of U.S. application Ser. No. 19/364,721, filed on Oct. 21, 2025, which is a continuation of U.S. application Ser. No. 18/914,717, filed on Oct. 14, 2024, which is a continuation of U.S. application Ser. No. 18/648,788, filed on Apr. 29, 2024, now U.S. Pat. No. 12,164,868, issued Dec. 10, 2024, which is a continuation of U.S. application Ser. No. 18/301,615, filed on Apr. 17, 2023, now U.S. Pat. No. 11,989,507, issued May 21, 2024, which is a continuation of International Application No. PCT/GB2023/050405, filed on Feb. 22, 2023, which claims priority to GB Application No. GB2202347.7, filed on Feb. 22, 2022; GB Application No. GB2219268.6, filed on Dec. 20, 2022; GB Application No. GB2300624.0, filed on Jan. 16, 2023; and GB Application No. GB2302085.2, filed on Feb. 14, 2023, and is a continuation-in-part of U.S. application Ser. No. 18/001,368, filed on Dec. 9, 2022, which is the US national stage of International Application No. PCT/GB2021/052196, filed on Aug. 24, 2021, the entire contents of each of which being fully incorporated herein by reference. BACKGROUND OF THE INVENTION 1. Field of the Invention The field of the invention relates to computer implemented methods for the automated analysis or use of data, including use of a large language model (LLM), and to related computer implemented methods and systems. 2. Technical Background Natural language (NL) is language evolved for humans such as the English language. Although significant advances have been made in computers' ability to process natural language, computers are still not able to deeply understand the meaning of natural language and use that meaning internally. For this reason most computer applications typically use structured data to store information that they need for processing—e.g. a relational database: designing the schema, populating the database and writing code to process the fields in the database. Use of structured data can work well if the application has limited requirements for the type of data required. However, some applications naturally require an extremely broad, heterogeneous collection of data to work well. This means that the schema required would have to be enormous, making building and coding for such an application impractical. We refer to such applications herein as HUB applications (Heterogeneous and Unreasonably Broad). Examples of HUB applications include an application for managing a person's general health data where there are thousands of tests, thousands of medical conditions and thousands of symptoms. Another related application could be a nutrition tracking application where there are many thousands of substances and foods that can be ingested, each with different metabolic effects on the body. Another example is an application to match the resume of potential candidates with a job specification: in principle such an application would need structured data to represent every skill that might be of value to any role, every type of experience, every type of previous job. Accounting is another application where vast heterogeneous data would be valuable: the perfect accounting application would represent every type of contract, every type of service. In practice some of these applications, where they exist, work with a limited schema that doesn't cover the full range of their ideal properties. Health applications for example, typically work like this ignoring many types of data that they do not cover and instead end up being narrow—limiting the application to only certain verticals within health. Applications may also use natural language or augment a limited schema with natural language—such as with current resume matching applications which might represent a few key skills in a structured form but rely largely on keyword searching or statistical natural language processing (NLP) techniques on written resumes otherwise. In the case of accounting, transactions are represented with limited structured data—debits and credits on virtual ledgers with natural language names. The meaning of the natural language names and thus what these transactions represent is generally opaque to the application. Virtual ledgers often group different types of transaction together but fail to represent semantic differences which may be important. There is no exact threshold for when an application becomes a HUB application but the difficulty of building an application with a hand created schema grows more than linearly with the number of tables as managing these tables as well as the code that maintains them becomes increasingly difficult to do. These issues could be addressed if there existed a language or way of representing data that computers could fully process and understand but that also had an extremely broad scope. In conventional Artificial Intelligence (AI), statistical Machine Learning (ML)—particularly Deep Learning (DL)—has been widely used. This has pr