Search

US-20260127364-A1 - COMPUTER IMPLEMENTED METHODS FOR THE AUTOMATED ANALYSIS OR USE OF DATA, INCLUDING USE OF A LARGE LANGUAGE MODEL

US20260127364A1US 20260127364 A1US20260127364 A1US 20260127364A1US-20260127364-A1

Abstract

There is provided a method of improving the operation of a generative AI large language model (LLM)-based data processing system, by operating the LLM-based system in conjunction with a non-LLM data processing system; and in which (a) the LLM-based system sends a continuation as an input to the non-LLM system, and (b) the non-LLM system (i) uses symbolic representations to perform non-statistical reasoning on the input from the LLM-based system and (ii) generates a reasoned prompt or other context.

Inventors

  • William Tunstall-Pedoe
  • Robert Heywood
  • Seth WARREN
  • Paul BENN
  • Duncan REYNOLDS
  • Ayush Shah
  • Luci KRNIC
  • Ziyi Zhu

Assignees

  • UNLIKELY ARTIFICIAL INTELLIGENCE LIMITED

Dates

Publication Date
20260507
Application Date
20251107
Priority Date
20220222

Claims (20)

  1. 1 . A computer implemented method of improving the accuracy or reliability of an AI system including a LLM (large language model) based system, in which the LLM-based system uses a deep learning model capable of processing natural language and the AI system is capable of generating a sequence of reasoning steps; and in which the LLM-based system produces a structured, machine-readable output that conforms to a declared schema and that is distinct from natural-language text; and a non-LLM processor receives and is capable of processing that structured output to modify, correct, or control provision of a response or output of the LLM-based system.
  2. 2 . The computer implemented method of claim 1 , in which the structured, machine-readable output comprises JSON that conforms to the declared schema.
  3. 3 . The computer implemented method of claim 1 , in which the structured, machine-readable output includes a schema identifier and a schema version embedded by the LLM-based system.
  4. 4 . The computer implemented method of claim 1 , in which the declared schema is registered with the AI system prior to inference.
  5. 5 . The computer implemented method of claim 1 , in which fields of the structured, machine-readable output are typed and required or optional according to the declared schema.
  6. 6 . The computer implemented method of claim 1 , in which the structured, machine-readable output is distinct from natural-language text and excludes free form code.
  7. 7 . The computer implemented method of claim 1 , in which the non LLM processor deterministically parses the structured, machine readable output and validates it against the declared schema.
  8. 8 . The computer implemented method of claim 1 , in which, responsive to a validation failure, the LLM-based system regenerates or repairs the structured, machine readable output and resubmits it for validation.
  9. 9 . The computer implemented method of claim 1 , in which, upon successful validation, the non LLM processor amends one or more fields in the structured, machine readable output.
  10. 10 . The computer implemented method of claim 1 , in which the non LLM processor withholds provision to a user interface unless validation succeeds and any amendments are applied.
  11. 11 . The computer implemented method of claim 1 , in which the non LLM processor validates the structured, machine readable output against the declared schema and then (i) accepts, or (ii) rejects, or (iii) rejects, repairs and re validates the structured, machine readable output.
  12. 12 . The computer implemented method of claim 1 , in which the non LLM processor applies policy or guardrail checks to the structured, machine readable output and refuses, redacts, or rephrases content on non compliance before enabling or permitting display of the LLM response or output of the LLM-based system.
  13. 13 . The computer implemented method of claim 1 , in which policies are represented using a machine readable language distinct from natural language text and stored as tenets.
  14. 14 . The computer implemented method of claim 1 , in which the non LLM processor reasons over the structured, machine readable output.
  15. 15 . The computer implemented method of claim 1 , in which the structured, machine readable output references a computation unit, and the non LLM processor executes the computation unit with typed parameters defined by the declared schema.
  16. 16 . The computer implemented method of claim 1 , in which results from the non LLM processor are returned as structured data and injected as augmented context for the LLM-based system to generate a revised response or output.
  17. 17 . The computer implemented method of claim 1 , in which the non LLM processor stores validated structured, machine readable outputs in long term memory and reuses them across sessions.
  18. 18 . The computer implemented method of claim 1 , in which resource limits comprising a time budget or a compute budget constrain validation and amendment by the non LLM processor.
  19. 19 . The computer implemented method of claim 1 , in which the non LLM processor retries validation or amendment according to a retry policy and emits a structured failure after a threshold number of retries.
  20. 20 . The computer implemented method of claim 1 , in which parallel validation is applied to sub portions of the structured, machine readable output and combined prior to enabling display.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This is a continuation of U.S. application Ser. No. 19/364,721, filed on Oct. 21, 2025, which is a continuation of U.S. application Ser. No. 18/914,717, filed on Oct. 14, 2024, which is a continuation of U.S. application Ser. No. 18/648,788, filed on Apr. 29, 2024, now U.S. Pat. No. 12,164,868, issued Dec. 10, 2024, which is a continuation of U.S. application Ser. No. 18/301,615, filed on Apr. 17, 2023, now U.S. Pat. No. 11,989,507, issued May 21, 2024, which is a continuation of International Application No. PCT/GB2023/050405, filed on Feb. 22, 2023, which claims priority to GB Application No. GB2202347.7, filed on Feb. 22, 2022; GB Application No. GB2219268.6, filed on Dec. 20, 2022; GB Application No. GB2300624.0, filed on Jan. 16, 2023; and GB Application No. GB2302085.2, filed on Feb. 14, 2023, and is a continuation-in-part of U.S. application Ser. No. 18/001,368, filed on Dec. 9, 2022, which is the US national stage of International Application No. PCT/GB2021/052196, filed on Aug. 24, 2021, the entire contents of each of which being fully incorporated herein by reference. BACKGROUND OF THE INVENTION 1. Field of the Invention The field of the invention relates to computer implemented methods for the automated analysis or use of data, including use of a large language model (LLM), and to related computer implemented methods and systems. 2. Technical Background Natural language (NL) is language evolved for humans such as the English language. Although significant advances have been made in computers' ability to process natural language, computers are still not able to deeply understand the meaning of natural language and use that meaning internally. For this reason most computer applications typically use structured data to store information that they need for processing—e.g. a relational database: designing the schema, populating the database and writing code to process the fields in the database. Use of structured data can work well if the application has limited requirements for the type of data required. However, some applications naturally require an extremely broad, heterogeneous collection of data to work well. This means that the schema required would have to be enormous, making building and coding for such an application impractical. We refer to such applications herein as HUB applications (Heterogeneous and Unreasonably Broad). Examples of HUB applications include an application for managing a person's general health data where there are thousands of tests, thousands of medical conditions and thousands of symptoms. Another related application could be a nutrition tracking application where there are many thousands of substances and foods that can be ingested, each with different metabolic effects on the body. Another example is an application to match the resume of potential candidates with a job specification: in principle such an application would need structured data to represent every skill that might be of value to any role, every type of experience, every type of previous job. Accounting is another application where vast heterogeneous data would be valuable: the perfect accounting application would represent every type of contract, every type of service. In practice some of these applications, where they exist, work with a limited schema that doesn't cover the full range of their ideal properties. Health applications for example, typically work like this ignoring many types of data that they do not cover and instead end up being narrow—limiting the application to only certain verticals within health. Applications may also use natural language or augment a limited schema with natural language—such as with current resume matching applications which might represent a few key skills in a structured form but rely largely on keyword searching or statistical natural language processing (NLP) techniques on written resumes otherwise. In the case of accounting, transactions are represented with limited structured data—debits and credits on virtual ledgers with natural language names. The meaning of the natural language names and thus what these transactions represent is generally opaque to the application. Virtual ledgers often group different types of transaction together but fail to represent semantic differences which may be important. There is no exact threshold for when an application becomes a HUB application but the difficulty of building an application with a hand created schema grows more than linearly with the number of tables as managing these tables as well as the code that maintains them becomes increasingly difficult to do. These issues could be addressed if there existed a language or way of representing data that computers could fully process and understand but that also had an extremely broad scope. In conventional Artificial Intelligence (AI), statistical Machine Learning (ML)—particularly Deep Learning (DL)—has been widely used. This has pr