US-12626152-B2 - Artificial intelligence-based auditors of artificial intelligence

US12626152B2US 12626152 B2US12626152 B2US 12626152B2US-12626152-B2

Abstract

Systems and methods for an artificial intelligence-based auditor of artificial intelligence may be provided. The artificial-intelligence-based auditor may operate on one or more hardware processors. The artificial intelligence-based auditor may continually scan one or more operating artificial intelligence systems for productivity and operability. The scan may identify a data source powering the one or more operating artificial intelligence systems. The scan may measure a delta between an output from each of the one or more operating artificial intelligence systems. The scan may identify trends in quality of the one or more operating artificial intelligence systems. The scan may analyze instances where the one or more operating artificial intelligence systems outputted data outside of a parameter range. Based on the scan, the auditor may label each of the one or more operating artificial intelligence systems as positive, neutral or negative.

Inventors

Manu Kurian
Ana Maxim
Vinesh Patel
Michael Young

Assignees

BANK OF AMERICA CORPORATION

Dates

Publication Date: 20260512
Application Date: 20230720

Claims (2)

1 . A network of artificial intelligence-based auditors of one or more external operating artificial intelligence systems, the network of artificial-intelligence-based auditors operating on a plurality of hardware processors within an entity distributed computing network, the network of artificial intelligence-based auditors comprising: a level one artificial intelligence-based auditor operating a first artificial intelligence auditing technique that identifies a data source powering one or more external operating artificial intelligence systems, said one or more external operating artificial intelligence systems external to the entity distributed computing network, the level one artificial intelligence-based auditor operating on a first hardware processor included in a workstation included among the plurality of hardware processors within the entity distributed computing network, the level one auditor configured to: continually scan the one or more external operating artificial intelligence systems to determine whether the data source powering the one or more external operating artificial intelligence systems is an open-source data source comprising publicly available data or a closed-source data source comprising data only available within a private network, the data source comprising data used by the one or more external operating artificial intelligence systems for generating outputs; and output a first operability score for the one or more external operating artificial intelligence systems based on the amount of the data source comprising publicly available data, wherein a delta from a predetermined baseline corresponds to a negative score; a level two artificial intelligence-based auditor operating a second artificial intelligence auditing technique that identifies a trend, wherein a trend reflects a stability of a value within a predetermined time period, for each of: an artificial intelligence security clearance level corresponding to a quantity of personal or confidential data included in outputs of the one or more external operating artificial intelligence systems, an artificial intelligence skill level comprising a metric corresponding to an accuracy of a correlation between outputs of the one or more external operating artificial intelligence systems and a query transmitted to the one or more external operating artificial intelligence systems, and an artificial intelligence intent level comprising a metric corresponding to a correlation between an output of the one or more external operating artificial intelligence systems and an identified intent, the level two artificial intelligence-based auditor operating on a second hardware processor included in a laptop included among the plurality of hardware processors within the entity distributed computing network, the level two auditor configured to: continually scan the one or more external operating artificial intelligence systems to determine a trend of each of the security clearance level, the artificial intelligence skill level and the artificial intelligence intent level of the one or more external operating artificial intelligence systems; and output a second operability score for the one or more operating artificial intelligence system based on the determined trends of the security clearance level, skill level, and intent level, wherein a delta of any of the determined trends from a predetermined baseline corresponds to a negative score; a level three artificial intelligence-based auditor operating a third artificial intelligence auditing technique that analyzes instances where the one or more external operating artificial intelligence system outputs data outside of a structural parameter range that determines a plot for output, the level three artificial intelligence-based auditor operating on a third hardware processor included in a tablet included among the plurality of hardware processors within the entity distributed computing network, the level three auditor configured to: continually scan the one or more operating artificial intelligence systems to determine instances where the one or more external operating artificial intelligence systems output data outside of the structural parameter range; and output a third operability score for the one or more external operating artificial intelligence systems based on an amount of data outside the structural parameter range, wherein a delta of the amount of data from a predetermined baseline corresponds to a negative score; a level four artificial intelligence-based auditor operating a fourth artificial intelligence auditing technique, the level four artificial intelligence-based auditor operating on a fourth hardware processor included in a mobile device included among the plurality of hardware processors within the entity distributed computing network, the level four auditor configured to: continually scan the one or more external operating artificial intelligence systems to measure a delta between output from a different operating artificial intelligence system and output of the one or more external operating artificial intelligence systems, using a same database as input, and identifying whether the one or more external operating artificial intelligence systems have manipulative bias based on the delta between outputs being greater than a predetermined baseline; and output a fourth operability score for the one or more external operating artificial intelligence systems based on the delta between outputs, wherein a delta reflecting manipulative bias corresponds to a negative score; an artificial intelligence-based syndicator configured to: group the first, second, third, and fourth operability scores, into an operability score grouping; weight the first, second, third, and fourth operability scores to a common scale; identify an outlier within the operability score grouping, the outlier being one of the first, the second, the third, or the fourth weighted operability scores; remove the outlier from the grouping; compute and output a syndicated operability score, the syndicated operability score based on a combination of the remaining operability scores included within the grouping; an artificial intelligence-based labeler configured to label the one or more external operating artificial intelligence systems as negative based on the syndicated operability score being negative; and the entity distributed computing network is configured to modify at least one network access control for the one or more external operating artificial intelligence systems based on the label assigned by the artificial intelligence based labeler, the modified network access control blocking transmission between the entity distributed computing network and the one or more external operating artificial intelligence systems when the assigned label is negative; wherein: each of the level one, level two, level three, and level four auditors operate independently from each other.
2 . The artificial intelligence-based auditor of claim 1 wherein the trends further relate to a sophistication degree, an empathy degree, an emotional capacity degree a creativity degree and/or a thought capacity degree.

Description

FIELD OF TECHNOLOGY Aspects of the disclosure relate to artificial intelligence. BACKGROUND OF THE DISCLOSURE Recently, artificial intelligence has been incorporated into a variety of computer systems. These computer systems use artificial intelligence to continually improve the processing of the computer systems. Traditional artificial intelligence systems were supervised. Therefore, operators monitored the artificial intelligence systems to manipulate the output. However, as artificial intelligence systems become increasingly used in a variety of commercial industries, the artificial intelligence systems are progressively becoming stand-alone computer systems with little or no supervision. The lack of supervision may cause a decrease in AI performance, productivity and/or efficiency. Therefore, it may be desirable to harness the capabilities of artificial intelligence to audit or analyze the activities executed by artificial intelligence in order to monitor artificial intelligence performance, productivity and/or efficiency. SUMMARY OF THE DISCLOSURE Apparatus, methods and systems for auditor artificial intelligence systems are provided. Artificial intelligence systems may be computer-operated systems that perform tasks commonly associated with humans. The auditor systems may audit artificial intelligence systems to monitor the performance of artificial intelligence systems. For the purposes of this application, artificial intelligence systems that monitor other artificial intelligence systems may be referred to as auditor artificial intelligence systems. Also, for the purposes of this application, artificial intelligence systems that may be monitored may be referred to as operating artificial intelligence systems. Operating artificial intelligence systems may utilize a variety of sources to power and/or provide data to the artificial intelligence systems. The operating artificial intelligence systems may learn from historical data, provided by the sources, to respond to future events and/or requests. The sources may be databases or dictionaries that include data. The sources may be open-source sources. Open-source sources may be updatable and updated by a public group. An example of an open-source source may be the internet. The sources may be closed-source sources. Closed-source sources may be updatable and updated by a private group. An example of closed-source source may be a private entity network that is only accessible to a predetermined group. Because operating artificial intelligence may make decisions based on the sources being used to power the operating artificial intelligence, at times, operating artificial intelligence may be compared to a live object. Similar to a live object, operating artificial intelligence systems may be erratic and therefore, provide better quality responses at certain times and provide poorer quality responses at other times. Therefore, an auditor artificial intelligence system may identify the source in order to classify an outcome provided by the operating artificial intelligence system. Furthermore, trends in operating artificial intelligence quality may be identified to classify the outcomes provided by operating artificial intelligence. At times, an operating artificial intelligence system may manipulate an outcome based on one or more biases. Therefore, an auditor artificial intelligence system may audit outputs from multiple operating artificial intelligence systems within the same environment. The deltas between the outputs may identify whether an operating artificial intelligence system produces manipulative outputs that are based on one or more biases. Additionally, auditor artificial intelligence systems may study and analyze instances where operating artificial intelligence systems output data outside of a parameter range. Many times, an operating artificial intelligence system may be provided with a parameter range that determines a plot for output. If the operating artificial intelligence system produced an output outside of a parameter range, the auditor artificial intelligence may inspect the cause of such an output. Auditor artificial intelligence systems may work in a team. As such, multiple auditor artificial intelligence systems may operate in tandem. Each of the artificial intelligence systems may, or may not, be aware of the other auditor artificial intelligence systems. Each of the auditor artificial intelligence systems may use the same or different auditing techniques. The combination of the results of the auditor artificial intelligence systems may be used to label an operating artificial intelligence system with a positive, negative or neutral label. Various scales may be used to label an operating artificial intelligence system. The scales may include a number scale, in which a completely negative operating artificial intelligence system may be identified as −1, a completely positive operating artificial intelligence system may be identifie