Search

US-20260127536-A1 - ARTIFICIAL INTELLIGENCE INTEGRATION SCORING SYSTEM AND METHOD

US20260127536A1US 20260127536 A1US20260127536 A1US 20260127536A1US-20260127536-A1

Abstract

Disclosed herein are systems and methods for generating an Al work score for tasks assisted by artificial intelligence. The score is determined by collecting and processing user data using scoring algorithms that consider at least one of the following: Al integration, task complexity, baseline metrics, and time calibration. The system and methods also generate composite scores across users, tasks, groups or industries and classifications across various metrics, enabling correlation reports. These scores can assist in evaluating Al integration, optimizing task assignments and improving Al usage in supervised environments.

Inventors

  • Rodney Patrick Mock
  • Martin Mehl

Assignees

  • Rodney Patrick Mock
  • Martin Mehl

Dates

Publication Date
20260507
Application Date
20241102

Claims (19)

  1. 1 . A method for calculating an AI work score, the method comprising: collecting, by a processor, online data, associated with an AI user; collecting, by the processor, online data, associated with a supervisor; determining, by the processor, using scoring algorithms, an AI work score, wherein the online data is assigned numerical values corresponding to at least one of: an AI integration factor, a work baseline factor, a complexity factor, or a time calibration; and communicating, by the processor, the AI work score, to at least one of: the AI user, the supervisor or stakeholders.
  2. 2 . The method of claim 1 , wherein the online data associated with the AI user comprises at least one of: an Al-assigned task, an AI user disclosure, AI user data, or a final work product.
  3. 3 . The method of claim 1 , wherein the online data associated with the supervisor comprises at least one of: supervisor input, the Al-assigned task, supervisor data, or a final work product.
  4. 4 . The method of claim 1 , further comprising facilitating, by the processor, the exchange of online data between the AI user and the supervisor.
  5. 5 . The method of claim 1 , further comprising enabling the supervisor, via an interface provided by the processor, to customize the AI work score by selecting and adjusting the weights assigned to at least one of: an AI integration factor, a work baseline factor, a complexity factor or a time calibration.
  6. 6 . The method of claim 1 , wherein the online data is further processed to generate at least one of: a composite AI work score, composite scores by tasks, composite scores by users, sub-scores by users, sub-scores by tasks, or classifications by at least one of: groupings, rankings, subject matter, time, industries, KPI standards, or other sub-scoring categorizations.
  7. 7 . The method of claim 1 , further comprising analyzing the AI work score, the composite scores, the sub-scores and classifications to produce one or more correlation reports.
  8. 8 . The method of claim 1 , wherein communicating the AI work score, the composite scores, the sub-scores, classifications or correlation reports, comprises dynamically generated data visualizations, including at least one of: graphs, charts, illustrations, or customizable dashboards tailored to user preferences and task requirements.
  9. 9 . The method of claim 1 , wherein communicating further comprises generating notifications, alerts, or messages when at least one of: the AI work score, the composite scores, or the sub-scores exceed or fall below certain predefined thresholds.
  10. 10 . A system for calculating an AI work score, the system comprising: a processor configured to: collect online data, associated with an AI user; collect online data, associated with a supervisor; determine, using scoring algorithms, an AI work score, wherein the online data is assigned numerical values corresponding to at least one of: an AI integration factor, a work baseline factor, a complexity factor, or a time calibration; and communicate, the AI work score to at least one of: the AI user, the supervisor or stakeholders.
  11. 11 . The system of claim 10 , wherein the online data associated with the AI user comprises at least one of: an AI-assigned task, an AI user disclosure, AI user data, or a final work product.
  12. 12 . The system of claim 10 , wherein the online data associated with the supervisor comprises at least one of: supervisor input, an Al-assigned task, supervisor data, or a final work product.
  13. 13 . The system of claim 10 , wherein the processor is further configured to facilitate the exchange of online data between the AI user and the supervisor.
  14. 14 . The system of claim 10 , wherein the processor is configured to provide an interface enabling the supervisor to customize the AI work score by selecting and adjusting the weights assigned to at least one of: an AI integration factor, a work baseline factor, a complexity factor or a time calibration.
  15. 15 . The system of claim 10 , wherein the processor is further configured to generate at least one of: a composite AI work score, composite scores by tasks, composite scores by users, sub-scores by users, sub-scores by tasks, or classifications by at least one of: groupings, rankings, subject matter, time, industries, KPI standards, or other sub-scoring categorizations based on the online data.
  16. 16 . The system of claim 10 , wherein the processor is further configured to analyze the AI work score, the composite scores, the sub-scores and classifications to generate one or more correlation reports.
  17. 17 . The system of claim 10 , wherein the processor is further configured to generate dynamic data visualizations, such as at least one of: graphs, charts, illustrations, or customizable dashboards tailored to user preferences and task requirements, for the AI work score, composite scores, sub-scores, classifications or correlation reports.
  18. 18 . The system of claim 10 , wherein the processor is further configured to generate notifications, alerts, or messages when at least one of: the AI work score, the composite scores, or the sub-scores exceed or fall below predefined thresholds.
  19. 19 . A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, perform operations comprising: collecting, online data, associated with an AI user; collecting, online data, associated with a supervisor; determining, using scoring algorithms, an AI work score, wherein the online data is assigned numerical values corresponding to at least one of: an AI integration factor, a work baseline factor, a complexity factor, or a time calibration; and communicating the AI work score to at least one of: the AI user, the supervisor or stakeholders.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority from U.S. provisional application number 63/596,256 filed on Nov. 4, 2023, the entirety of which is hereby fully incorporated by reference herein. FIELD This disclosure relates to methods and systems for evaluating artificial intelligence integration into the performance of tasks. Specifically, the present disclosure relates to collecting applicable data, computing and communicating certain performance scores involving the integration of artificial intelligence into assigned tasks. BACKGROUND The science and engineering of making machines intelligent has resulted in several definitions of “artificial intelligence” and various subfields such as deep learning and machine learning. At its core, artificial intelligence involves combining computer science with extremely large data sets to solve problems thereby simulating human intelligence. Artificial intelligence has been classified into four types, reactive machines, limited memory, theory of mind and self-awareness. For this disclosure, and its discussion of artificial intelligence, any references herein to “AI” or “artificial intelligence” refers to all types of artificial intelligence. AI is being incorporated into everything from automation to machine learning, machine vision, natural language processing, self-driving cars, text image, audio, and computer code generation. AI is also making its way into aerospace, aviation, agriculture, automotive and transportation, banking, business construction and architecture, criminal justice, customer service, education, energy and utilities, environmental, government and defense, monitoring healthcare, human resources, IT processes, law, manufacturing, media, music, non-profits, pharmaceutical and drug discovery, real estate, retail and e-commerce, social security, media and advertising, software development and supply chain management. Regardless of the discipline or industry involved, however, when AI is incorporated into such, some sort of “supervisor” and “supervisee” relationship will generally be present. For example, in the employment context, the employer is the supervisor, and the employee is the supervisee. In academia, it is the educator that fills the role of supervisor, and the student of the supervisee. In these relationships, the quality of AI integration often affects the supervisor's evaluation of the supervisee's performance, but traditional metrics overlook this critical interaction. Thus far, the primary focus on assessing the performance of AI has been largely centered on the input and output of the various AI models and determining confidence scores. AI models are evaluated based on certain metrics such as, among other things, precision, recall, AUC/ROC Curve and F-Score. These metrics focus on the AI's output quality in isolation rather than its impact when integrated into supervised tasks where human-AI collaboration is critical. Metrics such as precision and recall may indicate the accuracy of AI outputs, but they do not capture the unique aspect of the extent to which the AI's recommendations are successfully integrated into final work product. In other words, when a supervisor assigns an AI assisted task to a supervisee what exactly is the AI user doing with the AI output to “integrate” such into the final work product and how is the assessment of that measured. This disclosure technologically addresses such by, among other things, collecting and computing certain online data associated with the supervisor and the AI user to determine and communicate, among other things, an AI work score. Unlike traditional metrics, the AI work score measures the effectiveness of AI integration by considering, among other things, user interaction, context-specific adjustments and communicated feedback in a supervised environment. This AI work score also is designed to be versatile, allowing for certain adjustments by the supervisor to the scoring system to account for the type of AI assisted task and the nature of the supervised environment. While some existing approaches attempt to include user feedback in AI evaluation, they fail to account for the variability in task complexity, supervisor input, individual user proficiency and how the Al's suggestions ultimately influence final work product quality. There is a significant business and educational need for an AI work score method and system to properly assess AI assisted tasks when AI users are integrating AI into their final work product. The lack of an AI work score in education and business results in inefficient recourse allocation, suboptimal training outcomes and challenges in proper performance evaluations. To address these gaps, the present invention introduces a unique method and system for determining an AI work score that comprehensively assesses AI assisted tasks. For all relevant purposes herein “AI work score” and “AI work scores” shall be synonymous. SUMMARY Set for