US-20260123866-A1 - PHYSIOLOGICAL FEEDBACK FOR PREDICTIVE MODELS

Abstract

This document relates to employing biosignals to evaluate predictions made by predictive models. For example, user attention can be inferred from a user attention signal such as gaze. When the user directs attention to a prediction output by a given predictive model, a user reaction signal such as an electroencephalogram or pupillary diameter measurement can be processed to determine whether the user perceives an error. If the user perceives an error, an error indication can be output. Error indications can be used to evaluate the predictive model, replace predictions generated by the predictive model, train the predictive model, etc.

Inventors

• Yu-Te Wang
• Nemanja Djuric
• Ivan J. Tashev
• Raymond Michael WINTERS
• Hannes Gamper
• Dimitra Emmanouilidou

Assignees

• MICROSOFT TECHNOLOGY LICENSING, LLC

Dates

Publication Date

20260507

Application Date

20260101

Claims (20)

1. 1 - 20 . (canceled)
2. 21 . A computer-implemented method comprising: receiving a user attention signal conveying where a particular user directs attention over a period of time; based on the user attention signal, identifying a particular time when the particular user directs attention to a prediction output by a predictive model; receiving a user reaction signal conveying a physiological reaction of the particular user to the prediction; determining whether the physiological reaction indicates that the particular user perceives an error responsive to directing attention to the prediction; and when the physiological reaction indicates that the particular user has perceived an error, replacing the prediction with another prediction output by the predictive model, wherein the predictive model has previously been trained based at least in part on a corpus of training data having physiological reaction signals indicating whether errors are perceived during training of the predictive model.
3. 22 . The computer-implemented method of claim 21 , wherein the predictive model comprises a generative text model.
4. 23 . The computer-implemented method of claim 22 , wherein the prediction comprises at least one word output by the generative text model and the another prediction comprises at least one other word output by the generative text model.
5. 24 . The computer-implemented method of claim 23 , wherein the user attention signal indicates when the user gazes at the at least one word output by the generative text model.
6. 25 . The computer-implemented method of claim 21 , wherein the user attention signal indicates where the user is gazing during the period of time.
7. 26 . The computer-implemented method of claim 25 , wherein the user reaction signal comprises an electroencephalogram signal or a pupil diameter measurement.
8. 27 . The computer-implemented method of claim 21 , wherein the prediction relates to a first meaning of a word and the another prediction relates to another meaning of the word.
9. 28 . The computer-implemented method of claim 27 , wherein the user reaction signal conveys a reaction of the particular user to first search results associated with the first meaning of the word, and the replacing involves replacing the first search results with second search results associated with the second meaning of the word.
10. 29 . The computer-implemented method of claim 21 , wherein the prediction is output by moving a targeting mechanism over a first object and the prediction is replaced by moving the targeting mechanism over a second object.
11. 30 . The computer-implemented method of claim 29 , wherein the targeting mechanism is a cursor.
12. 31 . A system comprising: a processor; and a computer-readable storage medium storing instructions which, when executed by the processor, cause the system to: receive a user attention signal conveying where a particular user directs attention over a period of time; based on the user attention signal, identify a particular time when the particular user directs attention to a prediction output by a predictive model; receive a user reaction signal conveying a physiological reaction of the particular user to the prediction; determine whether the physiological reaction indicates that the particular user perceives an error responsive to directing attention to the prediction; and when the physiological reaction indicates that the particular user has perceived an error, replace the prediction with another prediction output by the predictive model, wherein the predictive model has previously been trained based at least in part on a corpus of training data having physiological reaction signals indicating whether errors are perceived during training of the predictive model.
13. 32 . The system of claim 31 , wherein the predictive model comprises a generative text model, the prediction comprises at least one word output by the generative text model, and the another prediction comprises at least one other word output by the generative text model.
14. 33 . The system of claim 32 , wherein the instructions, when executed by the processor, cause the system to: collect the training data from at least one of text messages or email messages.
15. 34 . The system of claim 33 , wherein the instructions, when executed by the processor, cause the system to: select a first generative text model trained on the text messages to generate the prediction and the another prediction in instances when the particular user is generating a new text message; and select a second generative text model trained on the email messages to generate the prediction and the another prediction in instances when the particular user is generating a new email message.
16. 35 . The system of claim 32 , wherein the instructions, when executed by the processor, cause the system to: generate the prediction and the another prediction based at least on a letter entered by the particular user.
17. 36 . The system of claim 35 , wherein the prediction and the another prediction both begin with the letter entered by the particular user.
18. 37 . The system of claim 31 , wherein the instructions, when executed by the processor, cause the system to: apply a machine-trained classifier to the user reaction signal to determine whether the physiological reaction indicates that the particular user has perceived the error.
19. 38 . The system of claim 37 , further comprising: a wearable device having one or more sensors configured to generate the user reaction signal.
20. 39 . The system of claim 38 , the wearable device having one or more other sensors configured to generate the user attention signal.

Description

BACKGROUND One important use case for computing technologies involves inferring user intent and outputting content that a user is predicted to be interested in based on the inferred intent. For instance, predictive text models can be employed to suggest the next word in a sentence to a user, or search engines can predict suggested queries that a user might want to use to search for documents. However, in some cases, predictive models generate incorrect suggestions, which can cause users to lose trust in the models and, in some cases, stop using them altogether. SUMMARY This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. The description generally relates to techniques for employing physiological feedback to evaluate predictions made by predictive models. One example relates to a method or technique that can include receiving a user attention signal conveying where a user directs attention over a period of time. The method or technique can also include, based on the user attention signal, identifying a particular time when the user directs attention to a prediction output by a predictive model. The method or technique can also include receiving a user reaction signal conveying a physiological reaction of the user to the prediction. The method or technique can also include determining whether the physiological reaction indicates that the user perceives an error responsive to directing attention to the prediction. The method or technique can also include, in an instance when the physiological reaction indicates that the user perceives an error responsive, outputting an error indication. Another example includes a system that can include a processor and a storage medium. The storage medium can store instructions which, when executed by the processor, cause the system to output a prediction and receive an error indication indicating that a user perceives an error in the prediction. The error indication can be based on a user attention signal indicating that the user directs attention to the prediction at a particular time and a user reaction signal indicating that the user perceives an error responsive to directing attention to the prediction. The instructions can also cause the system to, based on the error indication, replace the prediction with another prediction. Another example includes a computer-readable storage medium storing instructions which, when executed by a computing device, cause the computing device to perform acts. The acts can include receiving a user attention signal conveying where a user directs attention over a period of time. The acts can also include, based on the user attention signal, identifying a particular time when the user directs attention to a prediction output by a predictive model. The acts can also include receiving a user reaction signal conveying a physiological reaction of the user to the prediction. The acts can also include determining whether the physiological reaction indicates that the user perceives an error responsive to directing attention to the prediction. The acts can also include, in an instance when the physiological reaction indicates that the user perceives an error responsive, outputting an error indication. The above-listed examples are intended to provide a quick reference to aid the reader and are not intended to define the scope of the concepts described herein. BRIEF DESCRIPTION OF THE DRAWINGS The Detailed Description is described with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of similar reference numbers in different instances in the description and the figures may indicate similar or identical items. FIG. 1 illustrates an example system, consistent with some implementations of the present concepts. FIGS. 2A-2I illustrate an example application scenario over a period of time, consistent with some implementations of the present concepts. FIGS. 3 and 4 illustrate example methods or techniques, consistent with some implementations of the present concepts. FIG. 5 illustrates an example experimental workflow, consistent with some implementations of the present concepts. FIGS. 6, 7, 8, 9A, and 9B illustrate experimental results, consistent with some implementations of the present concepts. DETAILED DESCRIPTION As noted previously, predictive models enjoy widespread usage in various applications. For instance, predictive text models can help users write words, or other predictive models can be employed to help disambiguate user inputs such as mouse or cursor locations. However, predictive models are not perfect and occasionally make incorrect prediction