US-12620181-B2 - Determining an assignment of virtual objects to positions in a user field of view to render in a mixed reality display

Abstract

Provided are a computer program product, system, and method for determining an assignment of virtual objects to positions in a user field of view to render in a mixed reality display. A reward score for a virtual object is calculated based on an easiness score indicating an easiness to view the virtual object in previous interactions with the virtual object and frequency of interactions with the virtual object. The reward scores and the viewability scores for the positions are inputted into a virtual object placement program to output an assignment of the virtual objects to the positions. The assignment of the virtual objects to the positions are transmitted to the mixed reality display to cause the mixed reality display to render the virtual objects at the positions in the field of view of the user indicated in the assignment.

Inventors

• Shikhar KWATRA
• Smitkumar Narotambhai MARVANIYA
• Jeremy R. Fox

Assignees

• INTERNATIONAL BUSINESS MACHINES CORPORATION

Dates

Publication Date

20260505

Application Date

20220923

Claims (20)

1. 1 . A computer program product for determining virtual objects to render in a mixed reality display, the computer program product comprising a computer readable storage medium having computer readable program code embodied therein that is executable to perform operations, the operations comprising: providing viewability scores for positions in a field of view of a user in which virtual objects are rendered, wherein a viewability score for a position is based on a distance from the user and change in viewing angle to view the position; calculating reward scores for virtual objects, wherein a reward score for a virtual object is calculated based on an easiness score indicating an easiness to view the virtual object in previous interactions with the virtual object and frequency of interactions with the virtual object; inputting the reward scores and the viewability scores for the positions into a virtual object placement program to output an assignment of the virtual objects to the positions; and transmitting the assignment of the virtual objects to the positions to the mixed reality display to cause the mixed reality display to render the virtual objects at the positions in the field of view of the user indicated in the assignment.
2. 2 . The computer program product of claim 1 , wherein the virtual object placement program implements machine learning, wherein the operations further comprise: receiving feedback indicating interactions with the virtual objects rendered at the positions according to the assignment, wherein the virtual object placement program outputted the assignment with a confidence level indicating a probability that the assignment is optimal; determining whether the interactions satisfy a positive threshold indicating a high level of interactions with the virtual objects in the positions indicated in the assignment or a low threshold indicating a low level of interactions with the virtual objects in the positions indicated in the assignment; training the virtual object placement program to receive as input the reward scores for the virtual objects and the viewability scores for the positions and output the assignment of the virtual objects to the positions with an increased confidence level comprising an increase to the confidence level in response to determining that the interactions satisfy the positive threshold; and training the virtual object placement program to receive as input the reward scores for the virtual objects and the viewability scores for the positions and output the assignment of the virtual objects to the positions with a decreased confidence level comprising a decrease to the confidence level in response to determining that the interactions do not satisfy the positive threshold.
3. 3 . The computer program product of claim 1 , wherein the easiness to view a virtual object is a function of a sum of a weight based on a depth of the user from the virtual object and a change in a viewing angle by the user to view the virtual object in the field of view times a size of the virtual object in the field of view for previous interactions with the virtual object.
4. 4 . The computer program product of claim 1 , wherein calculating a reward score for a virtual object further comprises: calculating an interaction score based on user previous interactions with the virtual object, wherein the reward score is a function of the easiness score and the interaction score.
5. 5 . The computer program product of claim 4 , wherein calculating the interaction score for the virtual object further comprises calculating an implicit interaction score based on relationships of the virtual object with other virtual objects, wherein the interaction score is a function of user previous interactions with the virtual object and the implicit interaction score.
6. 6 . The computer program product of claim 5 , wherein the implicit interaction score is calculated from a graph of nodes representing virtual objects and edges between the nodes that captures a relationship of the virtual objects at the nodes on the edges.
7. 7 . The computer program product of claim 1 , wherein the operations further comprise: processing personal information on the user to compare to metadata on a database of virtual objects to determine virtual objects relevant to the user, wherein the reward scores are calculated for the virtual objects relevant to the user, and wherein the virtual objects assigned to the positions comprise the virtual objects relevant to the user.
8. 8 . The computer program product of claim 1 , wherein the operations further comprise: receiving, from the mixed reality display, information on a user interaction with a virtual object rendered in the field of view including an identifier of the virtual object, a distance of the user from the virtual object in the field of view, and a change in viewing angle for the user to view and interact with the virtual object; and storing the information on the user interaction with the virtual object to later use to determine the easiness score indicating an easiness to view the virtual object and frequency of interactions with the virtual object.
9. 9 . A system for determining virtual objects to render in a mixed reality display, comprising: a processor; and a computer readable storage medium having computer readable program code embodied therein that when executed by the processor performs operations, the operations comprising: providing viewability scores for positions in a field of view of a user in which virtual objects are rendered, wherein a viewability score for a position is based on a distance from the user and change in viewing angle to view the position; calculating reward scores for virtual objects, wherein a reward score for a virtual object is calculated based on an easiness score indicating an easiness to view the virtual object in previous interactions with the virtual object and frequency of interactions with the virtual object; inputting the reward scores and the viewability scores for the positions into a virtual object placement program to output an assignment of the virtual objects to the positions; and transmitting the assignment of the virtual objects to the positions to the mixed reality display to cause the mixed reality display to render the virtual objects at the positions in the field of view of the user indicated in the assignment.
10. 10 . The system of claim 9 , wherein the virtual object placement program implements machine learning, wherein the operations further comprise: receiving feedback indicating interactions with the virtual objects rendered at the positions according to the assignment, wherein the virtual object placement program outputted the assignment with a confidence level indicating a probability that the assignment is optimal; determining whether the interactions satisfy a positive threshold indicating a high level of interactions with the virtual objects in the positions indicated in the assignment or a low threshold indicating a low level of interactions with the virtual objects in the positions indicated in the assignment; training the virtual object placement program to receive as input the reward scores for the virtual objects and the viewability scores for the positions and output the assignment of the virtual objects to the positions with an increased confidence level comprising an increase to the confidence level in response to determining that the interactions satisfy the positive threshold; and training the virtual object placement program to receive as input the reward scores for the virtual objects and the viewability scores for the positions and output the assignment of the virtual objects to the positions with a decreased confidence level comprising a decrease to the confidence level in response to determining that the interactions do not satisfy the positive threshold.
11. 11 . The system of claim 9 , wherein calculating a reward score for a virtual object further comprises: calculating an interaction score based on user previous interactions with the virtual object, wherein the reward score is a function of the easiness score and the interaction score.
12. 12 . The system of claim 11 , wherein calculating the interaction score for the virtual object further comprises calculating an implicit interaction score based on relationships of the virtual object with other virtual objects, wherein the interaction score is a function of user previous interactions with the virtual object and the implicit interaction score.
13. 13 . The system of claim 12 , wherein the implicit interaction score is calculated from a graph of nodes representing virtual objects and edges between the nodes that captures a relationship of the virtual objects at the nodes on the edges.
14. 14 . The system of claim 11 , wherein the operations further comprise: processing personal information on the user to compare to metadata on a database of virtual objects to determine virtual objects relevant to the user, wherein the reward scores are calculated for the virtual objects relevant to the user, and wherein the virtual objects assigned to the positions comprise the virtual objects relevant to the user.
15. 15 . The system of claim 11 , wherein the operations further comprise: receiving, from the mixed reality display, information on a user interaction with a virtual object rendered in the field of view including an identifier of the virtual object, a distance of the user from the virtual object in the field of view, and a change in viewing angle for the user to view and interact with the virtual object; and storing the information on the user interaction with the virtual object to later use to determine the easiness score indicating an easiness to view the virtual object and frequency of interactions with the virtual object.
16. 16 . A method for determining virtual objects to render in a mixed reality display, comprising: providing viewability scores for positions in a field of view of a user in which virtual objects are rendered, wherein a viewability score for a position is based on a distance from the user and change in viewing angle to view the position; calculating reward scores for virtual objects, wherein a reward score for a virtual object is calculated based on an easiness score indicating an easiness to view the virtual object in previous interactions with the virtual object and frequency of interactions with the virtual object; inputting the reward scores and the viewability scores for the positions into a virtual object placement program to output an assignment of the virtual objects to the positions; and transmitting the assignment of the virtual objects to the positions to the mixed reality display to cause the mixed reality display to render the virtual objects at the positions in the field of view of the user indicated in the assignment.
17. 17 . The method of claim 16 , wherein the virtual object placement program implements machine learning, further comprising: receiving feedback indicating interactions with the virtual objects rendered at the positions according to the assignment, wherein the virtual object placement program outputted the assignment with a confidence level indicating a probability that the assignment is optimal; determining whether the interactions satisfy a positive threshold indicating a high level of interactions with the virtual objects in the positions indicated in the assignment or a low threshold indicating a low level of interactions with the virtual objects in the positions indicated in the assignment; training the virtual object placement program to receive as input the reward scores for the virtual objects and the viewability scores for the positions and output the assignment of the virtual objects to the positions with an increased confidence level comprising an increase to the confidence level in response to determining that the interactions satisfy the positive threshold; and training the virtual object placement program to receive as input the reward scores for the virtual objects and the viewability scores for the positions and output the assignment of the virtual objects to the positions with a decreased confidence level comprising a decrease to the confidence level in response to determining that the interactions do not satisfy the positive threshold.
18. 18 . The method of claim 16 , wherein calculating a reward score for a virtual object further comprises: calculating an interaction score based on user previous interactions with the virtual object, wherein the reward score is a function of the easiness score and the interaction score.
19. 19 . The method of claim 18 , wherein calculating the interaction score for the virtual object further comprises calculating an implicit interaction score based on relationships of the virtual object with other virtual objects, wherein the interaction score is a function of user previous interactions with the virtual object and the implicit interaction score.
20. 20 . The method of claim 16 , further comprising: receiving, from the mixed reality display, information on a user interaction with a virtual object rendered in the field of view including an identifier of the virtual object, a distance of the user from the virtual object in the field of view, and a change in viewing angle for the user to view and interact with the virtual object; and storing the information on the user interaction with the virtual object to later use to determine the easiness score indicating an easiness to view the virtual object and frequency of interactions with the virtual object.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a computer program product, system, and method for determining an assignment of virtual objects to positions in a user field of view to render in a mixed reality display. 2. Description of the Related Art Mixed reality (MR) smart glasses are wearable computer-capable glasses that generate virtual objects, such as three-dimensional images, text, animations, and videos, to overlay into the wearer's field of vision to enable the wearer to view and interact with the virtual objects. Mixed reality is used to generate virtual objects in a user's field of view to enable the user to view and interact with the virtual objects. Mixed Reality (MR) merges the real world and the virtual world to create an immersive experience. Mixed Reality is a combination of virtual reality and augmented reality. It creates a world in which more than one person can participate in an immersive environment and collaborate on a common task together. Mixed Reality glasses create visualizations where objects from both virtual world and real world interact with each other. There is a need in the art to provide improved techniques for rendering mixed reality virtual objects in a mixed reality display to enhance user experiences. SUMMARY Provided are a computer program product, system, and method for determining an assignment of virtual objects to positions in a user field of view to render in a mixed reality display. Provided are viewability scores for positions in a field of view of a user in which virtual objects are rendered. A viewability score for a position is based on a distance from the user and change in viewing angle to view the position. Reward scores for virtual objects are calculated. A reward score for a virtual object is calculated based on an easiness score indicating an easiness to view the virtual object in previous interactions with the virtual object and frequency of interactions with the virtual object. The reward scores and the viewability scores for the positions are inputted into a virtual object placement program to output an assignment of the virtual objects to the positions. The assignment of the virtual objects to the positions are transmitted to the mixed reality display to cause the mixed reality display to render the virtual objects at the positions in the field of view of the user indicated in the assignment. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 illustrates an embodiment of a computing environment to control a mixed reality display to render virtual objects in a user field of view. FIG. 2 illustrates an embodiment of a virtual object request from a user wearing the mixed reality display. FIG. 3 illustrates an embodiment of object view results having interaction on user interaction with virtual objects rendered in a mixed reality display. FIG. 4 illustrates an embodiment of a training set to train a virtual object placement program to assign virtual objects to positions in a user field of view. FIGS. 5, 6a, 6b, and 6c illustrate examples of positions in a user field of view in which virtual objects may be rendered. FIGS. 7a, and 7b illustrate an embodiment of operations to generate an assignment of virtual objects to positions in a user field of view for a virtual object request. FIG. 8 illustrates an embodiment of machine learning training of a virtual object placement program to generate assignments of virtual objects to positions in a user field of view. FIG. 9 illustrates a computing environment in which the components of FIG. 1 may be implemented. DETAILED DESCRIPTION Prior art systems for assigning virtual objects to positions in a user field of view may favor placement of virtual objects at positions that are easy to view, even if the virtual object content is not necessarily popular and does not receive a high level of interactions. For instance, if a virtual object is placed in a hallway of a mixed reality space that is easily viewable by the user, such placement will inherently have a high number of hits, even though the virtual object content has no or minimal relevance to the user. Similarly, a very relevant virtual object placed at a less viewable or less easy to view position will only attract a few interactions. Traditional object placement techniques will assign the low relevance virtual object a high positive reward/relevance score even though the virtual object content itself is of little relevance to the user, and the highly relevant virtual object will have a low reward/relevance value even though it is highly relevant to the user. Described embodiments provide improved techniques for scoring positions in a mixed reality space, such as segment regions, by considering both ease of view and feedback metrics, such as number of interactions, duration of interactions, etc., to determine a reward score to assign to a virtual object. Further described embodiments may incorporate a relatedness of the content o