US-12623143-B2 - AI responsive layout for cross-platform environments

US12623143B2US 12623143 B2US12623143 B2US 12623143B2US-12623143-B2

Abstract

A method including executing a video game to generate a video frame for presentation on a device of a first platform for a game play by a user. The method including determining a target device of a second platform. The method including mapping the video frame to the target device. The method including determining an area of focus in a scene in the video frame based on a game context. The method including classifying an asset in the area of focus using a computer vision model implementing artificial intelligence. The method including determining that the asset is important in the game play using the computer vision model. The method including determining that the asset in the video frame that is mapped does not meet a threshold of visibility. The method including modifying the video frame that is mapped so that the asset meets the threshold of visibility.

Inventors

Elizabeth Osborne
Angela Sun Wu
Jin Zhang
Xi Zhou
Hsin-yi Chien
Olga Rudi

Assignees

SONY INTERACTIVE ENTERTAINMENT INC.

Dates

Publication Date: 20260512
Application Date: 20231004

Claims (17)

1 . A method, comprising: executing a video game to generate a video frame for presentation on a first device of a first platform, wherein the video frame is generated for a game play of the video game by a user; determining a target device, wherein the target device is of a second platform; mapping the video frame from the first device to the target device, wherein the video frame that is mapped is presented on the target device; determining an area of focus in a scene of the game play presented in the video frame based on a game context of the scene in the game play; classifying an asset in the area of focus using a computer vision model implementing artificial intelligence; determining that the asset is important in the game play of the video game using the computer vision model; dumb scaling the video frame to a smaller size for presentation on the target device; determining that the asset in the dumb scaled video frame that is mapped to the target device does not meet a threshold of visibility; and modifying the dumb scaled video frame that is mapped so that the asset meets the threshold of visibility.
2 . The method of claim 1 , further comprising: sending the video frame that is mapped and modified to the target device.
3 . The method of claim 1 , wherein the modifying the video frame that is mapped includes: transforming the asset through rescaling; and overlaying the asset that is rescaled within the video frame that is mapped.
4 . The method of claim 1 , wherein the modifying the video frame that is mapped includes: changing a field of view (FOV) of the scene to the area of focus; and limiting the video frame that is modified to the FOV that is changed.
5 . The method of claim 1 , wherein the determining the area of focus includes: determining a focus of interaction by the user in the scene of the game play.
6 . The method of claim 1 , further comprising: training the computer vision model using a list of assets for the video game.
7 . A non-transitory computer-readable medium storing a computer program for execution by a processor to perform a method, the non-transitory computer-readable medium comprising: program instructions for executing a video game to generate a video frame for presentation on a first device of a first platform, wherein the video frame is generated for a game play of the video game by a user; program instructions for determining a target device, wherein the target device is of a second platform; program instructions for mapping the video frame from the first device to the target device, wherein the video frame that is mapped is presented on the target device; program instructions for determining an area of focus in a scene of the game play presented in the video frame based on a game context of the scene in the game play; program instructions for classifying an asset in the area of focus using a computer vision model implementing artificial intelligence; program instructions for determining that the asset is important in the game play of the video game using the computer vision model; program instructions for dumb scaling the video frame for presentation on the target device; program instructions for determining that the asset in the dumb scaled video frame that is mapped to the target device does not meet a threshold of visibility; and program instructions for modifying the dumb scaled video frame that is mapped so that the asset meets the threshold of visibility.
8 . The non-transitory computer-readable medium of claim 7 , further comprising: program instructions for sending the video frame that is mapped and modified to the target device.
9 . The non-transitory computer-readable medium of claim 7 , wherein the program instructions for modifying the video frame that is mapped includes: program instructions for transforming the asset through rescaling; and program instructions for overlaying the asset that is rescaled within the video frame that is mapped.
10 . The non-transitory computer-readable medium of claim 7 , wherein the program instructions for modifying the video frame that is mapped includes: program instructions for changing a field of view (FOV) of the scene to the area of focus; and program instructions for limiting the video frame that is modified to the FOV that is changed.
11 . The non-transitory computer-readable medium of claim 7 , wherein the program instructions for determining the area of focus includes: program instructions for determining a focus of interaction by the user in the scene of the game play.
12 . The non-transitory computer-readable medium of claim 7 , further comprising: program instructions for training the computer vision model using a list of assets for the video game.
13 . A computer system comprising: a processor; memory coupled to the processor and having stored therein instructions that, if executed by the computer system, cause the computer system to execute a method for implementing a graphics pipeline, comprising: executing a video game to generate a video frame for presentation on a first device of a first platform, wherein the video frame is generated for a game play of the video game by a user; determining a target device, wherein the target device is of a second platform; mapping the video frame from the first device to the target device, wherein the video frame that is mapped is presented on the target device; determining an area of focus in a scene of the game play presented in the video frame based on a game context of the scene in the game play; dumb scaling the video frame for presentation on the target device; determining an asset in the area of focus is important in the game play of the video game using a computer vision model implementing artificial intelligence; classifying the asset using the computer vision model; determining that the asset in the dumb scaled video frame that is mapped to the target device does not meet a threshold of visibility; and modifying the dumb scaled video frame that is mapped so that the asset meets the threshold of visibility.
14 . The computer system of claim 13 , the method further comprising: sending the video frame that is mapped and modified to the target device.
15 . The computer system of claim 13 , wherein in the method the modifying the video frame that is mapped includes: changing a field of view (FOV) of the scene to the area of focus; and limiting the video frame that is modified to the FOV that is changed.
16 . The computer system of claim 13 , wherein in the method the determining the area of focus includes: determining a focus of interaction by the user in the scene of the game play.
17 . The computer system of claim 13 , the method further comprising: training the computer vision model using a list of assets for the video game.

Description

TECHNICAL FIELD The present disclosure is related to modifying content of a video game when delivering video frames for display on a cross-platform device using artificial intelligence techniques, such as computer vision. BACKGROUND OF THE DISCLOSURE Video games and/or gaming applications and their related industries (e.g., video gaming) are extremely popular and represent a large percentage of the worldwide entertainment market. Video games are played anywhere and at any time using various types of platforms, including gaming consoles, desktop computers, laptop computers, mobile phones, etc. A user may play a video game that is designed for presentation on a particular platform without any issues, but is playing the game on another platform that is more limiting, such as a smaller display. For example, the video game may be executing on a game console for display of video frames on a large screen television or screen monitor, but the user is actually viewing the game play on a mobile device (e.g., tablet, phone, etc.). This may occur when the video game is executing on a game console with video frames being transmitted to the mobile device for display, or the video frame is executing on the mobile device for display. In these cases, the content in the video frames may be compromised, such that one or more objects may not be displayed in a manner that is satisfactory to the user, including when content has been reduced to fit the screen of the mobile device so much such that the user is unable to easily view the content or portions of the content. It is in this context that embodiments of the disclosure arise. SUMMARY Embodiments of the present disclosure relate to cross-platform modification of content using artificial intelligence such that important assets meet a threshold of visibility requirement. Identification of issues when displaying content on a cross-platform device using artificial intelligence may be performed during development of a video game or in real-time during game play of the video game. In one embodiment, a method is disclosed. The method including executing a video game to generate a video frame for presentation on a first device of a first platform, wherein the video frame is generated for a game play of the video game by a user. The method including determining a target device, wherein the target device is of a second platform. The method including mapping the video frame from the first device to the target device, wherein the video frame that is mapped is presented on the target device. The method including determining an area of focus in a scene of the game play presented in the video frame based on a game context of the scene in the game play. The method including classifying an asset in the area of focus using a computer vision model implementing artificial intelligence. The method including determining that the asset is important in the game play of the video game using the computer vision model. The method including determining that the asset in the video frame that is mapped to the target device does not meet a threshold of visibility. The method including modifying the video frame that is mapped so that the asset meets the threshold of visibility. In another embodiment, a non-transitory computer-readable medium storing a computer program for execution by a processor to perform a method is disclosed. The computer-readable medium including program instructions for executing a video game to generate a video frame for presentation on a first device of a first platform, wherein the video frame is generated for a game play of the video game by a user. The computer-readable medium including program instructions for determining a target device, wherein the target device is of a second platform. The computer-readable medium including program instructions for mapping the video frame from the first device to the target device, wherein the video frame that is mapped is presented on the target device. The computer-readable medium including program instructions for determining an area of focus in a scene of the game play presented in the video frame based on a game context of the scene in the game play. The computer-readable medium including program instructions for classifying an asset in the area of focus using a computer vision model implementing artificial intelligence. The computer-readable medium including program instructions for determining that the asset is important in the game play of the video game using the computer vision model. The computer-readable medium including program instructions for determining that the asset in the video frame that is mapped to the target device does not meet a threshold of visibility. The computer-readable medium including program instructions for modifying the video frame that is mapped so that the asset meets the threshold of visibility. In still another embodiment, a computer system is disclosed, wherein the computer system includes a processor