US-20260124544-A1 - SYNCHRONIZATION OF AUDIO WITH TEMPO OF IN-GAME RELATED ACTIONS FOR A VIDEO CLIP OF A GAME PLAY OF VIDEO GAME

US20260124544A1US 20260124544 A1US20260124544 A1US 20260124544A1US-20260124544-A1

Abstract

A method including capturing information related to a game play of a video game presented in a video clip. The method includes executing an artificial intelligence (AI) model using the information to determine a game play tempo of in-game related actions for the game play of the video game. The method including executing the AI model to synchronize an audio track to the game play tempo. The method including overlaying the audio track that is synchronized to the video clip for presentation.

Inventors

Michael Harrison Prosinski

Assignees

SONY INTERACTIVE ENTERTAINMENT INC.

Dates

Publication Date: 20260507
Application Date: 20241101

Claims (20)

1 . A method, comprising: capturing information related to a game play of a video game presented in a video clip; executing an artificial intelligence (AI) model using the information to determine a game play tempo of in-game related actions for the game play of the video game; executing the AI model to synchronize an audio track to the game play tempo; and overlaying the audio track that is synchronized to the video clip for presentation.
2 . The method of claim 1 , wherein the capturing information includes: accessing game state data for the game play of the video game, wherein the AI model analyzes the game state data to determine the game play tempo.
3 . The method of claim 1 , wherein the capturing information includes: capturing a plurality of image frames of the video clip, wherein the AI model analyzes the plurality of image frames to determine the game play tempo.
4 . The method of claim 1 , wherein the capturing information includes: capturing biometric information related to a user controlling the game play of the video game, wherein the AI model analyzes the biometric information to determine the game play tempo.
5 . The method of claim 1 , wherein the executing the AI model to synchronize the audio track includes: manipulating a beat of the audio track to synchronize with one or more the in-game related actions.
6 . The method of claim 5 , wherein the in-game related actions includes user based actions, or character based actions of a character controlled by the user in the game play of the video game.
7 . The method of claim 1 , wherein the executing the AI model to synchronize the audio track includes one or more of the following: manipulating a volume of the audio track; or stretching the audio track; or stretching the audio track while keeping a pitch of the audio track; or shrinking the audio track; or shrinking the audio track keeping the pitch of the audio track.
8 . The method of claim 1 , further comprising: accessing the audio track, wherein the audio track includes at least one of the following: an audio segment broadcast during the video clip, wherein the audio segment is taken from a base sound track of the video game; or original music; or a play list of one or more songs selected by the user; or a play list of one or more songs of a genre selected by the user.
9 . The method of claim 1 , further comprising: executing the AI model to identify one or more key events in the game play of the video game presented in the video clip; executing the AI model to determine a context of the game play of the video game presented in the video clip; executing the AI model to determine a style of music corresponding with the one or more key events and the context; and executing the AI model to generate the audio track based on the style of music.
10 . The method of claim 1 , further comprising: accessing a highlight reel including a plurality of video clips of a plurality of game plays of a plurality of video games; accessing a plurality of audio segments broadcast during the plurality of video clips, wherein the plurality of audio clips is taken from a plurality of base sound tracks of the plurality of video games; and executing the AI model to generate an audio track based on the plurality of audio segments.
11 . The method of claim 1 , further comprising: extracting a plurality of features from the information, wherein the AI model analyzes the plurality of features to determine the game play tempo.
12 . The method of claim 11 , wherein the plurality of features includes at least one of the following: rate of user input configured for controlling the game play of the video game; or one or more controller inputs of a controller input sequence; or magnitude and timing for each of the one or more controller inputs.
13 . A computer system comprising: a processor; and memory coupled to the processor and having stored therein instructions that, if executed by the computer system, cause the computer system to execute a method comprising: capturing information related to a game play of a video game presented in a video clip; executing an artificial intelligence (AI) model using the information to determine a game play tempo of in-game related actions for the game play of the video game; executing the AI model to synchronize an audio track to the game play tempo; and overlaying the audio track that is synchronized to the video clip for presentation.
14 . The computer system of claim 13 , wherein in the method the capturing information includes: accessing game state data for the game play of the video game, wherein the AI model analyzes the game state data to determine the game play tempo.
15 . The computer system of claim 13 , wherein in the method the capturing information includes: capturing a plurality of image frames of the video clip, wherein the AI model analyzes the plurality of image frames to determine the game play tempo.
16 . The computer system of claim 13 , wherein in the method the executing the AI model to synchronize the audio track includes: manipulating a beat of the audio track to synchronize with one or more the in-game related actions, wherein the in-game related actions includes user based actions, or character based actions of a character controlled by the user in the game play of the video game.
17 . The computer system of claim 13 , wherein in the method the executing the AI model to synchronize the audio track includes one or more of the following: manipulating a volume of the audio track; or stretching the audio track; or stretching the audio track while keeping a pitch of the audio track; or shrinking the audio track; or shrinking the audio track keeping the pitch of the audio track.
18 . A non-transitory computer-readable medium storing a computer program for performing a method, the computer-readable medium comprising: program instructions for capturing information related to a game play of a video game presented in a video clip; program instructions for executing an artificial intelligence (AI) model using the information to determine a game play tempo of in-game related actions for the game play of the video game; program instructions for executing the AI model to synchronize an audio track to the game play tempo; and program instructions for overlaying the audio track that is synchronized to the video clip for presentation.
19 . The non-transitory computer-readable medium of claim 18 , wherein the program instructions for capturing information includes: program instructions for accessing game state data for the game play of the video game, wherein the AI model analyzes the game state data to determine the game play tempo.
20 . The non-transitory computer-readable medium of claim 18 , wherein the program instructions for executing the AI model to synchronize the audio track includes: program instructions for manipulating a beat of the audio track to synchronize with one or more the in-game related actions, wherein the in-game related actions includes user based actions, or character based actions of a character controlled by the user in the game play of the video game.

Description

TECHNICAL FIELD The present disclosure is related to synchronizing an audio track with a game play tempo corresponding with a video clip of a game play of a video game. In particular, artificial intelligence is used to identify in-game related actions corresponding with the game play in the video clip, and determine the game play tempo based on the in-game related actions. Further, artificial intelligence is used to manipulate the audio track to be in alignment with the game play tempo. In that manner, the video clip may be overlaid with a new audio track that provides a theatrical and/or impactful user experience. BACKGROUND OF THE DISCLOSURE Video games and/or gaming applications and their related industries (e.g., video gaming) are extremely popular and represent a large percentage of the worldwide entertainment market. Video games are played anywhere and at any time using various types of platforms, including gaming consoles, desktop computers, laptop computers, mobile phones, tablet computers, etc. When a video game is played, a base soundtrack accompanies the game play. The base soundtrack is typically created by the developer of the video game. This base soundtrack plays in the background, and is preselected for each of the scenes in the video game. For example, a scene may include walking along a path to reach a desired location in a virtual environment. The player may not necessarily be stressed in this scenario. As such, the corresponding soundtrack may produce peaceful sounds of lower volume that are indicative of a low stress part of the video game. On the other hand, another scene may include the final operations to complete a task that may be stressful for the player. In this case, the corresponding soundtrack may produce sounds that are loud and climactic, and indicative of a high stress part of the game. However, the base sound track may not satisfy the player playing the video game. That is, the user may find the base soundtrack unstimulating, and thus may be bored with the base soundtrack accompanying the game play of the video game. Many players may even turn down the volume of the base soundtrack, and instead play their own music on their sound systems. For example, some players may play classical music or hard rock in the background. Even though the player prefers this music over the base soundtrack, there are limitations, as the player selected music may not necessarily correspond with the actions in the game play of the video game It is in this context that embodiments of the disclosure arise. SUMMARY Embodiments of the present disclosure relate to synchronizing an audio track with a game play tempo corresponding with a video clip of a game play of a video game. Artificial intelligence is used to determine a game play tempo for a video clip of a game play of a video game based on identified in-game related actions (e.g., user based, character based, etc.). An audio track accompanying the video clip is synchronized with the game play tempo. The audio track may be the base soundtrack, user selected audio, or audio generated for the video clip using artificial intelligence. The video clip may be a recorded clip of a previous game play, which may stand-alone or may be incorporated into a highlight reel, wherein the audio track is manipulated during post processing after the game play. The video clip may also be generated for a live game play, wherein the audio track is dynamically manipulated during the game play. In that manner, an audio track for a video clip is newly generated and provides provide a more impactful experience for the viewer, wherein the audio track is manipulated to be in synchronization with the game play tempo of the video clip. For example, because the audio track is in synchronization with in-game related actions, such as those used for determining the game play tempo, the audio track innately supports and corresponds with the video clip to provide the viewer a more intimate experience. In one embodiment, a method is disclosed. The method including capturing information related to a game play of a video game presented in a video clip. The method including executing an artificial intelligence (AI) model using the information to determine a game play tempo of in-game related actions for the game play of the video game. The method including executing the AI model to synchronize an audio track to the game play tempo. The method including overlaying the audio track that is synchronized to the video clip for presentation. In another embodiment, a non-transitory computer-readable medium storing a computer program for performing a method is disclosed. The non-transitory computer-readable medium including program instructions for capturing information related to a game play of a video game presented in a video clip. The non-transitory computer-readable medium including program instructions for executing an artificial intelligence (AI) model using the information to determine