EP-4738237-A1 - INSPECTION SYSTEM, INSPECTION METHOD, AND INSPECTION PROGRAM

EP4738237A1EP 4738237 A1EP4738237 A1EP 4738237A1EP-4738237-A1

Abstract

Provided are an inspection system, an inspection method, and an inspection program that can easily determine whether information included in each frame of an original low-quality moving image is accurate. A game program inspection system including a processor, a memory that stores instructions to be executed by the processor and a machine learning model trained using a plurality of training data sets, and a display, wherein the memory stores first to Nth (N is a natural number greater than or equal to 2) input frames each having a predetermined number of input pixels, along with attached information for each of the input frames, the processor edits nth (n = 2, 3, ..., N) attached information stored in the memory according to user instructions, the machine learning model outputs an nth estimated frame having a number of estimated pixels greater than or equal to the number of input pixels, based on first to n-1th input frames and the attached information, and an nth input frame and edited attached information, and the display displays the nth estimated frame.

Inventors

KAWAMATA, WATARU
YOKOTA, KENICHIRO
AMADA, Takashi
IKENOUE, SHOICHI
YASUE, NOBUYUKI
MURAMOTO, JUNICHI
WU, JUNG-HSUAN

Assignees

Sony Interactive Entertainment Inc.

Dates

Publication Date: 20260506
Application Date: 20240621

Claims (11)

A game program inspection system comprising: a processor; a memory that stores instructions to be executed by the processor and a machine learning model trained using a plurality of training data sets; and a display, wherein: the memory stores first to Nth (N is a natural number greater than or equal to 2) input frames each having a predetermined number of input pixels, along with attached information for each of the input frames; the processor edits nth (n = 2, 3, ..., N) attached information stored in the memory according to user instructions; the machine learning model outputs an nth estimated frame having a number of estimated pixels greater than or equal to the number of input pixels, based on first to n-1th input frames and attached information, and an nth input frame and edited attached information; and the display displays the nth estimated frame.
The inspection system according to claim 1, wherein: the processor: based on each of the input frames, acquires respective first to Nth intermediate frames by generating an intermediate frame corresponding to the input frame and having a number of intermediate pixels greater than or equal to the number of input pixels; and inputs each of the intermediate frames into the machine learning model and acquires first to Nth estimated frames each having a number of estimated pixels greater than the number of input pixels and equal to or greater than the number of intermediate pixels, the machine learning model comprises: a cumulative feature information output layer that receives an nth intermediate frame and n-1th auxiliary information based on n-1th cumulative feature information that indicates features of first to n-1th intermediate frames, and outputs nth cumulative feature information that indicates features of the first to nth input frames; and an estimated frame output layer that receives the nth cumulative feature information and outputs an nth estimated frame; and the machine learning model is trained using a plurality of training data sets, each of which comprises a learning intermediate frame having the number of intermediate pixels generated based on a learning input frame having the number of input pixels and a learning estimated frame having the number of estimated pixels.
The inspection system according to claim 2, wherein: the attached information comprises nth motion information, which is information indicating an amount and a direction of motion from the n-1th input frame to the nth input frame; and the processor acquires nth auxiliary information by applying motion compensation to the n-1th cumulative feature information based on the nth motion information.
The inspection system according to claim 3, wherein: the processor generates a difference image between an image obtained by applying the motion compensation to the n-1th input frame based on the nth motion information and an image represented by the nth input frame; and the display displays the difference image.
The inspection system according to claim 2, wherein each of the input frames is an image obtained by executing rendering of three-dimensional data depicting one or more objects as seen from a predetermined viewpoint.
The inspection system according to claim 5, wherein: the attached information comprises nth depth information indicating a depth of each pixel of the nth input frame; and the processor: identifies an nth emergent pixel, which is a pixel among the nth intermediate frames in which all or part of an object not displayed in the n-1th intermediate frame is displayed, based on n-1th depth information and the nth depth information; and acquires the n-1th auxiliary information by replacing a pixel value of the nth emergent pixel in the n-1th cumulative feature information with a predetermined value.
The inspection system according to claim 6, wherein: each of the input frames is an image obtained by executing rendering so that the viewpoint varies for each of the input frames; and the processor acquires variation information, which is information relating to variation of the viewpoint for each of the input frames in the rendering, and generates each of the intermediate frames found by interpolating a pixel value of a position corresponding to each pixel before variation in the input frame based on the variation information and each pixel of each of the input frames.
The game program inspection system according to claim 2, wherein the cumulative feature information output layer receives the first intermediate frame and given auxiliary information and outputs first cumulative feature information.
The inspection system according to claim 2, wherein the cumulative feature information is image information having the same number of pixels as the number of intermediate pixels.
An inspection method, wherein: a memory stores first to Nth (N is a natural number equal to or greater than 2) input frames having a predetermined number of input pixels, and attached information of each of the input frames; a processor edits nth (n = 2, 3, ..., N) attached information stored in the memory according to user instructions; a machine learning model trained using a plurality of training data sets outputs an nth estimated frame having a number of estimated pixels greater than or equal to the number of input pixels, based on first to n-1th input frames and the attached information, and an nth input frame and edited attached information; and the display displays the nth estimated frame.
An inspection program for causing a computer to function as: memory for storing first to Nth (N is a natural number greater than or equal to 2) input frames each having a predetermined number of input pixels, along with attached information for each of the input frames; editor for editing nth (n = 2, 3, ..., N) attached information stored in the memory according to user instructions; output for outputting an nth estimated frame having a number of estimated pixels greater than or equal to the number of input pixels based on first to n-1th input frames and the attached information and an nth input frame and edited attached information by a machine learning model trained using a plurality of training data sets; and display for displaying the nth estimated frame.

Description

[Technical Field] The present disclosure relates to an inspection system, an inspection method, and an inspection program. [Background Art] Conventionally, techniques for using a machine learning model to estimate a high quality still image based on a low quality still image (super-resolution) is known (see Non-Patent Literature 1 below). [Citation List] [Non-Patent Literature] [Non-Patent Document 1] Chao Dong, Chen Change Loy, Kaiming He, Xiaoou Tang. Learning a Deep Convolutional Network for Image Super-Resolution, in Proceedings of European Conference on Computer Vision (ECCV), 2014 [Summary of Invention] [Technical Problem] In super-resolution of moving images, it is believed that moving images of higher image quality can be estimated by taking into consideration not only information about each frame to be processed but also information about a past frame of that frame. On the other hand, in an image processing system that estimates an image taking into account information about a past frame when estimating high-quality moving images, it is difficult to determine whether the information contained in each frame input to the system is accurate for each frame to be input into the system. An object of the present disclosure is to provide an inspection system, inspection method, and inspection program capable of easily determining whether the information contained in each frame of the original low-quality moving images is accurate in an image processing technique that uses information from a past frame to estimate high-quality moving images based on low-quality moving images. [Solution to Problem] A game program inspection system of the present disclosure is a game program inspection system including a processor, a memory that stores instructions to be executed by the processor and a machine learning model trained using a plurality of training data sets, and a display, wherein the memory stores first to Nth (N is a natural number greater than or equal to 2) input frames each having a predetermined number of input pixels, along with attached information for each of the input frames, the processor edits nth (n = 2, 3, ..., N) attached information stored in the memory according to user instructions, the machine learning model outputs an nth estimated frame having a number of estimated pixels greater than or equal to the number of input pixels, based on first to n-1th input frames and the attached information, and an nth input frame and edited attached information, and the display displays the nth estimated frame. [Brief Description of Drawings] [FIG. 1] A diagram illustrating one example of a hardware configuration of an image processing system and an inspection system.[FIG. 2] A diagram illustrating an overview of the image processing system.[FIG. 3] A diagram schematically illustrating processing in the image processing system.[FIG. 4] A functional block diagram illustrating one example of functions realized by the image processing system.[FIG. 5] A diagram describing processing at a renderer.[FIG. 6] A diagram describing processing at an intermediate frame acquisition.[FIG. 7] A flowchart illustrating one example of a processing flow executed at the image processing system.[FIG. 8] A diagram describing processing executed at the inspection system.[FIG. 9] A diagram describing the processing executed at the inspection system.[FIG. 10] A diagram illustrating information stored by a rendering information memory for each frame. [Description of Embodiments] One example of an embodiment of an image processing system according to the present disclosure will be described below with reference to the drawings. [1. Hardware Configuration of Image Processing System] FIG. 1 is a diagram illustrating one example of a hardware configuration of an image processing system 1 and inspection system 2. The image processing system 1 is, for example, a computer such as a game console (game device). The inspection system 2 is, for example, a computer in which a program for executing a game program is installed. As illustrated in FIG. 1, the image processing system 1 and inspection system 2 includes a controller 10, a memory 12, a communication 14, an operation 16, a display 18, and an audio output 19. The controller 10, for example, includes a program control device such as a CPU that operates according to a program installed in the image processing system 1 and inspection system 2. The controller 10 also includes a GPU (Graphics Processing Unit) that renders images in a frame buffer based on graphics commands and data supplied from the CPU. The memory 12 includes, for example, a main memory such as ROM or RAM, and an auxiliary memory such as an HDD or an SSD. The memory 12 stores programs such as those executed by the controller 10. The memory 12 stores instructions to be executed by the processor and a machine learning model trained using a plurality of training data sets. For instance, the memory 12 stores, for example, a game