US-12623338-B2 - Information processing device, robot manipulating system and robot manipulating method

US12623338B2US 12623338 B2US12623338 B2US 12623338B2US-12623338-B2

Abstract

A robot manipulating system includes a game terminal having a game computer, a game controller, and a display configured to display a virtual space, a robot configured to perform a work in a real space based on robot control data, and an information processing device configured to mediate between the game terminal and the robot. The information processing device supplies game data associated with a content of work to the game terminal, acquires game manipulation data including a history of an input of manipulation accepted by the game controller while a game program to which the game data is reflected is executed, converts the game manipulation data into the robot control data based on a given conversion rule, and supplies the robot control data to the robot.

Inventors

Yasuhiko Hashimoto
Masayuki Kamon
Shigetsugu Tanaka
Yoshihiko Maruyama

Assignees

KAWASAKI JUKOGYO KABUSHIKI KAISHA

Dates

Publication Date: 20260512
Application Date: 20230831
Priority Date: 20180810

Claims (15)

1 . An information processing device configured to mediate between a game terminal having a game computer configured to execute a game program, a game controller configured to accept an input of manipulation into the game computer, and a display configured to display a virtual space outputted from the game computer, and a robot configured to perform a work in a real space based on robot control data, the information processing device comprising: a game data generator configured to receive work data including information pertaining to the work to be performed by the robot, generate game data based on the work data, and transmit the game data to the game terminal; and a robot control data generator configured to receive game manipulation generated by the game terminal based on the game data, the game manipulation data including a history of the input of the manipulation accepted by the game controller while the game program to which the game data is reflected is executed, convert the game manipulation data into the robot control data by inputting the game manipulation data into a first learned model where a relation between the game manipulation data and the robot control data is learned, and transmit the robot control data to the robot.
2 . The information processing device of claim 1 , further comprising: a first learned model generator configured to learn the relation between the game manipulation data and the robot control data by using first learned data including the game manipulation data associated with the work data and the robot control data, and generate the first learned model.
3 . The information processing device of claim 1 , wherein the game data generator is configured to convert the work data into the game data based on a given conversion rule.
4 . An information processing device configured to mediate between a game terminal having a game computer configured to execute a game program, a game controller configured to accept an input of manipulation into the game computer, and a display configured to display a virtual space outputted from the game computer, and a robot configured to perform a work in a real space based on robot control data, the information processing device comprising: a game data generator configured to receive work data including information pertaining to the work to be performed by the robot, generate game data based on the work data, and transmit the game data to the game terminal; and a robot control data generator configured to receive game manipulation generated by the game terminal based on the game data, the game manipulation data including a history of the input of the manipulation accepted by the game controller while the game program to which the game data is reflected is executed, convert the game manipulation data into the robot control data, and transmit the robot control data to the robot, wherein the game data generator is configured to convert the work data into the game data based on a given conversion rule by inputting the work data into a second learned model where a relation between the work data and the game data is learned.
5 . The information processing device of claim 4 , further comprising: a second learned model generator configured to learn the relation between the work data and the game data by using second learned data including the work data and the game data associated with the work data, and generate the second learned model.
6 . A robot manipulating system, comprising: a game terminal having a game computer configured to execute a game program, a game controller configured to accept an input of manipulation into the game computer, and a display configured to display a virtual space outputted from the game computer; a robot configured to perform a work in a real space based on robot control data; and an information processing device comprising: a game data generator configured to receive work data including information pertaining to the work to be performed by the robot, generate game data based on the work data, and transmit the game data to the game terminal; and a robot control data generator configured to receive game manipulation data-generated by the game terminal based on the game data, the game manipulation data including a history of the input of the manipulation accepted by the game controller while the game program to which the game data is reflected is executed, convert the game manipulation data into the robot control data by inputting the game manipulation data into a first learned model where a relation between the game manipulation data and the robot control data is learned, and transmit the robot control data to the robot.
7 . The robot manipulating system of claim 6 , further comprising: a first learned model generator configured to learn the relation between the game manipulation data and the robot control data by using first learned data including the game manipulation data associated with the work data and the robot control data, and generate the first learned model.
8 . The robot manipulating system of claim 6 , wherein the game data generator is configured to convert the work data into the game data based on a given conversion rule.
9 . A method of manipulating a robot configured to perform a work in a real space based on robot control data by using a game terminal having a game computer configured to execute a game program, a game controller configured to accept an input of manipulation into the game computer, and a display configured to display a virtual space outputted from the game computer, the method comprising: generating game data based on work data including information pertaining to the work to be performed by the robot, and transmitting the game data to the game terminal; and converting game manipulation data, including a history of the input of the manipulation accepted by the game controller while the game program to which the game data is reflected is executed, that was generated by the game terminal based on the game data, into the robot control data by inputting the game manipulation data into a first learned model where a relation between the game manipulation data and the robot control data is learned, and transmitting the robot control data to the robot.
10 . The method of claim 9 , further comprising: generating the first learned model by learning the relation between the game manipulation data and the robot control data by using first learned data including the game manipulation data associated with the work data and the robot control data.
11 . The method of claim 9 , further comprising: converting the work data into the game data based on a given conversion rule.
12 . A method of manipulating a robot configured to perform a work in a real space based on robot control data by using a game terminal having a game computer configured to execute a game program, a game controller configured to accept an input of manipulation into the game computer, and a display configured to display a virtual space outputted from the game computer, the method comprising: generating game data based on work data including information pertaining to the work to be performed by the robot, and transmitting the game data to the game terminal; converting game manipulation data, including a history of the input of the manipulation accepted by the game controller while the game program to which the game data is reflected is executed, that was generated by the game terminal based on the game data, into the robot control data, and transmitting the robot control data to the robot; and converting the work data into the game data based on a given conversion rule by inputting the work data into a second learned model where a relation between the work data and the game data is learned.
13 . The method of claim 12 , further comprising: generating the second learned model by learning the relation between the work data and the game data by using second learned data including the work data and the game data associated with the work data.
14 . A robot manipulating system, comprising: a game terminal having a game computer configured to execute a game program, a game controller configured to accept an input of manipulation into the game computer, and a display configured to display a virtual space outputted from the game computer; a robot configured to perform a work in a real space based on robot control data; and an information processing device comprising: a game data generator configured to receive work data including information pertaining to the work to be performed by the robot, generate game data based on the work data, and transmit the game data to the game terminal; and a robot control data generator configured to receive game manipulation data-generated by the game terminal based on the game data, the game manipulation data including a history of the input of the manipulation accepted by the game controller while the game program to which the game data is reflected is executed, convert the game manipulation data into the robot control data, and transmit the robot control data to the robot, wherein the game data generator is configured to convert the work data into the game data based on a given conversion rule by inputting the work data into a second learned model where a relation between the work data and the game data is learned.
15 . The robot manipulating system of claim 14 , further comprising: a second learned model generator configured to learn the relation between the work data and the game data by using second learned data including the work data and the game data associated with the work data, and generate the second learned model.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application is a Continuation-in-Part of U.S. patent application Ser. No. 17/267,447 filed Feb. 9, 2021, which is the U.S. National Stage of International Application No. PCT/JP2019/031479 filed Aug. 8, 2019, which claims benefit of priority to Japanese Patent Application No. 2018-151917 filed Aug. 10, 2018, and Japanese Patent Application No. 2019-105728 filed Jun. 5, 2019, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD The present disclosure relates to a technology which unites a computer game and an industrial robot. BACKGROUND ART Conventionally, a technology which combines a computer game and a robot which is an actual object is proposed. Patent Documents 1 and 2 disclose this kind of technology. Note that the “computer game” as used herein is a general term for a game which operates on a computer and is performed between the computer and an actual human being, and it is also referred to as a “video game” etc. Patent Document 1 discloses that an industrial robot for welding and laser machining is provided with a game controller for a home computer game terminal as a controller for teaching and manipulational inputting. Correlation between the game controller to the industrial robot is performed through a well-known program using a personal computer which is commercially available. Patent Document 2 discloses a system provided with a real space where a plurality of physical agents (for example, gaming robots such as vehicles and their accessories) exist, a controller which receives an operating command to the physical agent by a user input, and a host device which mediates between the physical agent and the controller. The system performs a game which causes the plurality of physical agents to compete in the real space (for example, a car racing game). A host device operates the physical agent so that a virtual space under the user's control and the real space maintain a real-time equivalence to change a state of the real space or change a state of the virtual space. REFERENCE DOCUMENTS OF CONVENTIONAL ART Patent Documents [Patent Document 1] JP2012-139746A[Patent Document 2] JP2015-533534A DESCRIPTION OF THE DISCLOSURE Problem(s) to be Solved by the Disclosure A skillful game user can instantly grasp the state of the virtual space projected on a display of a computer-game terminal and input a corresponding command by operating the game controller intuitively and correctly. The present inventors have examined utilizing such game user's excellent skills as a resource for the industry. Patent Document 1 describes an effect that the robot operator can manipulate the industrial robot like a game. However, since the robot operator actually sees the real space where the industrial robot performs a work, he/she more strongly feels like working rather than gaming. In Patent Document 2, since the virtual space under the game user's control has the equivalence to the real space, the virtual space recognized by the game user never exceed the real space. Originally, the game aims at amusement. For the game user, the game in which he/she feels like working lacks in interest, which may lead to lowering of his/her motivation. The present disclosure proposes a system in which a robot performs a work (labor) by a game user inputting a manipulation while playing a game, without spoiling the aspect of amusement in which the user can enjoy an unusual virtual space. SUMMARY OF THE DISCLOSURE An information processing device according to one aspect of the present disclosure mediates between a game terminal having a game computer configured to execute a game program, a game controller configured to accept an input of manipulation into the game computer, and a display configured to display a virtual space outputted from the game computer, and a robot configured to perform a work in a real space based on robot control data. The information processing device includes a game data feeding part configured to supply game data associated with the content of work to the game terminal, a game manipulation data acquiring part configured to acquire game manipulation data including a history of the input of the manipulation accepted by the game controller while the game program to which the game data is reflected is executed, a first converting part configured to convert the game manipulation data into the robot control data based on a given conversion rule, and a robot control data feeding part configured to supply the robot control data to the robot. A robot manipulating system according to another aspect of the present disclosure includes a game terminal having a game computer configured to execute a game program, a game controller configured to receive an input of manipulation into the game computer, and a display configured to display a virtual space outputted from the game computer, a robot configured to perform a work in a real space based on robot con