US-12620142-B2 - Remote operation system and remote operation composite system

US12620142B2US 12620142 B2US12620142 B2US 12620142B2US-12620142-B2

Abstract

A remote operation system has respective mutual communication functions of communicating with a work machine ( 40 ) and a remote operation apparatus ( 20 ) and includes a support processing element ( 101 ) configured to transmit, to the remote operation apparatus ( 20 ), a combined image in which an index image is superimposed at least partially on the work machine ( 40 ) or a region around the work machine ( 40 ) in a captured image obtained through an imaging device disposed around the work machine. The index image is generated based on a mode of appearance of the lower traveling body ( 430 ) in the captured image and a mode of appearance of a designated component in the captured image. The designated component is unevenly disposed in a front or back direction of the lower traveling body ( 430 ). The index image indicates at least one of the front and back directions of the lower traveling body ( 430 ).

Inventors

Hitoshi Sasaki
Seiji Saiki
Yoichiro Yamazaki

Assignees

KOBELCO CONSTRUCTION MACHINERY CO., LTD.

Dates

Publication Date: 20260505
Application Date: 20220330
Priority Date: 20210827

Claims (11)

1 . A remote operation system having respective mutual communication functions of communicating with a work machine and a remote operation apparatus, the work machine including a lower traveling body and an upper revolving body, the upper revolving body being revolvable with respect to the lower traveling body, the remote operation apparatus being for remotely operating the work machine, the remote operation system comprising a support processing element configured to transmit, to the remote operation apparatus, a combined image in which an index image is superimposed at least partially on the work machine or a region around the work machine in a captured image obtained through an imaging device disposed around the work machine, the index image being generated based on a mode of appearance of the lower traveling body in the captured image and a mode of appearance of a designated component in the captured image, the designated component being unevenly disposed in a front or back direction of the lower traveling body, the index image indicating at least one of the front and back directions of the lower traveling body, wherein the support processing element superimposes the index image at least partially on the work machine or the region around the work machine in the captured image at an angle corresponding to a revolution angle of the upper revolving body with respect to the lower traveling body based on the mode of the appearance of the lower traveling body and the mode of the appearance of the designated component which change in accordance with a revolution of the upper revolving body with respect to the lower traveling body.
2 . The remote operation system according to claim 1 , wherein a plurality of the imaging devices is disposed at positions whose directions are different from each other as viewed from the work machine, and when the designated component of the lower traveling body does not appear in the one captured image obtained through the one imaging device, the support processing element superimposes an index image on the captured image, the index image being generated based on a mode of appearance of the lower traveling body and a mode of appearance of the designated component in another captured image obtained through another imaging device, the index image indicating the front or back direction of the lower traveling body.
3 . The remote operation system according to claim 2 , wherein the support processing element recognizes the revolution angle of the upper revolving body with respect to the lower traveling body based on a number of the designated components appearing in the one captured image and a number of the designated components appearing in the other captured image.
4 . The remote operation system according to claim 1 , wherein the support processing element recognizes the revolution angle of the upper revolving body with respect to the lower traveling body, the revolution angle being based on a position of the designated component in a left or right direction in the one captured image, the position serving as the mode of the appearance of the designated component included in the one captured image.
5 . The remote operation system according to claim 1 , wherein the support processing element obtains the captured image including the designated component having a three-dimensional shape, the designated component being formed by protruding from the lower traveling body.
6 . The remote operation system according to claim 1 , wherein the support processing element transmits, to the remote operation apparatus, the combined image in which the index image is superimposed on the one captured image based on image coordinates of the other captured image, a position at which the one imaging device is disposed, and an imaging area of the one captured image.
7 . The remote operation system according to claim 1 , wherein the lower traveling body of the work machine includes a pair of left and right crawlers, and the support processing element generates the index image in accordance with sizes of the crawlers in the combined image and positions of the crawlers in the combined image.
8 . The remote operation system according to claim 1 , wherein the support processing element obtains operation acceptance information that is information indicating whether or not an operation on the work machine is acceptable to the remote operation apparatus and transmits the combined image to the remote operation apparatus when the operation acceptance information indicates that the operation on the work machine is acceptable.
9 . The remote operation system according to claim 1 , wherein the support processing element obtains traveling operation information for a preparation action that is an action representing an intention of an operation input for moving forward or moving backward the work machine in the remote operation apparatus, transmits, to the remote operation apparatus, the combined image generated by using the index image indicating a forward direction of the lower traveling body when the intention represented by the preparation action is the operation input for moving forward, and transmits, to the remote operation apparatus, the combined image generated by using the index image indicating a backward direction of the lower traveling body when the intention represented by the preparation action is the operation input for moving backward.
10 . The remote operation system according to claim 1 , wherein a plurality of the work machines is disposed close to each other, and any one of the support processing element obtains positional information related to a position of the work machine that is a candidate for an operation target in a real space and recognizes the work machine that is the candidate for the operation target as the work machine that is the operation target when the position of the work machine calculated from the captured image matches the positional information related to the position of the work machine that is the candidate for the operation target in the real space.
11 . A remote operation composite system comprising: the remote operation system according to claim 1 ; and at least one of the remote operation apparatus and the work machine.

Description

TECHNICAL FIELD The present invention relates to a remote operation system having respective mutual communication functions of communicating with a work machine and a remote operation apparatus for remotely operating the work machine. BACKGROUND ART It has been practiced to provide a plurality of cameras that each takes an image of an area around a work machine or a plurality of cameras that is capable of imaging the work machine at angles different from each other, so as to generate a bird's-eye image of the work machine by using a plurality of captured images taken by the plurality of cameras. As an example of such a work machine, Patent Literature 1 discloses a hydraulic excavator comprising, for example, a plurality of cameras that each image an area around the hydraulic excavator and a bird's-eye image generator that generates a bird's-eye image based on images taken by the plurality of cameras. In addition, the work machine such as a hydraulic excavator comprises, for example, a lower traveling body and an upper revolving body that is revolvable with respect to this lower traveling body. The upper revolving body is also provided with a cab included in an operator's room. A revolution of the upper revolving body with respect to the lower traveling body in such a work machine may cause a front direction of the cab, and a forward direction or a backward direction of the work machine to be different from each other. This raises a problem in that it is difficult for an operator to learn in which direction the work machine advances when performing a traveling operation. Accordingly, an advancing direction of the work machine has been indicated with an arrow on a monitor provided inside the cab. For example. Patent Literature 2 discloses a monitor device for a construction machine in which an upper revolving body is revolvable with respect to a lower traveling body. The construction machine comprises a camera that takes an image of an area around the machine. The monitor device displays the image taken by the camera on a monitor screen disposed inside a cab. CITATION LIST Patent Literature Patent Literature 1: Japanese Patent Laid-Open No. 2020-112030 Patent Literature 2: Japanese Patent No. 5473870 SUMMARY OF INVENTION Technical Problem It is, however, difficult for an operator to recognize the front or back direction of the lower traveling body by simply watching the bird's-eye image. This raises a problem in that the operator has to move forward or backward the lower traveling body to recognize the front or back direction of the lower traveling body. In addition, the monitor device in Patent Literature 2 estimates the angle of the lower traveling body relative to the upper revolving body by using a temporal change in the image displayed on the monitor screen. This requires complicated image processing for estimating the relative angle and raises a problem of a heavier processing load. An object of the present invention, which has been made in view of such problems, is to provide a remote operation system and a remote operation composite system including the remote operation system. The remote operation system makes it possible to reduce a processing load and indicate a forward direction or a backward direction of a work machine in a simple configuration. Solution to Problem A remote operation system according to the present invention for achieving the object is a remote operation system having respective mutual communication functions of communicating with a work machine and a remote operation apparatus. The work machine includes a lower traveling body and an upper revolving body. The upper revolving body is revolvable with respect to the lower traveling body. The remote operation apparatus is for remotely operating the work machine. The remote operation system includes a support processing element configured to transmit, to the remote operation apparatus, a combined image in which an index image is superimposed at least partially on the work machine or a region around the work machine in a captured image obtained through an imaging device disposed around the work machine. The index image is generated based on a mode of appearance of the lower traveling body in the captured image and a mode of appearance of a designated component in the captured image. The designated component is unevenly disposed in a front or back direction of the lower traveling body. The index image indicates at least one of the front and back directions of the lower traveling body. In addition, a remote operation composite system according to the present invention includes: the remote operation system; and at least one of the remote operation apparatus and the work machine. According to the remote operation system and the remote operation composite system according to the present invention, a combined image is generated in which an index image indicating a front or back direction of a lower traveling body of a work machine is superim