JP-7855109-B2 - Display system and display method

JP7855109B2JP 7855109 B2JP7855109 B2JP 7855109B2JP-7855109-B2

Inventors

北橋正光

Assignees

株式会社小松製作所

Dates

Publication Date: 20260507
Application Date: 20250325

Claims (6)

A visible light image acquisition unit acquires a visible light image showing the target being imaged, which is captured by a visible light imaging device installed on the work machine. An infrared image acquisition unit acquires an infrared image showing the target being imaged, which is captured by an infrared imaging device provided on the work machine , simultaneously with the acquisition of the visible light image . A visible light distance calculation unit calculates a visible light distance, which indicates the distance from the visible light imaging device to the object to be imaged , for each of the multiple first partitioned regions defined in the visible light image . An infrared distance calculation unit calculates an infrared distance indicating the distance from the infrared imaging device to the object to be imaged for each of a plurality of second partition regions defined in the infrared image to correspond to the first partition region , If the difference between the visible light distance and the infrared distance is greater than or equal to a distance threshold, a combining unit generates a composite image by combining the infrared image and the visible light image. The system includes a display output unit that outputs the composite image so that it is displayed on a display device , The synthesis unit synthesizes the second partition region and the visible light image such that the second partition region, whose difference is greater than or equal to a distance threshold, is superimposed on the first partition region of the visible light image corresponding to the second partition region. Display system.
The composite image includes a second partitioned region where the difference is greater than or equal to a distance threshold and a first partitioned region where the difference is less than a distance threshold. The display system according to claim 1 .
The visible light imaging device includes a first visible light camera and a second visible light camera. The visible light distance calculation unit calculates the visible light distance by stereo processing the visible light image captured by the first visible light camera and the visible light image captured by the second visible light camera. The display system according to claim 1 or claim 2 .
The infrared imaging device includes a first infrared camera and a second infrared camera. The infrared distance calculation unit calculates the infrared distance by stereo processing the infrared image captured by the first infrared camera and the infrared image captured by the second infrared camera. The display system according to any one of claims 1 to 3 .
The system includes a guideline generation unit that generates a guideline image indicating the distance from the work machine based on the infrared distance, The display output unit outputs the guideline image so that the guideline image is displayed on the display device. The display system according to any one of claims 1 to 4 .
To acquire a visible light image showing the target being imaged by a visible light imaging device installed on the work machine, Simultaneously with the acquisition of the visible light image, an infrared image showing the target being imaged is acquired by an infrared imaging device provided on the work machine. The visible light distance, which indicates the distance from the visible light imaging device to the object to be imaged, is calculated for each of the multiple first partition regions defined in the visible light image . The infrared distance, which indicates the distance from the infrared imaging device to the object to be imaged, is calculated for each of the multiple second partition regions defined in the infrared image to correspond to the first partition region . If the difference between the visible light distance and the infrared distance is greater than or equal to a distance threshold, the infrared image and the visible light image are combined to generate a composite image. This includes outputting the composite image so that it is displayed on a display device, The second partitioned region and the visible light image are combined such that the second partitioned region, whose difference is greater than or equal to a distance threshold, is superimposed on the first partitioned region of the visible light image corresponding to the second partitioned region. Display method.

Description

This disclosure relates to a display system and a display method. In the technical field related to display systems, an image processing device such as the one disclosed in Patent Document 1 is known. Japanese Patent Publication No. 2018-106239 Figure 1 is a schematic diagram showing a remote control system for a work machine according to an embodiment.Figure 2 is a perspective view showing a work machine according to an embodiment.Figure 3 is a perspective view showing a visible light imaging device and a far-infrared imaging device according to an embodiment.Figure 4 is a functional block diagram showing a remote control system for a work machine according to an embodiment.Figure 5 is a schematic diagram illustrating the visible light imaging device and the infrared imaging device according to the embodiment.Figure 6 is a schematic diagram illustrating the visible light image and infrared image according to the embodiment.Figure 7 is a schematic diagram illustrating the visible light imaging device and the infrared imaging device according to the embodiment.Figure 8 is a schematic diagram illustrating the visible light image and infrared image according to the embodiment.Figure 9 is a schematic diagram illustrating the method for generating a composite image according to this embodiment.Figure 10 is a schematic diagram illustrating the method for generating a composite image according to the embodiment.Figure 11 is a schematic diagram showing a composite image and a guideline image displayed in the display device according to the embodiment.Figure 12 is a flowchart showing the display method according to the embodiment.Figure 13 is a block diagram showing a computer system according to an embodiment. The embodiments described below will be explained with reference to the drawings, but this disclosure is not limited to these embodiments. The components of the embodiments described below can be combined as appropriate. Furthermore, some components may be omitted. [Remote control system] Figure 1 is a schematic diagram showing a remote control system 100 for a work machine 1 according to an embodiment. The remote control system 100 remotely controls the work machine 1 located at the work site. At least a part of the remote control system 100 is located in a remote control room 200 at the remote control site. The remote control system 100 comprises a remote control device 40, a display device 50, and a control device 60. The remote control device 40 is located in a remote control room 200 outside the work machine 1. The remote control device 40 is operated by an operator in the remote control room 200. The operator can operate the remote control device 40 while seated in the control seat 45. The display device 50 is located in a remote control room 200 outside the work machine 1. The display device 50 displays images of the work site. The work site images include images of a predetermined area surrounding the work machine 1. The images of the predetermined area surrounding the work machine 1 include images of the work object of the work machine 1. The work object of the work machine 1 includes the construction target of the work machine 1. The display device 50 includes a flat panel display such as a liquid crystal display (LCD) or an organic electroluminescence display (OLED). In embodiments, the display device 50 includes a plurality of flat panel displays arranged adjacent to each other. The display device 50 may consist of a single flat panel display. The operator operates the remote control device 40 while viewing the image of the work site displayed on the display device 50. The work machine 1 is remotely controlled by the remote control device 40. The control device 60 is located in a remote control room 200 outside the work machine 1. The control device 60 includes a computer system. The work machine 1 is equipped with a control device 300. The control device 300 includes a computer system. The control device 60 and the control device 300 communicate via the communication system 400. Examples of the communication system 400 include the Internet, a Local Area Network (LAN), a mobile phone network, and a satellite communication network. [Working machinery] Figure 2 is a perspective view showing a work machine 1 according to an embodiment. In this embodiment, the work machine 1 is a hydraulic excavator. The work machine 1 operates at the work site. As shown in Figure 2, the work machine 1 comprises a traveling body 2, a rotating body 3 supported by the traveling body 2, a work machine 4 supported by the rotating body 3, a hydraulic cylinder 5 that drives the work machine 4, a visible light imaging device 20, and an infrared imaging device 30. The mobile unit 2 is capable of moving while supporting the slewing unit 3. The slewing unit 3 is capable of rotating around the slewing axis RX while supported by the mobile unit 2. The work implement 4 includes a boom 4A connected to the slewing unit 3, an arm 4B