Search

EP-4372317-B1 - ROUTE GUIDANCE DEVICE AND ROUTE GUIDANCE SYSTEM BASED ON AUGMENTED REALITY AND MIXED REALITY

EP4372317B1EP 4372317 B1EP4372317 B1EP 4372317B1EP-4372317-B1

Inventors

  • JANG, Yujung
  • LEE, KIHYUNG
  • CHOI, Sunghwan
  • KIM, Seungman

Dates

Publication Date
20260506
Application Date
20220712

Claims (14)

  1. A route guidance device comprising: a communication unit (1310) that communicates with a cloud server; an interface unit (1320) that receives, from at least one sensor disposed in a vehicle, a camera image including an image of a road on which the vehicle is traveling, and sensing information obtained by sensing a traveling state of the vehicle; a mixed reality, MR, module (900) that renders MR information including at least one virtual object, based on the camera image, the sensing information, and map information received from the cloud server; and a processor (1330) that controls the interface unit so that an MR view image including the MR information is displayed on a display of the vehicle, characterized in that the processor (1330) converts the MR view image to display a scene corresponding to a place where the vehicle is to travel when a preset condition is satisfied, and the preset condition includes at least one of a case where the vehicle stops for a predetermined time, a case where the vehicle enters within a predetermined distance from a destination, a case where the vehicle enters within a predetermined distance from an intersection, and a case where a POI object displayed on the MR view image is selected.
  2. The route guidance device of claim 1, wherein the processor (1330) displays route information for guiding a route, along which the vehicle is to travel up to a destination, on the MR view image, and controls the MR view image to display a scene of a place ahead of a current location of the vehicle according to the route information, on the basis of the satisfaction of the preset condition.
  3. The route guidance device of claim 1, wherein an icon for representing the scene corresponding to the place where the vehicle is to travel is displayed on the MR view image, and the preset condition includes that the icon is selected by a user.
  4. The route guidance device of claim 1, wherein the MR view image is a scene looking at a digital-twin map, to which a current situation is reflected in real time, at one point at a predetermined viewing angle.
  5. The route guidance device of claim 4, wherein the processor (1330) outputs an MR view image reflecting the current situation in real time on the display of the vehicle when the scene corresponding to the place where the vehicle is to travel is displayed.
  6. The route guidance device of claim 1, wherein the processor (1330) outputs the MR view image to correspond to a scene ahead of the vehicle based on a vehicle object indicating a current location of the vehicle.
  7. The route guidance device of claim 6, wherein the processor (1330) generates a ghost car that runs ahead of the vehicle object along a route, on which the vehicle is to travel, based on the satisfaction of the preset condition, and converts the MR view image to display a scene ahead of the ghost car based on the ghost car.
  8. The route guidance device of claim 1, wherein the MR view image converted based on the satisfaction of the preset condition is a predicted view image that shows a situation ahead along a route, on which the vehicle is to travel, on a digital twin map in advance.
  9. The route guidance device of claim 1, wherein the processor (1330) displays in advance the scene corresponding to the place where the vehicle is to travel by varying an altitude of a point looking at a digital-twin map when the MR view image is converted based on the satisfaction of the preset condition.
  10. The route guidance device of claim 1, wherein the processor (1330) controls the interface unit to output, on the display of the vehicle, a first MR view image, which displays a scene ahead of the vehicle, and a second MR view image, which displays the scene corresponding to the place where the vehicle is to travel, based on a current location of the vehicle, when the preset condition is satisfied.
  11. The route guidance device of claim 10, wherein the processor (1330) outputs the second MR view image in a form of a pop-up window.
  12. The route guidance device of claim 10, wherein the processor (1330) splits a screen of the display disposed in the vehicle into a first area and a second area when the preset condition is satisfied, and controls the interface unit to output the first MR view image in the first area of the display, and output the second MR view image in the second area.
  13. The route guidance device of claim 10, wherein the second MR view image is an image playing a scene that the vehicle is to move by a predetermined distance along a route for the vehicle to travel from a scene, which the vehicle is viewing at the current location.
  14. The route guidance device of claim 10, wherein the processor (1330) outputs the second MR view image outputting a scene that the vehicle is to move by a predetermined distance along a route for the vehicle to travel, and then controls the second MR view image to disappear when the vehicle moves by the predetermined distance.

Description

Technical Field The present invention relates to a route guidance device for guiding a route for a vehicle to travel. Background Art Recently, Augmented Reality (AR) that outputs a graphic object through a windshield or a Head Up Display (HUD) of a vehicle or additionally outputs a virtual object to the real world by using a graphic object that is overlaid on an image captured by a camera has appeared. A vehicle is currently providing a driver with additional information related to an environment around the vehicle, a vehicle status, and a driving route (travel route) of the vehicle through the AR technology, and thus the driver can intuitively recognize the vehicle and the traveling environment of the vehicle. Therefore, traveling efficiency and convenience can be further improved. Meanwhile, when using such AR technology, various types of information necessary for driving a vehicle may be provided based on the real world. In other words, the AR technology uses images of the real world acquired through a camera, and requires acquisition of clear images of the real world. However, since a sensor, namely, a camera that acquires images of the real world senses a real-time environment around the vehicle, there is a problem that route guidance information cannot be accurately identified from the images acquired from the sensor, due to obstacles, such as rain, snow, shadows of street trees, or vehicles ahead in case of bad weather such as the rain or snow or in a complex traffic situation such as traffic jams. As one example, the camera may not be able to recognize a lane in which the vehicle is currently traveling due to snow, rain, shadows, or a vehicle ahead. Additionally, in the case of a road with different heights, such as a ramp, on which a vehicle travels, or a road with complex curves, the slope or curves of the road may not be recognized. In this case, there is a problem that AR objects related to lanes may not be displayed or incorrect AR objects may be displayed. In other words, there is a problem that discrepancy may occur between the AR object and the real environment depending on the complexity of the real world acquired through the camera or the state of an image obtained. Meanwhile, following this AR technology, a technology related to Mixed Reality (MR), which can provide various simulation information related to a vehicle by applying Digital Twin (DT) technology, is actively being developed. As an effort of developing such MR-related technologies, a method of providing information related to route guidance to a driver using the MR is being actively researched. The route guidance using the MR has an advantage of providing a driver with various types of information that the driver in a cockpit cannot check, such as displaying a graphic object corresponding to a vehicle on a 3D map digitized through the digital twinning technology and providing information related to a driving route on which the driver has not driven the vehicle yet through the map and the graphic object, or providing a field of view (viewing angle) such as a bird's-eye view. This MR provides vehicle-related information through virtual objects displayed through a digitized 3D map, and may provide information regardless of images of the real world obtained through a camera. Therefore, a problem that discrepancy may occur between provided information and an actual environment depending on the complexity of the real world acquired through the camera or the state of an image obtained. However, the MR provides information through images of a digitized 3D map. Therefore, depending on the degree of correspondence between the 3D map image and the real world around the vehicle, discrepancy may occur between a graphic object provided through the MR, that is, an MR object, and the real environment. However, it is very difficult to provide a 3D map that is completely identical to the real world, and thereby information related to stationary objects such as buildings or objects with a specific size or greater such as vehicles can be merely provided, but it is difficult to display objects, such as people or animals around the vehicles, which are small or difficult to be sensed, through the MR using the 3D map images. Due to this problem, it is difficult to completely replace AR, which directly uses images of the real world, with MR. Accordingly, technology development for effective ways to use both AR and MR is being actively researched. US 2021/190530 A discloses an XR device for providing XR contents, in which the XR device is designed to use information (for example, road course image based on a real time streaming service) acquired by a cloud server or a road image previously stored in its memory. Disclosure of Invention Technical Problem The present invention is directed to solving those problems and other drawbacks. One aspect of the present invention is to make up for shortcomings of augmented reality (AR) by using mixed reality (MR), n