Search

EP-4742192-A2 - ROUTE GUIDANCE DEVICE AND ROUTE GUIDANCE SYSTEM BASED ON AUGMENTED REALITY AND MIXED REALITY

EP4742192A2EP 4742192 A2EP4742192 A2EP 4742192A2EP-4742192-A2

Abstract

The present invention provides a route guidance device and a route guidance system. The route guidance device according to an embodiment of the present invention comprises: a communication part that communicates with a cloud server; an interface part that receives, from at least one sensor provided in a vehicle, a camera image including an image of a road on which the vehicle is traveling, and sensing information obtained by sensing a driving state of the vehicle; an MR module that renders MR information including at least one virtual object on the basis of the camera image, the sensing information, and map information received from the cloud server; and a processor that controls the interface part such that an MR view image including the MR information is displayed on a display part of the vehicle, wherein the processor, when a preset condition is satisfied, changes the MR view image to display a scene corresponding to a place at which the vehicle is to travel.

Inventors

  • JANG, Yujung
  • LEE, KIHYUNG
  • CHOI, Sunghwan
  • KIM, Seungman

Assignees

  • LG Electronics Inc.

Dates

Publication Date
20260513
Application Date
20220712

Claims (10)

  1. A route guidance device for a vehicle, comprising: a mixed reality module (900) that renders mixed reality (MR) information including a vehicle object (2411, 2550) corresponding to the vehicle and one or more virtual objects based on map information; an interface unit (1320) that interfaces with the vehicle; and, a processor that controls the MR module (900) and the interface unit (1320) so that, based on a user's selection, an MR view image including a vehicle object (2411, 2550) corresponding to the vehicle and a virtual vehicle (2412, 2511) corresponding to the vehicle object (2411, 2550) and driving ahead of the vehicle object (2411, 2550) along a route to be traveled by the vehicle is displayed on a display of the vehicle.
  2. The route guidance device of claim 1, wherein the processor displays a graphic object so that the user can select whether to display the MR view image including the virtual vehicle (2412, 2511) when the preset condition is met, and wherein the preset condition includes at least one of a case where the vehicle stops for a predetermined time, a case where the vehicle enters within a predetermined distance from a destination, a case where the vehicle enters within a predetermined distance from an intersection, and a case where a point of interest (POI) object displayed on the MR view image is selected.
  3. The route guidance device of claim 1, wherein the processor controls the interface unit (1320) to receive sensing information about a road and driving conditions on which the vehicle is traveling from at least one sensor located on the vehicle, and controls the MR module (900) and the interface device to display the MR view image of the virtual vehicle (2412, 2511) , reflecting the road and driving conditions received from the at least one sensor, on the display.
  4. The route guidance device of claim 1, wherein the route guidance device further includes a communication unit that communicates with at least one of a cloud server that provides the map information for an area including the vehicle's location, a traffic control server that provides traffic information, and a weather server that provides weather information around the vehicle, and wherein the processor controls the MR module (900) and the interface unit (1320) to display a driving image of the virtual vehicle (2412, 2511) on the display, reflecting real-time conditions surrounding the vehicle based on at least one piece of information received through the communication device.
  5. The route guidance device of claim 1, wherein the processor varies the altitude to display an image of the virtual vehicle (2412, 2511) driving ahead of the vehicle object.
  6. The route guidance device of claim 5, wherein the processor displays a first MR view image providing a view image viewing the virtual vehicle (2412, 2511) at a first altitude through different areas of the display, and a second MR view image providing a view image viewing the virtual vehicle (2412, 2511) at a second altitude higher than the first altitude.
  7. The route guidance device of claim 1, wherein the route guidance device further comprises an augmented reality (AR) module that renders augmented reality (AR) information based on an image of the vehicle's front acquired from the vehicle's camera, and wherein the processor displays an AR view image based on the AR information and an MR view image, including a virtual vehicle (2412, 2511) , corresponding to the vehicle object, driving ahead of the vehicle object (2411, 2550) along the route the vehicle will travel, on the display.
  8. The route guidance device of claim 1, wherein the processor displays map information corresponding to the current location of the vehicle on the display together with an MR view image including a virtual vehicle (2412, 2511) driving ahead of the vehicle object.
  9. The route guidance device of claim 1, wherein the processor stops playing the MR view image including the virtual vehicle (2412, 2511) when the virtual vehicle (2412, 2511) moves a predetermined distance along the route the vehicle is to proceed.
  10. The route guidance device of claim 1, wherein the MR view image is a scene viewed from a certain angle at a certain point on the digital twin map, and wherein the processor displays the scene corresponding to the location the vehicle is expected to drive to in the future as the MR view image, based on the scene viewed from a certain angle based on the virtual vehicle (2412, 2511).

Description

Technical Field The present disclosure relates to a route guidance device and a route guidance system for guiding a route for a vehicle to travel. Background Art Recently, Augmented Reality (AR) that outputs a graphic object through a windshield or a Head Up Display (HUD) of a vehicle or additionally outputs a virtual object to the real world by using a graphic object that is overlaid on an image captured by a camera has appeared. A vehicle is currently providing a driver with additional information related to an environment around the vehicle, a vehicle status, and a driving route (travel route) of the vehicle through the AR technology, and thus the driver can intuitively recognize the vehicle and the traveling environment of the vehicle. Therefore, traveling efficiency and convenience can be further improved. Meanwhile, when using such AR technology, various types of information necessary for driving a vehicle may be provided based on the real world. In other words, the AR technology uses images of the real world acquired through a camera, and requires acquisition of clear images of the real world. However, since a sensor, namely, a camera that acquires images of the real world senses a real-time environment around the vehicle, there is a problem that route guidance information cannot be accurately identified from the images acquired from the sensor, due to obstacles, such as rain, snow, shadows of street trees, or vehicles ahead in case of bad weather such as the rain or snow or in a complex traffic situation such as traffic jams. As one example, the camera may not be able to recognize a lane in which the vehicle is currently traveling due to snow, rain, shadows, or a vehicle ahead. Additionally, in the case of a road with different heights, such as a ramp, on which a vehicle travels, or a road with complex curves, the slope or curves of the road may not be recognized. In this case, there is a problem that AR objects related to lanes may not be displayed or incorrect AR objects may be displayed. In other words, there is a problem that discrepancy may occur between the AR object and the real environment depending on the complexity of the real world acquired through the camera or the state of an image obtained. Meanwhile, following this AR technology, a technology related to Mixed Reality (MR), which can provide various simulation information related to a vehicle by applying Digital Twin (DT) technology, is actively being developed. As an effort of developing such MR-related technologies, a method of providing information related to route guidance to a driver using the MR is being actively researched. The route guidance using the MR has an advantage of providing a driver with various types of information that the driver in a cockpit cannot check, such as displaying a graphic object corresponding to a vehicle on a 3D map digitized through the digital twinning technology and providing information related to a driving route on which the driver has not driven the vehicle yet through the map and the graphic object, or providing a field of view (viewing angle) such as a bird's-eye view. This MR provides vehicle-related information through virtual objects displayed through a digitized 3D map, and may provide information regardless of images of the real world obtained through a camera. Therefore, a problem that discrepancy may occur between provided information and an actual environment depending on the complexity of the real world acquired through the camera or the state of an image obtained. However, the MR provides information through images of a digitized 3D map. Therefore, depending on the degree of correspondence between the 3D map image and the real world around the vehicle, discrepancy may occur between a graphic object provided through the MR, that is, an MR object, and the real environment. However, it is very difficult to provide a 3D map that is completely identical to the real world, and thereby information related to stationary objects such as buildings or objects with a specific size or greater such as vehicles can be merely provided, but it is difficult to display objects, such as people or animals around the vehicles, which are small or difficult to be sensed, through the MR using the 3D map images. Due to this problem, it is difficult to completely replace AR, which directly uses images of the real world, with MR. Accordingly, technology development for effective ways to use both AR and MR is being actively researched. Disclosure of Invention Technical Problem The present disclosure is directed to solving those problems and other drawbacks. One aspect of the present disclosure is to make up for shortcomings of augmented reality (AR) by using mixed reality (MR), namely, to provide a route guidance device and a route guidance system capable of providing route guidance information using MR when it is difficult to provide the route guidance information through AR. Another aspect of the present disclo