CN-116805294-B - Method for enhancing environment scene and automatic driving vehicle testing system

CN116805294BCN 116805294 BCN116805294 BCN 116805294BCN-116805294-B

Abstract

A method of enhancing an environmental scene includes obtaining an image from an autonomous vehicle, the image captured by a camera mounted on the autonomous vehicle and depicting an environment surrounding the autonomous vehicle, generating a virtual object graphic containing one or more virtual objects, generating an object enhanced image when the virtual object graphic is rendered on the image, generating a global scene graphic representing a weather macroscopically static effect, generating a weather dynamics effect graphic representing at least one specific weather dynamics effect, synthesizing the generated environment enhanced image based on the virtual object graphic, the weather global scene graphic, and the weather dynamics effect graphic, resulting in a visual representation of the environment as the environment would appear when subjected to predetermined weather conditions and traffic conditions, and inputting the synthesized environment enhanced image into an onboard vehicle controller of the autonomous vehicle such that the autonomous vehicle performs at least one autonomous operation based on the environment enhanced image.

Inventors

XU SHAOBING
LI SHANGYI
WANG JIANQIANG
LI SHENGBO
CHENG BO
LI KEQIANG

Assignees

清华大学

Dates

Publication Date: 20260505
Application Date: 20221202

Claims (14)

1. A method of enhancing an environmental scenario for use by an autonomous car during testing, the method comprising the steps of: obtaining an image representing scene information from an autonomous vehicle, wherein the image is captured by a camera mounted on the autonomous vehicle and depicts an environment in which the autonomous vehicle is traveling; Generating a virtual object graphic representing one or more virtual objects, generating an object enhanced image when the virtual object graphic is rendered on the image; Generating a global scene graph characterizing a macroscopic static effect of weather based on the object enhanced image to simulate a macroscopic visual effect of the environment under predetermined weather conditions, the global scene graph generated by one or more Artificial Intelligence (AI) techniques; Generating a weather dynamic effect graphic representing at least one specific weather dynamic effect, wherein the weather dynamic effect graphic is generated by one or more physical model-based computing techniques; Generating a composite weather object enhanced image based on the virtual object graphic, the global scene graphic, and the weather dynamic effect graphic, and Inputting the synthetic weather object enhancement image into an on-board vehicle controller of the autonomous vehicle such that the autonomous vehicle performs at least one autonomous operation based on the synthetic weather object enhancement image; The method further includes the step of obtaining camera pose information representing a pose of the camera during capture of the image by the camera, and wherein the pose of the one or more objects is determined based on the camera pose information such that the one or more objects appear realistic when rendered on the image; The method further includes the step of obtaining vehicle pose information representing a pose of the autonomous vehicle during capture of the image by the camera; the camera pose information and/or the vehicle pose information are used for generating the virtual object graph; the generating virtual object graphics characterizing one or more virtual objects includes performing rendering of an appearance of the object by a depth learning based algorithm, wherein the neural network rendered input includes potential factors controlling the shape and appearance of the virtual object and camera/object pose information; The generating a global scene graph that characterizes macroscopic static effects of weather includes modifying an image using a first AI model such that it appears to have a low level of illumination at night, modifying an image using a second AI model such that it appears to be moist to a road, and modifying an image using a third AI model such that it appears to be covered by snow.
2. The method of claim 1, wherein the virtual object graph is generated based on at least one AI technology.
3. The method of claim 1, wherein the method is performed by an onboard independent computer, and wherein the onboard independent computer is connected to and communicates with vehicle electronics of the autonomous vehicle.
4. The method of claim 1, wherein each step of the method is performed more than once to generate a plurality of synthetic weather object enhancement images and to input the plurality of synthetic weather object enhancement images into the onboard vehicle controller.
5. The method of claim 1, wherein the image is acquired at the autonomous vehicle and sent from the autonomous vehicle to a test site server, wherein the transmission of the image includes use of 5G cellular communications.
6. The method of claim 5, wherein the test site server is configured to perform the steps of generating a virtual object graphic, generating a global scene graphic, generating a weather dynamic effect graphic, generating a synthetic weather object enhancement image, and inputting the synthetic weather object enhancement image into the on-board vehicle controller of the autonomous vehicle.
7. The method of claim 6, wherein the step of inputting the synthetic weather object enhancement image into the on-board vehicle controller of the autonomous vehicle comprises transmitting the synthetic weather object enhancement image to the autonomous vehicle using 5G cellular communication.
8. The method of claim 1, wherein the at least one particular weather dynamic effect corresponds to the predetermined weather condition.
9. The method of claim 1, wherein the synthetic weather object enhanced image is input into the on-board vehicle controller in a manner that causes the on-board vehicle controller to appear as if the synthetic weather object enhanced image is a non-enhanced image.
10. The method of claim 1, wherein the predetermined weather condition is different from a current weather condition of the environment such that the global scene graph, when rendered with the object enhanced image, results in a visual representation of the environment as if the environment would appear when subjected to weather different from the current weather of the environment.
11. An autonomous vehicle testing system comprising one or more electronic processors and a non-transitory computer readable memory accessible to the one or more electronic processors and storing computer instructions; wherein the automated driving vehicle testing system, when the computer instructions are executed by the one or more electronic processors: Obtaining an image representing scene information from an autonomous vehicle, wherein the image is captured by a camera mounted on the autonomous vehicle and depicts an environment in which the autonomous vehicle is traveling; Generating a virtual object graphic representing one or more virtual objects, generating an object enhanced image when the virtual object graphic is rendered on the image; Generating a global scene graph characterizing a macroscopic static effect of weather based on the object enhanced image to simulate a macroscopic visual effect of the environment under predetermined weather conditions, the global scene graph generated by one or more Artificial Intelligence (AI) techniques; Generating a weather dynamic effect graphic representing at least one specific weather dynamic effect, wherein the weather dynamic effect graphic is generated by one or more physical model-based computing techniques; Generating a composite weather object enhanced image based on the virtual object graphic, the global scene graphic, and the weather dynamic effect graphic, and Inputting the synthetic weather object enhancement image into an on-board vehicle controller of the autonomous vehicle such that the autonomous vehicle performs at least one autonomous operation based on the synthetic weather object enhancement image; Wherein the autonomous vehicle testing system is further configured such that when the one or more electronic processors execute the computer instructions, the autonomous vehicle testing system is further operable to obtain camera pose information representing a pose of the camera during capture of the image by the camera, and wherein the pose of the one or more objects is determined based on the camera pose information such that the one or more objects appear realistic when rendered on the image; the autonomous vehicle testing system is further configured such that when the computer instructions are executed by the one or more electronic processors, the autonomous vehicle testing system further obtains vehicle pose information representing a pose of the autonomous vehicle during capture of the image by the camera; the camera pose information and/or the vehicle pose information are used for generating the virtual object graph; The generating virtual object graphics characterizing one or more virtual objects includes performing rendering of an appearance of the object through Visual Object Networks (VONs) and HoloGAN, wherein inputs to the neural network rendering include potential factors that control shape and appearance of the virtual object and camera/object pose information; The generating a global scene graph that characterizes macroscopic static effects of weather includes modifying an image using a first AI model such that it appears to have a low level of illumination at night, modifying an image using a second AI model such that it appears to be moist to a road, and modifying an image using a third AI model such that it appears to be covered by snow.
12. The autonomous vehicle testing system of claim 11, wherein at least one of the one or more electronic processors is a Graphics Processing Unit (GPU).
13. The autonomous vehicle testing system of claim 11, wherein the one or more electronic processors are mounted at a location remote from the autonomous vehicle, and wherein the autonomous vehicle is configured to send the image to a test site server and receive the synthetic weather object enhancement image from the test site server using 5G cellular communication.
14. The automated driven vehicle test system of claim 11 wherein the global scene graph comprises a graph that adds a predetermined lighting effect to the synthetic weather object enhancement image.

Description

Method for enhancing environment scene and automatic driving vehicle testing system Technical Field The invention relates to a method and a system for rendering scenes used by an autonomous vehicle during testing. Background There are no methods and systems in the prior art that use weather-related and traffic-related graphics to enhance images captured by an Autonomous Vehicle (AV). Disclosure of Invention A method of enhancing an environmental scene for use by an autonomous vehicle during a test, the method comprising the steps of obtaining an image representing scene information from an autonomous vehicle, wherein the image is captured by a camera mounted on the autonomous vehicle and depicts an environment in which the autonomous vehicle is traveling, generating a virtual object graphic representing one or more virtual objects, generating an object enhanced image when the virtual object graphic is rendered on the image, generating a global scene graphic representing a weather macroscopic static effect based on the object enhanced image to simulate a macroscopic visual effect of the environment under predetermined weather conditions, the global scene graphic being generated by one or more Artificial Intelligence (AI) techniques, generating a weather dynamic effect graphic representing at least one specific weather dynamic effect, wherein the weather dynamic effect graphic is generated by one or more physical model-based computing techniques, generating a synthetic weather object enhanced image based on the virtual object graphic, the global scene graphic and the weather dynamic effect graphic, and inputting the synthetic object enhanced image into a controller of the autonomous vehicle such that the autonomous vehicle is on-board the autonomous vehicle such that the weather dynamic vehicle performs at least one weather dynamic effect based on the weather dynamic object enhanced image. Drawings Preferred exemplary embodiments will hereinafter be described in conjunction with the appended drawings, wherein like numerals denote like elements, and wherein: According to a first embodiment, fig. 1 depicts a communication system including an Autonomous Vehicle (AV) and a test site server, which may be used to perform one or more of the methods described herein; FIG. 2 depicts a communication system including an Autonomous Vehicle (AV) and a test site server, which may be used to perform one or more methods described herein, according to a second embodiment; FIG. 3 is a flow chart of a method of enhancing an environmental scenario for an autonomous vehicle test, according to one embodiment; FIG. 4 is a flow chart of a method of enhancing an environmental scenario for use by an autonomous vehicle during testing of the autonomous vehicle, according to one embodiment; FIG. 5 is a flowchart of a process for generating and rendering virtual object graphics, which may be used as part of the methods of FIGS. 3 and 4, according to one embodiment; FIG. 6 is a flowchart of a process for generating weather-related graphics, which may be used as part of the methods of FIGS. 3 and 4, according to one embodiment. Detailed Description The systems and methods described herein enable images captured by an Autonomous Vehicle (AV) to be enhanced using weather-related and traffic-related graphics such that the enhanced images are displayed as a realistic visual representation of the environment as the environment would appear when subjected to predetermined or selected weather and/or other environmental conditions. In addition, the systems and methods described herein enable the enhanced image to be input into an onboard vehicle controller of the AV such that the AV performs at least one autopilot operation based on the enhanced image. This allows the AV to be tested under a variety of different weather conditions, such as rain or snow. In one embodiment, a virtual object graphic representing one or more virtual objects, such as a virtual vehicle or pedestrian (or other traffic-related object), is also generated and then combined with the weather-related graphic to form a composite enhanced image that is displayed as a realistic visual representation of the environment as if the environment would appear when subjected to the predetermined weather condition, and as if the virtual object were a real object present in the environment. This allows AV to be tested according to various combinations of traffic scenarios/conditions and weather conditions. According to some embodiments, the weather related graphic may be generated in two steps. First, a global scene graph is generated and when rendered with an object enhanced image, the global scene graph produces a visual representation of the environment as if the environment would appear when subjected to predetermined weather conditions. Second, a weather dynamic effect graphic is generated that represents at least one particular weather dynamic effect. As one example, the predetermined