EP-3924866-B1 - SYSTEMS AND METHODS FOR IMAGE NORMALIZATION

EP3924866B1EP 3924866 B1EP3924866 B1EP 3924866B1EP-3924866-B1

Inventors

TAMURA, Kazuki
SHIMIZU, HIROAKI
PROESMANS, Marc
VERBIEST, Frank
VAN GOOL, LUC

Dates

Publication Date: 20260506
Application Date: 20190215

Claims (13)

A system for producing a virtual image view for a vehicle, comprising: a plurality of image capture means including different types of cameras configured to capture image data in proximity to the vehicle, the image data being defined at least in part by viewpoint parameters, and to provide an identifier identifying the respective plurality of image capture means; storage means configured to store a plurality of virtualization records containing conversion information related to a virtualized viewpoint and a plurality of image capture means; and processing means configured to: receive the captured image data; convert the viewpoint parameters of the captured image data into virtual viewpoint parameters based on the conversion information associated with a virtualization record stored by the storage means, to result in the virtual image view, wherein the virtualization record is identified at least based on the identifier; and execute at least one driver assistance and/or automated driving function based on the virtual image view; and wherein the conversion information includes calibration parameters for an identified image capture means, and wherein the processing means is configured to verify the calibration parameters for each of the plurality of image capture means.
The system for producing a virtual image view for a vehicle according to claim 1, wherein the conversion information comprises at least one of distortion compensation information, image rectification information, image refraction information, and rotational information.
The system for producing a virtual image view according to any of claims 1-2, wherein each virtualized viewpoint is defined by at least one of a resolution and a field of view; and at least one of a pinhole model, a fisheye model, a cylindrical model, a spherical model, and a rectilinear model.
The system for producing a virtual image view according to any of claims 1-3, wherein the storage means comprises means for identifying a processing application linked to a virtual image view.
The system for producing a virtual image view according to claim 4, wherein the storage means comprises a lookup table linking a camera identifier with conversion information for generation of the virtual image view required by the processing application.
The system for producing a virtual image view according to claim 5, wherein the processing application comprises at least one of an object detection and tracking application and a lane identifier application.
A vehicle comprising the system according to any of claims 1 to 6.
A computer-implemented method for producing a virtual image view for a vehicle, comprising: receiving image data captured from surroundings of the vehicle by a plurality of image capture means including different types of cameras, the image data being defined at least in part by viewpoint parameters; receiving an identifier identifying the plurality of image capture means providing the image data; determining a virtualization record containing conversion information related to a virtualized viewpoint and a plurality of image capture means, wherein the virtualization record is determined at least based on the identifier; converting the viewpoint parameters of the captured image data into virtual viewpoint parameters based on the conversion information associated with the determined virtualization record, to result in the virtual image view; and executing at least one driver assistance and/or automated driving function based on the virtual image view; wherein the conversion information includes calibration parameters for an identified image capture means, and the method further comprising verifying the calibration parameters for each of the plurality of image capture means.
The method for producing a virtual image view for a vehicle according to claim 8, wherein the conversion information comprises at least one of distortion compensation information, image rectification information, image refraction information, and rotational information.
The method for producing a virtual image view according to any of claims 8 to 9, wherein each virtualized viewpoint is defined by at least one of a resolution and a field of view; and at least one of a pinhole model, a fisheye model, a cylindrical model, a spherical model, and a rectilinear model.
The method for producing a virtual image view according to any of claims 8 to 10, wherein the determining includes identifying a processing application linked to a virtual image view.
The method for producing a virtual image view according to claim 11, wherein the determining includes searching a lookup table, the lookup table including information linking a camera identifier with conversion information for generation of the virtual image view required by the processing application.
The method for producing a virtual image view according to claim 12, wherein the processing application comprises at least one of an object detection and tracking application and a lane identifier application.

Description

Field of the Disclosure The present disclosure is related to systems and methods for automotive driver assistance, and more particularly, to virtualization of one or more images obtained from installed vehicle cameras. Background of the Disclosure Vehicles equipped with one or more cameras have become more common in recent years. These cameras have been introduced for various reasons, for example, visual feedback, parking assistance, collision avoidance, traffic guidance, etc., and have become an important input modality for autonomous driving and automated driving assistance. These cameras generally present different properties (e.g., optical properties), described by - for instance - pinhole and fisheye lens models, as well as different installation locations on a particular vehicle. Software designed and developed for automotive or automation applications typically expects images from a predefined viewpoint that is assumed for the one or more cameras installed in the vehicle, and the performance may degrade, or even cease, when the viewpoint provided by the one or more cameras is not within the assumed specifications. Some existing systems implement a camera device for improving recognition accuracy and robustness. The manufacturers of vehicles use front-mounted camera devices and means for combining signals with data relating to the driving state of the vehicle. Further still, means for combining image signals with navigation device output has also been implemented. US 2016/360104 discloses systems and methods for producing a combined view from a fisheye camera. US 2017/334356 discloses an image generation apparatus that generates a composite image viewed from an arbitrary virtual viewpoint. Summary of the Disclosure The present inventors have recognized that for mass production of a plurality of vehicle models, a number of issues arise because differences in viewing angle(s) in the camera setup for each specific and different vehicle type has a different geometry. This may result in viewpoint differences as well because of mounting constraints and/or limitations during the production process. The impact of these issues on the processing system may be complicated for traditional camera setups (e.g., pinhole), but the effect can be even more substantial for fisheye cameras. According to embodiments of the present disclosure, a system for producing a virtual image view for a vehicle is provided. The system includes a plurality of image capture means including different types of cameras configured to capture image data in proximity to the vehicle, the image data comprising viewpoint parameters, and to provide an identifier identifying the respective plurality of image capture means, storage means configured to store a plurality of virtualization records containing conversion information related to a virtualized viewpoint and a plurality of image capture means, and processing means. The processing means is configured to receive the captured image data, convert the viewpoint parameters of the captured image data into virtual viewpoint parameters based on the conversion information associated with a virtualization record stored by the storage means, to result in the virtual image view, wherein the virtualization record is identified at least based on the identifier, and execute at least one driver assistance and/or automated driving function based on the virtual image view; and wherein the conversion information includes calibration parameters for an identified image capture means, and wherein the processing means is configured to verify the calibration parameters for each of the plurality of image capture means. By providing such a system, data associated with one or more captured images and/or scenes surrounding a vehicle may be virtualized and normalized to a view desired by a particular processing application. For example, where a fisheye camera is provided on a side of a first vehicle model, and pinhole camera is provided on a side of second vehicle model, a single processing application installed on both the first and second vehicle may receive a nearly identical virtualized view, despite differences in camera type, camera install angle, camera location, etc. Thus, one processing application may be implemented across an entire fleet of mass produced models. In addition, because individual differences among cameras may exist, these differences may be rendered effectively undiscernible to the processing application by way of viewpoint parameter virtualization. The conversion information may include at least one of distortion compensation information, image rectification information, image refraction information, and rotational information. Each virtualized viewpoint may be defined by at least one of a resolution and a field of view, and at least one of a pinhole model, a fisheye model, a cylindrical model, a spherical model, and a rectilinear model. The storage means may include means for identifying a pr