EP-4092377-B1 - IMAGE PROCESSING DEVICE, IMAGING DEVICE, INFORMATION PROCESSING DEVICE, DETECTION DEVICE, ROADSIDE DEVICE, IMAGE PROCESSING METHOD, AND CALIBRATION METHOD

EP4092377B1EP 4092377 B1EP4092377 B1EP 4092377B1EP-4092377-B1

Inventors

ARATA KOJI
SAWADA RYUICHI
HATTORI SHIGETO
TANIGAWA HIRONOBU

Dates

Publication Date: 20260513
Application Date: 20210105

Claims (6)

An image processing device, comprising: an image interface that acquires a captured image from an imaging module; a memory in which positions of specific feature points in a world coordinate system and reference positions are stored, the specific feature points being located on a road surface or a floor surface in a captured image provided by the imaging module, the reference positions being positions of the specific feature points in a captured image provided by the imaging module at any specific timing; and a controller that detects the specific feature points in the captured image and calculates based on the specific feature points and the positions in the world coordinate system, a calibration parameter for calibrating a position and an orientation of the imaging module, wherein the controller detects the specific feature points in a captured image provided by the imaging module at a timing different from the specific timing, and recalculates the calibration parameter in response to finding a discrepancy between the position in the captured image and the reference position with regard to a predetermined percentage or more of the specific feature points, wherein the image processing device further comprises an information interface that acquires the positions of the specific feature points in the captured image provided by the imaging module and the positions of the specific feature points in the world coordinate system, wherein the controller obtains an initial value of the calibration parameter by calculation based on the positions acquired from the information interface and including the positions of the specific feature points in the captured image and the positions of the specific feature points in the world coordinate system, and the positions in the world coordinate system that are acquired by the information interface include positions calculated based on signals received by GPS receivers installed in locations corresponding to feature points.
The image processing device according to Claim 1, wherein the positions of the specific feature points in the world coordinate system are converted to positions in the captured image by using the calibration parameter and are stored in the memory as the reference positions by the controller.
The image processing device according to Claim 1 or 2, wherein in a case where not all the specific feature points are detectable in the captured image, the controller calculates the calibration parameter by using the feature points except for those undetectable in the captured image.
The image processing device according to Claim 3, wherein in a case where the number of the specific feature points detectable in the captured image is out of a predetermined range, the controller stops calculating the calibration parameter.
A roadside unit, comprising: an imaging module that captures an image to provide a captured image; and the image processing device according to claim 1.
An image processing method that is to be employed by an image processing device, the image processing method comprising: storing positions of specific feature points in a world coordinate system and reference positions, the specific feature points being located on a road surface or a floor surface in a captured image provided by an imaging module, the reference positions being positions of the specific feature points in a captured image provided by the imaging module at any specific timing; detecting the specific feature points in a captured image provided by the imaging module at a timing different from the specific timing; and calculating, based on the specific feature points and the positions in the world coordinate system, a calibration parameter for calibrating a position and an orientation of the imaging module in response to finding a discrepancy between the position in the captured image and the reference position with regard to a predetermined percentage or more of the specific feature points, wherein an initial value of the calibration parameter is obtained by calculation based on positions acquired from an information interface and including the positions of the specific feature points in the captured image and the positions of the specific feature points in the world coordinate system, and the positions in the world coordinate system that are acquired by the information interface include positions calculated based on signals received by GPS receivers installed in locations corresponding to feature points.

Description

Cross-Reference to Related Application The present application claims priority to Japanese Patent Application No. 2020-3721 filed on January 14, 2020 and Japanese Patent Application No. 2020-10286 filed on January 24, 2020. Technical Field The present invention relates to an image processing device, a roadside unit, and an image processing method. Background Art Studies have been conducted on deploying roadside units along roads to support various communications with passing vehicles. For example, roadside units are intended for use in alerting vehicles and pedestrians at intersections with obstructed view. To this end, such a roadside unit includes a camera that captures images of measurement subjects on a road, such as vehicles and pedestrians approaching an intersection. As vehicles and pedestrians on another road come close to the intersection, the roadside unit lets them know the positions of the measurement subjects. Camera calibration is used to calculate the positions of the measurement subjects on the basis of images captured by the camera. The camera calibration is the process of deriving a calibration parameter for the position and the orientation of the camera in a world coordinate system. An approach to calibrating a camera installed in a roadside unit is presented in PTL 1. The camera captures an image of a measurement vehicle having a mark placed thereon and detects the mark in the image. As the position of the measurement vehicle, the position of the mark in the world coordinate system is determined on the basis of information received by a GPS receiver. Sensors of different kinds may be used to locate measurement subjects in a wide area, in which case separate calibration for every sensor is required. An approach to performing axis adjustment for calibration of a millimeter-wave radar sensor and a camera that are mounted on a vehicle is presented in PTL 2. This approach employs a reflector and a target board that are integrated into a target unit. US 9 185 402 B2 discloses a method for updating camera geometric calibration utilizing scene analysis, said method comprising: receiving initial camera geometric calibration parameters with respect to at least one camera; discovering reference points of interest from an initial reference frame in a scene acquired by said at least one camera; applying said camera geometric calibration parameters to image coordinates of said discovered reference points of interest to provide real-world coordinates of said discovered reference points of interest; analyzing a subset of a video stream from said at least one camera to identify feature points of a current scene captured by said at least one camera that match said discovered reference points of interest; and providing a current update of said camera geometric calibration parameters with respect to said current scene. WO 2019/225681 A1 discloses a calibration device that comprises: an acquisition unit that acquires a plurality of video images captured by a plurality of imaging devices, each of the plurality of video images having a plurality of frames; an extraction unit that, for each of the plurality of video images, extracts a feature point from the relevant video image frames, and extracts a feature point from a region of a frame different from the relevant video image frame and in which a moving body appears; a matching unit that uses a plurality of feature points to carry out matching, and uses a plurality of feature points to carry out matching; and a reconstruction unit that, on the basis of a plurality of matching points as the plurality of matching results, calculates a parameter of the plurality of imaging devices. JP 2012 010036 A discloses a camera calibration system that comprises a measuring vehicle, which travels with a mark installed, and a roadside device having an image processing unit for recognizing the mark of the vehicle. The measuring vehicle contains a data transmitter for transmitting positional information and time information received by an installed GPS receiver. The roadside device, by using the image processing unit for recognizing the mark in the image taken by a camera for taking the image of the traveling measuring vehicle and set position information of the roadside device received by the GPS receiver as original points, calculates a distance between the two points from the positional information on the mark and the set position information on the roadside device, based on the time information and the positional information transmitted by the data transmitter, the image taken when the image processing unit recognizes the mark, and a coordinate of the mark. Citation List Patent Literature PTL 1: Japanese Unexamined Patent Application Publication No. 2012-010036PTL 2: Japanese Unexamined Patent Application Publication No. 2010-156609 Summary of Invention The present invention provides an image processing device according to claim 1, a roadside unit according to claim 5, and