KR-20260062433-A - Multimodal based road surface condition detection method and road surface condition detection system using the same

KR20260062433AKR 20260062433 AKR20260062433 AKR 20260062433AKR-20260062433-A

Abstract

The multimodal-based road surface condition detection method of the present invention is characterized by comprising: a step of receiving an image of a road surface and extracting a road image feature vector using a first learning algorithm; a step of receiving audio data of the road and extracting a road audio feature vector using a second learning algorithm; a weighted fusion step of fusing the road image feature vector and the road audio feature vector by assigning a pre-set weight to each of them; and a step of receiving the fused feature vector fused in the weighted fusion step and classifying and outputting the road surface condition using a pre-set classifier.

Inventors

김중현
이재민
김동성

Assignees

국립금오공과대학교 산학협력단

Dates

Publication Date: 20260507
Application Date: 20241029

Claims (10)

A step of receiving an image of a road surface and extracting a road image feature vector using a first learning algorithm; A step of receiving audio data of the above road and extracting road audio feature vectors using a second learning algorithm; A weighted fusion step for fusing the road image feature vector and the road audio feature vector by assigning preset weights to each of them; and A step of receiving a fused feature vector fused in the above weighted fusion step, classifying and outputting the road surface condition using a preset classifier; A multimodal-based road surface condition detection method including
In paragraph 1, A multimodal-based road surface condition detection method characterized by further including the step of determining whether the road surface condition is a condition for freezing to occur through environmental context information including road surface temperature, ambient environment temperature, and ambient environment humidity.
In paragraph 1, A multimodal-based road surface condition detection method characterized in that, in the classifier above, the fused feature vector is input into a fully connected layer, then passed to a softmax layer that determines the likelihood of each class, and is learned using an Adam optimizer that optimizes the loss function.
In paragraph 1, The above-mentioned first learning algorithm is, A multimodal-based road surface condition detection method characterized by analyzing and standardizing an image using the MobileNet architecture to extract the road image feature vector.
In paragraph 1, The above second learning algorithm is, A multimodal-based road surface condition detection method characterized by extracting the road audio feature vector from a Mel-spectrogram using the YAMNet architecture.
A first algorithm processing unit that receives an image of a road surface and extracts a road image feature vector using a first learning algorithm; A second algorithm processing unit that receives audio data of the above road and extracts road audio feature vectors using a second learning algorithm; A weighted fusion unit that fuses the road image feature vector and the road audio feature vector by assigning preset weights to each of them; and A classifier that receives a fused feature vector fused in the above-mentioned weighted fusion unit, classifies the road surface condition, and outputs it; A multimodal-based road surface condition detection system including
In paragraph 6, A multimodal-based road surface condition detection system characterized by further including a judgment unit that receives additional environmental context information including road surface temperature, ambient environment temperature, and ambient environment humidity to determine whether the road surface condition is a condition for freezing to occur.
In paragraph 6, A multimodal-based road surface condition detection system characterized in that, in the classifier above, the fused feature vector is input into a fully connected layer, then passed to a softmax layer that determines the likelihood of each class, and is learned using an Adam optimizer that optimizes the loss function.
In paragraph 6, The above-mentioned first learning algorithm is, A multimodal-based road surface condition detection system characterized by extracting road image feature vectors by analyzing and standardizing images using the MobileNet architecture.
In paragraph 1, The above second learning algorithm is, A multimodal-based road surface condition detection system characterized by extracting the road audio feature vector from a Mel-spectrogram using the YAMNet architecture.

Description

Multimodal-based road surface condition detection method and road surface condition detection system using the same The present invention relates to a technology for detecting road surface conditions that are frozen, i.e., black ice, and more specifically, to a multimodal-based road surface condition detection method and a road surface condition detection system using the same. In cold regions, most traffic accidents occur during the winter, primarily due to snow and ice. This leads to traffic delays and increases the likelihood of accidents. Similar to cold countries, Korea's hilly terrain and proximity to the sea make it prone to black ice formation, which is a major cause of winter traffic accidents. High humidity, frequent fog, and the presence of mountainous terrain provide an environment highly favorable for black ice to develop. Black ice is a nearly invisible, transparent layer on the road surface. Although invisible, it is difficult to predict because the wetness, snow, and ice conditions on this surface change rapidly. According to the Korea Transport Institute, 170 people died in traffic accidents related to black ice between 2015 and 2019, a level far more severe than accidents related to snow. Despite government efforts to reduce such accidents, the frequency of black ice accidents in winter increased from 6% in 2015 to 10% in 2018. Observations regarding this issue are not limited to Korea but are observed globally. In the United States, 23% of traffic accidents occur during the winter, and precipitation dramatically increases the risk of collisions and injuries. In Hokkaido, Japan, 90% of winter-related accidents are attributed to skidding, whereas in Sweden, road surface skidding is cited as the cause of 50% of winter accidents. The fatality rate from automobile accidents involving black ice peaks at 4.2% when temperatures are 2–3°C and drops to 0.7% when temperatures are between -1°C and 0°C. This pattern occurs because rising temperatures increase water vapor, which combines with cold road surfaces to form black ice. This highlights the urgent need for the development of black ice prediction and warning systems. Advancements in sensor technology have dramatically improved the ability to collect and utilize real-time weather data for specific road sections and unique conditions. Existing technologies for detecting road surface conditions include surface friction measurements, infrared thermal imaging, conductivity sensors, surface temperature sensors, Road Weather Information Systems (RWIS), optical sensors, microwave sensors, and radar sensors. However, these technologies have certain limitations, such as the impact on road debris and tire condition, the need for frequent maintenance, and the inability to immediately detect black ice. Road Weather Information Systems (RWIS) are comprehensive and utilize multiple sensors to monitor and predict road weather conditions. Nevertheless, their effectiveness is limited to specific regions and requires significant investment and ongoing maintenance. Optical sensors distinguish between dry, wet, and icy surfaces by examining the reflection and refraction of light. Despite their potential, their effectiveness can be diminished by environmental factors such as dust and moisture. Radar sensors can detect black ice by evaluating the reflection of radar signals, but this process is costly and requires careful calibration. Conversely, artificial intelligence (AI) approaches accurately detect black ice using sensor fusion and machine learning (ML) algorithms. These systems integrate data from multiple sources, such as temperature sensors, cameras, and microphones. Recent improvements in computational technology have led to increased use of deep learning (DL) techniques in pattern recognition applications. Convolutional Neural Networks (CNNs) and other deep learning (DL) models can identify subtle patterns and anomalies even in challenging situations. Through continuous learning and enhancement, AI models can adapt to new environments and provide comprehensive coverage. Capabilities that provide real-time alerts and predictive analytics enable the implementation of preventative safety measures that are impossible with existing technologies. A. Related Research and Existing Methods Vision-based models, such as deep learning algorithms, can identify road surface conditions, including black ice, by evaluating complex visual characteristics such as texture variations and color differences. Research on road surface condition identification has been conducted since the 1990s. For years, researchers have focused on developing technologies to determine road conditions in order to reduce accidents caused by slippery roads. In recent years, advancements in computing power and machine learning techniques have made it possible to easily analyze and extract information on road surface conditions. These models can detect these subtle signs even during the day. Research f