US-20260127027-A1 - INTEGRATED OBSERVATIONS PREPROCESSING SYSTEM AND METHOD FOR OCEAN DATA ASSIMILATION SYSTEM

US20260127027A1US 20260127027 A1US20260127027 A1US 20260127027A1US-20260127027-A1

Abstract

Provided herein is an integrated preprocessing system and method for observation data of an ocean data assimilation system, which can maximize efficiency in overall processes of observation data collection, quality control, data processing, and operation management, and enhance the stability and reliability of the preprocessing process. The integrated preprocessing system for observation data of an ocean data assimilation system according to the present disclosure includes: an observation data collection unit configured to collect ocean observation data required for the ocean data assimilation system; a data processing unit configured to perform quality control on the collected observation data and classify and process the collected observation data into a single executable file; and a monitoring unit configured to perform GUI-based job scheduling and monitoring using a Rose/Cylc operating system.

Inventors

Johan Lee
Sang-min Lee
Yu-Kyung HYUN

Assignees

National Institute of Meteorological Sciences

Dates

Publication Date: 20260507
Application Date: 20251031
Priority Date: 20241106

Claims (7)

1 . An integrated preprocessing system for observation data of an ocean data assimilation system, comprising: an observation data collection unit configured to collect ocean observation data; a data processing unit configured to perform quality control on the collected observation data and to classify and process the collected observation data into a single executable file; and a monitoring unit configured to perform GUI-based job scheduling and monitoring using a Rose/Cylc workflow management system, wherein the data processing unit comprises: a quality control unit including a QC module configured to apply QC flags to the collected observation data according to ocean depth, a refinement module configured to refine the collected observation data verified by the applied QC flags, and a conversion module configured to convert the refined observation data into a standardized data format in an explicit manner; and a single processing unit including a common subroutine module configured to classify and process the converted observation data into an executable file of a common subroutine, and a memory allocation module configured to process the converted observation data using a dynamic memory allocation method based on a linked list.
2 . The integrated preprocessing system of claim 1 , wherein the data processing unit comprises: a quality control unit configured to apply QC flags to the collected observation data, refine the collected observation data, and convert the refined observation data into standardized data; and a single processing unit configured to classify and process the converted observation data into a predetermined single executable file.
3 . The integrated preprocessing system of claim 1 , wherein the monitoring unit comprises: a parallel task management module configured to check task statuses in real time through a CYLC graph on a GUI basis using a Rose/Cylc workflow management system; and a log module configured to check logs of respective tasks on the GUI.
4 . An integrated preprocessing method for observation data of an ocean data assimilation system, comprising: (a) an observation data collecting step of collecting ocean observation data; (b) a data classifying and processing step of performing quality control on the collected observation data, and classifying and processing the collected observation data into a single executable file; and (c) a monitoring step of performing GUI-based job scheduling and monitoring using a Rose/Cylc workflow management system, wherein the step (b) comprises: a step of applying QC flags to the collected observation data according to ocean depth; a step of refining the collected observation data verified by the applied QC flags; a step of converting the refined observation data into a standardized data format in an explicit manner; a step of classifying and processing the converted observation data into an executable file of a common subroutine; and a step of processing the converted observation data using a dynamic memory allocation method based on a linked list.
5 . The integrated preprocessing method of claim 4 , wherein the step (b) comprises: (b1) a step of applying QC flags to the collected observation data, refining the collected observation data, and converting the refined observation data into standardized data; and (b2) a data processing step of classifying and processing the converted observation data into a predetermined single executable file.
6 . The integrated preprocessing method of claim 4 , wherein the step (c) comprises: a parallel task management step of checking task statuses in real time through a CYLC graph on a GUI basis using a Rose/Cylc workflow management system; and a step of checking logs of respective tasks on the GUI.
7 . A computer program stored in a medium for executing the integrated preprocessing method for observation data of an ocean data assimilation system according to claim 4 on a computer.

Description

CROSS-REFERENCE TO RELATED APPLICATION This application claims priority to Korean Patent Application No. 10-2024-0156454 filed on Nov. 6, 2024, and all the benefits accruing therefrom under 35 U.S.C. § 119, the contents of which are incorporated herein by reference in their entirety. BACKGROUND OF THE DISCLOSURE Field of the Disclosure The present disclosure relates to an integrated preprocessing system and method for observation data of an ocean data assimilation system. In particular, the present disclosure relates to an integrated preprocessing system and method for observation data of an ocean data assimilation system, which can enhance the stability and reliability of the preprocessing process. Description of the Related Art The Global Ocean Data Assimilation and Prediction System (GODAPS) has been operated since 2018, constructed based on the Forecasting Ocean Assimilation Models (FOAM), which is the operational system of the UK Met Office, in order to provide marine meteorological forecast information to the public, the Air Force, and related organizations (Chang et al., 2021). In order to independently produce and provide the ocean-sea ice initial fields for the improved climate prediction system (GloSea6), the previous Global Ocean Data Assimilation and Prediction System (GODAPS) was upgraded (GODAPS version 2, GODAPS2), and an operational system was established and put into operational service in October 2021. GODAPS was operated until February 2022 and then terminated. However, the preprocessing process of observation data in the previous Global Ocean Data Assimilation and Prediction System (GODAPS) has the following problems. 1) Since a simple scheduling tool such as crontab was used to manage tasks in the past, the task status had to be checked manually, and it was difficult to respond immediately when an error occurred. This caused task delays and led to a problem of lowering operational efficiency. Here, crontab refers to a scheduling tool used in UNIX or Linux operating systems to automatically execute periodically repeated tasks (e.g., backup, data processing). 2) In the existing system, the quality of the collected observation data was not finely managed, so there was a risk that low-quality data could be included in the data assimilation process. As a result, the possibility of data contamination increased, and this had a negative effect on prediction accuracy. 3) The existing system used a fixed memory allocation method, and thus, when processing large-volume data, memory usage was inefficient. In particular, for processing high-resolution satellite-observed sea surface temperature data, high-performance resources were required. This increased operating costs and reduced the flexibility of the system. 4) Since various observation data formats had to be managed with individual programs, whenever new data formats were added or changed, program updates and compilation were required, which increased the complexity of maintenance and could cause user confusion. Related Patent Document (Patent Document 1) Korean Registered Patent Publication No. 10-2220748 (Feb. 26, 2021) (Patent Document 2) Korean Registered Patent Publication No. 10-2492075 (Jan. 26, 2023) SUMMARY OF THE DISCLOSURE The purpose of the present disclosure, which is directed to solving the aforementioned conventional problems, is to provide an integrated preprocessing system and method for observation data of an ocean data assimilation system. The system and method can maximize efficiency in overall processes such as observation data collection, quality control, data processing, and operation management, and enhance the stability and reliability of the preprocessing. In addition, the purpose of the present disclosure is to provide an integrated preprocessing system and method for observation data in ocean data assimilation system, which incorporating centralized GUI-based management adopting the Rose/Cylc workflow management system, resource saving through dynamic memory allocation and enhanced data reliability using QC flags. In order to achieve the purpose, an aspect of the present disclosure provides an integrated preprocessing system for observation data of an ocean data assimilation system, comprising: an observation data collection unit configured to collect ocean observation data;a data processing unit configured to perform quality control on the collected observation data and to classify and process the collected observation data into a single executable file; anda monitoring unit configured to perform GUI-based job scheduling and monitoring using a Rose/Cylc workflow management system. In some exemplary embodiments, the integrated preprocessing system may further comprise an boundary data processing and generation unit for producing meteorological boundary data. In some exemplary embodiments, the data processing unit may comprise: a quality control unit configured to apply QC flags to the collected observation data, ref