KR-102963493-B1 - Method for processing large volumes of data and system for performing the same

KR102963493B1KR 102963493 B1KR102963493 B1KR 102963493B1KR-102963493-B1

Abstract

The method for processing data generated by a mass data generation device according to the present invention includes the steps of collecting image data and metadata generated by the device, generating a mapping table of the collected data, a first compression step of converting the image data into a compressed image, and a second compression step of converting the compressed image into a video. A device for processing data generated by a mass data generation device and uploading it to a server comprises: a communication unit for acquiring image data and metadata generated by the device; and a processor for generating a mapping table including the image data and the metadata, converting the image data into a compressed image, converting the compressed image into a video, and controlling the communication unit to upload the video and the mapping table to a server. The present invention is a technology that enables the uploading of data generated from a device that generates large amounts of data to a server and allows for efficient management. It maps and stores image frames and metadata from large amounts of data generated in real time for management, compresses image data, and transmits it to a server, thereby enabling efficient data transmission and allowing image extraction from the server without data loss.

Inventors

이준하
김명균

Assignees

주식회사 파트리지시스템즈

Dates

Publication Date: 20260512
Application Date: 20250529

Claims (10)

In a method for processing data generated by a mass data generation device, A step of collecting image data and metadata generated by the above device; Step of creating a mapping table for collected data; A first compression step for converting the above image data into a compressed image; A second compression step for converting the above compressed image into a video; The method includes the step of transmitting the above video and the above mapping table to a server; The step of generating the above mapping table is, A step comprising: setting a reference time stamp in the mapping table and inputting an offset from the reference time stamp to synchronize with the collection time when reproducing data on the server above; Data processing method.
In Article 1, The above metadata includes at least one of a time stamp, an image illuminance value, an exposure value, or GPS information. Data processing method.
In Article 2, The step of generating the above mapping table is, A step comprising: assigning identification information to the image data and inputting a corresponding time stamp; Data processing method.
delete
delete
In Article 1, The above server has the step of decoding the above video into image data; The above server further comprises the step of playing the image data according to the timestamp listed in the mapping table. Data processing methods,
In Article 1 The above first compression step is, The method comprises the step of clustering image frames in which the similarity of the above metadata is greater than or equal to a preset value, and storing and compressing a reference key frame and an offset from the reference key frame within the cluster. Data processing method.
In a device that processes data generated by a mass data generation device and uploads it to a server, A communication unit that acquires image data and metadata generated by a device; A processor comprising: generating a mapping table including the image data and the metadata, converting the image data into a compressed image, converting the compressed image into a video, and controlling the communication unit to upload the video and the mapping table to a server; The above processor is, Assigning identification information to the above image data and creating a mapping table by inputting a corresponding time stamp, setting a reference time stamp in the mapping table for synchronization with the collection time when playing the data on the server, and inputting an offset from the reference time stamp. device.
In a computing system composed of a data processing device and a server, The data processing device acquires image data and metadata, generates a mapping table including the image data and metadata, converts the image data into a compressed image, converts the compressed image into a video, transmits the video and the mapping table to a server, sets a reference time stamp in the mapping table to synchronize with the time of collection when playing data on the server, and inputs an offset from the reference time stamp. The above server decodes the above video into the above image data and sequentially plays the above image data according to the time stamps listed in the mapping table, thereby performing synchronization analysis by aligning the time standard with other sensing data collected from the data processing device. System.
A computer-readable recording medium having a program stored on it for executing a method according to any one of claims 1 through 3, 6 and 7 on a computer.

Description

Method for processing large volumes of data and system for performing the same The present invention relates to a device for collecting large amounts of data, a method for processing data generated from a product, and an apparatus for performing the same. With the advancement of measurement and data storage technologies, the size, diversity, and complexity of data generated in real-time from various devices and products are increasing significantly. Consequently, its importance is growing as an asset that must be collected and managed for product performance verification and improvement, as well as post-launch quality control. Data is continuously generated throughout the entire product lifecycle, from the initial design phase of development through production and launch to the operational phase. This generated data is stored and managed on data collection and preservation devices that can be connected and configured in various forms, or on servers connected via the Internet or private networks. Examples of devices or products that generate large amounts of data in real time include factory production equipment and facilities, as well as vehicles. Taking vehicles as a more specific example, recently produced automobiles are equipped with devices such as cameras, radar, lidar, and CAN communication, generating massive amounts of binary data while driving. Consequently, all automotive manufacturers and automotive component developers collect and manage this data for product performance verification and quality control. To verify the performance of a camera product developed to improve driving performance, the camera is typically installed on approximately 100 vehicles. The data generated by driving these vehicles for a certain period (e.g., video and sensor data) is uploaded to a server for analysis to verify the product's performance. When each vehicle drives for one minute, approximately 10GB of data is generated and stored as files of various formats and configurations. If all data generated from the 100 vehicles used for verification were uploaded to the server, the volume of data subject to analysis would become very large, and the complexity of the file structure and number of files would also increase. Data generated by factory production equipment, facilities, or vehicles is produced at a very rapid pace, is large in size, and features a complex structure. Consequently, attempting to understand, systematize, and manage this data only after it has been aggregated on a server leads to significant inefficiency in data management tasks. To address this issue, a method is required for data collection devices to rapidly process large volumes of data. Existing methods for compressing image data into video compressed images by simply connecting them into a sequence; however, this resulted in the loss of temporal information, making synchronization with other collected data impossible during image acquisition and causing unnatural transitions between frames within the video. FIG. 1 is a block diagram briefly showing the configuration of a device according to one embodiment of the present invention. FIG. 2 is a flowchart of a method for processing large amounts of data according to an embodiment of the present invention. FIG. 3 is a drawing for showing the effect of a processing method according to an embodiment of the present invention, showing (a) a timestamp of an image actually received from a camera, and (b) a problem that occurs when saving the image as a video with an FPS of 30. FIG. 4 is an example of a data processing method according to one embodiment of the present invention. FIG. 5 is a diagram showing a mapping table according to one embodiment of the present invention. The aforementioned objects, features, and advantages of the present invention will become more apparent from the following detailed description in conjunction with the accompanying drawings. However, as the present invention is subject to various modifications and may have various embodiments, specific embodiments are illustrated in the drawings and described in detail below. Throughout the specification, identical reference numbers generally represent identical components. Additionally, components with identical functions within the same scope of concept appearing in the drawings of each embodiment are described using the same reference numeral, and redundant descriptions thereof are omitted. If it is determined that a detailed description of known functions or configurations related to the present invention could unnecessarily obscure the essence of the invention, such detailed description is omitted. Furthermore, numbers used in the description of this specification (e.g., First, Second, etc.) are merely identification symbols to distinguish one component from another. Furthermore, the suffixes "module" and "part" for components used in the following embodiments are assigned or used interchangeably solely for the ease of drafting