KR-20260063759-A - Inter-monitor reliability measurement system for online monitoring

KR20260063759AKR 20260063759 AKR20260063759 AKR 20260063759AKR-20260063759-A

Abstract

The present invention relates to a system for measuring inter-monitor reliability for online monitoring. According to one embodiment of the present invention, the system comprises: a user terminal provided to an online monitoring agent; and a reliability measurement server connected to the user terminal via a network and measuring inter-monitor reliability based on a response from the user terminal. The reliability measurement server described above includes: a problem generation module that generates problems based on project requirements; a problem processing module that transmits the generated problems to a user terminal and receives and processes the response to the problem from the user terminal; a reliability measurement module for measuring the reliability of the response from the user terminal - wherein the reliability measurement module is configured to use the Cronbach's alpha coefficient when the input data is numeric and the Krippendorf's alpha coefficient when the input data is sentence-; and a reliability analysis module that determines the reliability as high if the reliability coefficient measured by the reliability measurement module is 0.6 or higher.

Inventors

박유경

Assignees

굿모니터링 주식회사

Dates

Publication Date: 20260507
Application Date: 20241031

Claims (4)

In a system for measuring inter-monitor reliability for online monitoring, User terminal provided to the online monitoring agent; and A reliability measurement server connected to the above-mentioned user terminal via a network and measuring reliability between monitors based on a response from the user terminal; comprising The above reliability measurement server is A problem generation module that generates problems based on project requirements; A problem processing module that transmits a generated problem to a user terminal and receives and processes a response to the problem from the user terminal; A reliability measurement module for measuring the reliability of responses from user terminals - where the reliability measurement module is configured to use the Cronbach's alpha coefficient when the input data is numeric and the Krippendorf's alpha coefficient when the input data is sentence- ; Characterized by including a reliability analysis module that determines the reliability as high when the reliability coefficient measured by the above reliability measurement module is 0.6 or higher. Inter-monitor reliability measurement system for online monitoring.
In paragraph 1, The above problem generation module generates problems considering the difficulty level, and if previous data exists, Characterized by the fact that the difficulty level is calculated as = (number of correct answers for individual questions ÷ total number of respondents) × 100 Inter-monitor reliability measurement system for online monitoring.
In paragraph 1, The above problem processing module operates to calculate the correct answer rate and the match rate, and * Correct Answer Rate = ((Number of Correct Answers ÷ Total Number of Responses) × 100) * Agreement Rate = ((Number of Correct Answers ÷ Total Number of Responses) × 100) Characterized by being obtained as Inter-monitor reliability measurement system for online monitoring.
In paragraph 1, Cronbach's alpha values range between 0 and 1; a value of 0.6 or higher is considered to indicate a stable level of reliability. Cronbach's alpha is calculated using the variance of each item and follows the following formula: A system for measuring inter-monitor reliability for online monitoring characterized by being obtained.

Description

Inter-monitor reliability measurement system for online monitoring The present invention relates to online monitoring, and more specifically, to a system capable of measuring inter-monitor reliability for online monitoring. According to the International Telecommunication Union (ITU) Annual Report, the number of internet users worldwide reached 5.4 billion in 2023, while domestically, according to the Informatization Survey, 29.92 million people used the internet in 2023. Futurist Alvin Toffler stated in his book *Power Shift* that the internet is the crystallization of information. The internet exists and has established itself as a part of reality. In particular, with the emergence of mobile devices and social network services (SNS), one-person media is growing and its influence is also increasing. The Internet has positive functions, such as the rapid exchange and sharing of information, improvement of quality of life, assistance in learning, and the formation of diverse networks and relationships. However, there are also negative aspects, including illegal information such as crimes involving information, the proliferation of unnecessary information, personal data leaks, and internet crimes. Examples include illegal online sales advertisements, violations of hazardous product labeling regulations, classification violations, copyright infringement, cyberbullying, and violations of personal information protection. As the level and intensity of these negative aspects gradually increase, relevant laws and systems have been established at the national level to conduct regular monitoring. Online monitoring is broadly divided into system monitoring and content monitoring. System monitoring refers to program and network monitoring, while content monitoring refers to the monitoring of content such as internet posts, online news, social media posts, videos, and games. In this invention, online monitoring refers to content monitoring. Along with the increase in users of online cafes, blogs, and websites, the development of social media has led to a surge in user-generated content, such as social media posts, YouTube videos, and posts from various communities. Governments and companies are conducting online monitoring of these posts to analyze public opinion trends and customer preferences. The data handled in online monitoring is primarily unstructured data. Unstructured data is data that is not organized, is highly diverse, and lacks consistency. Because unstructured data contains errors, inconsistencies, or irrelevant information, classification and judgment are critical, requiring significant time and cost. Consistency and reliability are essential when deploying a large number of monitoring personnel to collect unstructured data. Typically, monitors begin performing tasks and submitting deliverables after completing approximately two hours of training on predefined work rules. The deliverables undergo inspection, which involves checking for typos and errors, as well as reconfirming classification and judgment results. Classification involves sorting by designated categories, and for judgment purposes, the task involves assigning value judgment areas determined by the client, such as "inappropriate" or "problematic." Classification and judgment results are critical because they have a significant impact on data quality. However, these results often vary from monitor to monitor. Classification differs despite the existence of predefined rules, and judgment results frequently differ from monitor to monitor as well. In other words, subjective aspects play a significant role in the realm of classification and judgment by monitors, and there is a problem in that results vary because each monitor differs in experience, understanding, learning ability, and sociocultural background. FIG. 1 is a drawing showing an example of a reliability measurement system between monitors according to an embodiment of the present invention; FIG. 2 is a diagram showing the structure of a reliability measurement server of a reliability measurement system between monitors according to an embodiment of the present invention; FIG. 3 is a diagram showing the operational structure of a problem generation module of a reliability measurement server according to an embodiment of the present invention; FIG. 4 is a diagram showing the operational structure of a problem processing module of a reliability measurement server according to an embodiment of the present invention; FIG. 4a is a diagram showing examples of classification and judgment values input by a monitor; FIG. 4b is a diagram showing an example of a sentence-type case value input by a monitor; FIG. 4c is a diagram showing an example of a numerical case (before conversion) value input by a monitor; FIG. 5 is a diagram showing the operational structure of a reliability measurement module of a reliability measurement server according to an embodiment of the present invention; FIG. 5a is