KR-102964325-B1 - ON-DEVICE GENERATIVE AI AND ADAPTIVE ROLLING BUFFER-BASED MULTISENSOR REAL-TIME SITUATION AWARENESS AND CONTEXT ANALYSIS APPARATUS AND METHOD

KR102964325B1KR 102964325 B1KR102964325 B1KR 102964325B1KR-102964325-B1

Abstract

The present invention relates to an on-device Generative AI-based adaptive rolling buffer multi-sensor real-time situation awareness and context analysis device and method for activity assistance for the elderly, persons with disabilities, and workers in poor audiovisual environments. The device cyclically stores data collected from a camera, a microphone, and a composite environmental sensor in a rolling buffer module for a predetermined time interval, adaptively controls storage quality by calculating an activity index based on the stored data and reflecting weights according to the sensor change rate, user profile, and environmental conditions (noise, illuminance), detects events using data combined with the control results, analyzes the context of the situation based on data from the intervals before and after the detected event occurrence time, and the analysis results re-explain the meaning and cause of the event in natural language or multimodally by a Generative re-inference module linked to an AI inference unit, and provides alternative feedback of at least one of voice, text, LED, and vibration by an accessibility output control unit tailored to the user (visual or auditory constraints) and environmental conditions (high noise, low illuminance), the warning generation process is recorded in an audit/explanation log, and the analyzed data is encrypted and masked at the edge for privacy protection, after which the object This relates to a technology that transmits data to a server only in the form of feature-based data consisting of location, motion vector, audio spectrum, activity index, and event flag.

Inventors

이학준

Assignees

(주)싱스웰

Dates

Publication Date: 20260513
Application Date: 20251110

Claims (17)

An input unit that receives data from a camera, a microphone, and a composite environment sensor; A rolling buffer module that cyclically stores the received data over a predetermined time interval; An adaptive control unit that calculates a sensor change rate based on the stored data, calculates an activity index by reflecting the calculated change rate and weights according to user profiles and environmental conditions, and controls storage quality including video resolution, frame rate, audio sample rate, and sensor sampling period according to the activity index; An event detection unit that detects an event generated using the above-mentioned stored data and the above-mentioned storage quality control result; A context analysis unit that analyzes the context of a situation that occurred based on data before and after the time of the event occurrence in response to the detected event; A Generative re-inference module that receives the output of the above-mentioned context analysis unit and AI inference unit and pre- and post-event data, searches for relevant information from a knowledge database to reinforce the inference input, re-explains the meaning and cause of the event in natural language or multimodal form, and generates behavioral recommendations; An accessibility output control unit that applies priority to voice, text, LED, and vibration outputs according to user profile (visual/auditory constraints) and environmental conditions (noise/illumination); and An audit/explanatory log unit that ensures traceability by recording the output generation process of the above-mentioned Generative Re-inference Module and the basis for event detection by the above-mentioned Event Detection Unit. A multi-sensor real-time situation awareness and context analysis device including
In paragraph 1, A multi-sensor real-time situation awareness and context analysis device characterized by the above-mentioned composite environment sensor including an IMU, a gas sensor, an IAQ sensor, and GNSS.
In paragraph 1, A multi-sensor real-time situation awareness and context analysis device characterized by the adaptive control unit calculating the rate of change of the sensor data and calculating an activity index by reflecting the rate of change, a user profile, and weights according to environmental conditions (noise, illuminance).
In paragraph 3, A multi-sensor real-time situation awareness and context analysis device characterized by the adaptive control unit adaptively adjusting image resolution, frame rate, audio sample rate, and sensor sampling period using the calculated activity index and environment weight.
In paragraph 4, A multi-sensor real-time situation awareness and context analysis device characterized by the adaptive control unit storing at low resolution and low frame rate when the activity index is below a threshold, and storing at high resolution and high frame rate when the activity index is above a threshold.
In paragraph 4, A multi-sensor real-time situation awareness and context analysis device characterized by the adaptive control unit determining storage levels L0, L1, L2, and L3 according to the activity index, reflecting a storage quality policy corresponding to each level in the rolling buffer module, and enabling the storage quality policy to be overridden according to environmental conditions (noise, illumination) and failure mode.
In paragraph 1, A multi-sensor real-time situation recognition and context analysis device characterized by the above-mentioned event detection unit detecting at least one of a fall, collision sound, scream, gas concentration excess, air quality abnormality, and location abnormality.
In paragraph 1, A multi-sensor real-time situation awareness and context analysis device characterized by the above-mentioned rolling buffer module preserving data by extending the preceding and subsequent sections at the time of event occurrence.
In paragraph 8, A multi-sensor real-time situation awareness and context analysis device characterized by the above-described rolling buffer module recording a reference counter in the indexing to allow multiple events to reference the same data area in order to independently preserve each event segment when multiple events occur.
In paragraph 1, It further includes an AI inference unit that performs real-time inference based on multimodal input data in conjunction with the above-mentioned context analysis unit, and A multi-sensor real-time situation awareness and context analysis device characterized by the fact that, in conjunction with the output of the AI inference unit, the Generative re-inference module generates natural language or multimodal explanations and behavior recommendations based on the inference results and context information of the AI inference unit.
In paragraph 1, It further includes a privacy protection unit that performs encryption and masking processing on data received from the input unit and transmits only feature value-based data to the server, A multi-sensor real-time situation awareness and context analysis device characterized in that feature value-based data transmitted to the above server is limited to at least one of object location, motion vector, audio spectrum, activity index, and event flag, and the integrity of the transmission is guaranteed through hash chain logs and audit records.
A step of receiving data from a camera, microphone, and composite environment sensor; A step of cyclically storing the received data in a rolling buffer for a predetermined time interval; A step of calculating a sensor change rate based on the stored data, calculating an activity index by reflecting the calculated change rate and weights according to the user profile and environmental conditions (noise, illuminance), and controlling storage quality including video resolution, frame rate, audio sample rate, and sensor sampling period according to the activity index; A step of detecting an event using the stored data and the storage quality control result; A step of analyzing the context of a situation that occurred based on data before and after the time of the event occurrence in response to the detected event; A step of receiving the results of the above-mentioned context analysis step and AI inference unit, as well as pre- and post-event data, and performing a Generative re-inference step that searches for relevant information from a knowledge database to reinforce the inference input, re-explains the meaning and cause of the event in natural language or multimodal form, and generates behavioral recommendations; A step of performing an accessibility output control step that provides alternative feedback by applying priorities to voice, text, LED, and vibration outputs according to user profile and environmental conditions; and Includes an audit/explanatory log recording step that ensures traceability by recording reasons, supporting signals, and latency during the generation process of behavioral recommendations and warnings generated in the above Generative Re-inference step. Multi-sensor real-time situation awareness and context analysis method.
In Paragraph 12, A multi-sensor real-time situation awareness and context analysis method characterized by the above-mentioned composite environment sensor including an IMU, a gas sensor, an IAQ sensor, and GNSS.
In Paragraph 12, A multi-sensor real-time situation awareness and context analysis method characterized by the above storage quality control step adaptively adjusting image resolution, frame rate, audio sample rate, and sensor sampling period according to the calculated activity index.
In Paragraph 14, A multi-sensor real-time situation awareness and context analysis method characterized by the above-described storage quality control step storing at low resolution and low frame rate when the activity index is below a threshold, and storing at high resolution and high frame rate when the activity index is above a threshold.
In Paragraph 12, A multi-sensor real-time situation awareness and context analysis method characterized by the step of storing in the above-mentioned rolling buffer preserving data by extending the interval before and after the time of event occurrence.
delete

Description

On-device Generative AI and Adaptive Rolling Buffer-Based Multisensor Real-Time Situation Awareness and Context Analysis Apparatus and Method The present invention time-synchronizes and cyclically stores multi-sensor data in a rolling buffer, adaptively controls storage quality using a rate-of-change activity index and hysteresis levels (L0~L3) to detect a primary situation, receives context and analysis results, performs re-explanation and behavioral recommendations in natural language and multimodal forms, and an accessibility output control unit provides a combination of voice, text, LED, and vibration according to the user (visual/auditory constraints) and environmental conditions (noise/illumination). The system supports the safety of the elderly, the disabled, and workers in poor audiovisual environments, and ensures privacy protection by blocking the external transmission of original data and transmitting only feature-value-based data. With the recent advancements in various sensor and edge computing technologies, it can be observed that active research is underway on systems that collect and analyze video, audio, and environmental data in real time to perceive situations and detect abnormal events. In particular, technology that simultaneously utilizes complex environmental sensors such as cameras, microphones, IMUs, gas sensors, IAQ sensors, and GNSS is gaining attention, and this multi-sensor fusion-based data collection technology enables the integrated identification of user behavior, changes in the surrounding environment, and location information. In addition, the application of technologies that pre-process, filter, and extract features from data at the edge is reducing network latency and enabling real-time performance. In particular, real-time data storage methods based on a rolling buffer structure are widely used in various application fields, such as security surveillance systems, safety management systems, autonomous vehicles, smart homes, and healthcare monitoring, because they have the advantage of efficiently utilizing storage space while rapidly securing data at a specific point in time. Rolling buffers have established themselves as an essential technology because they maintain a specific range of data in a cyclical structure, allowing for the simultaneous acquisition of past and present data, and enable the rapid extraction of the relevant section for analysis upon the occurrence of an event. Furthermore, technologies capable of simultaneously processing multimodality data have recently been developed. These technologies combine video, audio, and sensor signals to enable situational awareness and analysis with much higher accuracy than single-modality data processing methods. Furthermore, advanced systems incorporating context analysis and situation interpretation based on discriminative AI, generative AI, and autonomous agentic AI are evolving beyond simple detection to the ability to interpret complex situations by grasping the context before and after events. Nevertheless, there are various limitations to this technology. Existing rolling buffer-based data management methods are limited to simply circulating data storage; because they maintain the same storage quality even in sections with minimal change, they lead to the problem of unnecessary consumption of storage resources. For example, when storing data at high resolution and high frame rates in quiet environments or video segments with little movement, storage and network bandwidth are used inefficiently. Furthermore, existing systems focus solely on acquiring data at the moment of event occurrence, failing to adequately preserve contextual information before and after the event. Due to this limitation, only fragmentary event detection is possible, imposing constraints on in-depth contextual analysis, such as analyzing the entire progression of an event or its causes. For instance, even if a fall is detected, it is difficult to accurately analyze the cause of the accident if the user's slipping motion or changes in the surrounding environment prior to the event are not recorded. Furthermore, when multiple events occur simultaneously or in close proximity, conflicts or data loss are prone to occur during data interval management. Existing simple buffer management methods fail to preserve data independently for each event; consequently, if multiple events share the same data area, data from previous events may be overwritten and lost. This leads to errors where only some events are recorded during the analysis phase, even though multiple events actually occurred consecutively. Furthermore, the existing method of transmitting video, audio, and sensor data directly to a server significantly increases the risk of leakage of personal and sensitive data. Video data can contain sensitive elements such as faces, license plates, and the interior structure of a home, while audio data may include speaker identification or the content