US-12623675-B1 - Intelligent agentic driver monitoring system for generating and executing a context-aware intervention

Abstract

Disclosed are urban intelligence methods and systems for generating and executing a context-aware intervention. An exemplary method includes: receiving, from one or more multimodal sensors, first sensor data associated with a vehicle; identifying an operator of the vehicle; initiating first analyzing the first sensor data; determining a first context associated with the operator of the vehicle; determining a second context associated with the vehicle; determining, first severity data associated with at least one of the first context or the second context; initiating generating a first intervention based on the first context, the second context, the first severity data, and historical data associated with at least one of: the operator of the vehicle and the vehicle; initiating execution of the first intervention; receiving, from the one or more multimodal sensors, second sensor data; initiating second analyzing, the second sensor data; and determining the first intervention was successful.

Inventors

• Amal Ayadh Alsulamy
• Salem Faiz Alelyani
• Khaled Helmi El-Maleh

Assignees

• Saudi Technology and Security Comprehensive Control Co. Ltd.

Dates

Publication Date

20260512

Application Date

20251125

Claims (20)

1. 1 . A method for utilizing an intelligent driver monitoring system to generate and execute a context-aware intervention, the method comprising: receiving, using one or more computing device processors, from one or more multimodal sensors, at a first time, first sensor data, wherein the one or more multimodal sensors are associated with or comprised in at least one of: a vehicle or an operator of the vehicle; identifying, using the one or more computing device processors, the operator of the vehicle; accessing, using the one or more computing device processors, a first database comprising historical data associated with at least one of: the vehicle or the operator of the vehicle; initiating first analyzing, using the one or more computing device processors, based on first accessing at least one intelligence model, the first sensor data; determining, using the one or more computing device processors, based on the first analyzing, based on the first accessing the at least one intelligence model, the first sensor data, a first context associated with the operator of the vehicle; determining, using the one or more computing device processors, based on the first analyzing, based on the first accessing the at least one intelligence model, the first sensor data, a second context associated with the vehicle; determining, using the one or more computing device processors, based on the first context associated with the operator of the vehicle and the second context associated with the vehicle, first severity data; initiating generating, using the one or more computing device processors, based on second accessing the at least one intelligence model, a first intervention, wherein the first intervention is based on: the first context associated with the operator of the vehicle, the second context associated with the vehicle, the first severity data, and the historical data associated with at least one of: the vehicle or the operator of the vehicle; initiating execution of, using the one or more computing device processors, at a second time after the first time, the first intervention; receiving, using the one or more computing device processors, from the one or more multimodal sensors, at a third time after the first time and the second time, second sensor data; initiating second analyzing, using the one or more computing device processors, based on third accessing the at least one intelligence model, the second sensor data; determining, using the one or more computing device processors, based on the second analyzing, based on the third accessing the at least one intelligence model, the second sensor data, a third context associated with the operator of the vehicle; determining, using the one or more computing device processors, based on the second analyzing, based on the third accessing the at least one intelligence model, the second sensor data, a fourth context associated with the vehicle; determining, using the one or more computing device processors, based on the third context associated with the operator of the vehicle and the fourth context associated with the vehicle, the first intervention was successful; and determining, using the one or more computing device processors, based on the third context associated with the operator of the vehicle and the fourth context associated with the vehicle, feedback associated with the first intervention for fine-tuning the at least one intelligence model.
2. 2 . The method of claim 1 , further comprising: receiving, using the one or more computing device processors, from the operator of the vehicle or a supervisor of the operator of the vehicle, a first authorization or a first restriction associated with generating interventions; and initiating storage of, using the one or more computing device processors, in the first database, the first authorization or the first restriction associated with generating interventions received from the operator of the vehicle or the supervisor of the operator of the vehicle.
3. 3 . The method of claim 1 , wherein the first context associated with the operator of the vehicle comprises or is associated with at least one of: an inattentiveness of the operator of the vehicle, a distraction affecting the operator of the vehicle, a medical ailment affecting the operator of the vehicle, a first activity associated with the operator of the vehicle, a physical state associated with the operator of the vehicle, or a cognitive state associated with the operator of the vehicle.
4. 4 . The method of claim 1 , wherein the second context associated with the vehicle comprises or is associated with at least one of: a first passenger in the vehicle, a first behavior associated with the first passenger in the vehicle, a speed associated with the vehicle, lighting within the vehicle, noise levels within the vehicle, environmental conditions, external conditions, road conditions, weather conditions, a time of day, or at least one additional vehicle in a vicinity of the vehicle.
5. 5 . The method of claim 1 , wherein the first intervention is personalized based on the historical data associated with at least one of: the vehicle or the operator of the vehicle.
6. 6 . The method of claim 1 , further comprising: initiating storage of, using the one or more computing device processors, in the first database comprising the historical data associated with at least one of: the vehicle or the operator of the vehicle, data associated with at least one of: the first intervention, the operator of the vehicle, or the feedback associated with the first intervention.
7. 7 . The method of claim 1 , wherein the one or more computing device processors are comprised in one or more computing systems, wherein the one or more computing systems are located in one or more locations.
8. 8 . One or more systems for utilizing an intelligent driver monitoring system to generate and execute a context-aware intervention, the one or more systems comprising: one or more computing databases; and one or more computing servers comprising one or more computing device processors and a memory storing instructions, the instructions being executable by the one or more computing device processors to: receive, from one or more multimodal sensors, at a first time, first sensor data, wherein the one or more multimodal sensors are associated with or comprised in at least one of: a vehicle or an operator of the vehicle; identify the operator of the vehicle; access a first database comprising historical data associated with at least one of: the vehicle or the operator of the vehicle; initiate first analyzing, using the one or more computing device processors, based on first accessing at least one intelligence model, the first sensor data; determine, based on the first analyzing, based on the first accessing the at least one intelligence model, the first sensor data, a first context associated with the operator of the vehicle; determine, based on the first analyzing, based on the first accessing the at least one intelligence model, the first sensor data, a second context associated with the vehicle; determine, based on the first context associated with the operator of the vehicle and the second context associated with the vehicle, first severity data; initiate generating, based on second accessing the at least one intelligence model, a first intervention, wherein the first intervention is based on: the first context associated with the operator of the vehicle, the second context associated with the vehicle, the first severity data associated with at least one of: the first context associated with the operator of the vehicle or the second context associated with the vehicle, and the historical data associated with at least one of: the vehicle or the operator of the vehicle; initiate execution of, using the one or more computing device processors, at a second time after the first time, the first intervention; receive, from the one or more multimodal sensors, at a third time after the first time and the second time, second sensor data; initiate second analyzing, using the one or more computing device processors, based on third accessing the at least one intelligence model, the second sensor data; determine, based on the second analyzing, based on the third accessing the at least one intelligence model, the second sensor data, a third context associated with the operator of the vehicle; determine, based on the second analyzing, based on the third accessing the at least one intelligence model, the second sensor data, a fourth context associated with the vehicle; determine, based on the third context associated with the operator of the vehicle and the fourth context associated with the vehicle, the first intervention was successful; and determine, based on the third context associated with the operator of the vehicle and the fourth context associated with the vehicle, first feedback associated with the first intervention for fine-tuning the at least one intelligence model.
9. 9 . The one or more systems of claim 8 , wherein the first sensor data comprises at least one of: visual data, auditory data, physiological data, contextual data, activity data, health data, or environmental data.
10. 10 . The one or more systems of claim 8 , wherein the second sensor data comprises at least one of: visual data, auditory data, physiological data, contextual data, activity data, health data, or environmental data.
11. 11 . The one or more systems of claim 8 , wherein the first severity data is associated with at least one of: a state associated with the operator of the vehicle or an indicator based on the first context associated with the operator of the vehicle and the second context associated with the vehicle.
12. 12 . The one or more systems of claim 8 , wherein the historical data comprises at least one of: at least one preference associated with the operator of the vehicle, second feedback previously received from the operator of the vehicle, at least one response previously received from the operator of the vehicle, at least one setting associated with the operator of the vehicle, at least one condition associated with the vehicle, or personal data associated with the operator of the vehicle.
13. 13 . The one or more systems of claim 8 , wherein the one or more systems are located in one or more locations.
14. 14 . A method for utilizing an intelligent driver monitoring system to generate and execute a context-aware intervention, the method comprising: receiving, using one or more computing device processors, from one or more multimodal sensors, at a first time, first sensor data, wherein the one or more multimodal sensors are associated with or comprised in at least one of: a vehicle or an operator of the vehicle; identifying, using the one or more computing device processors, the operator of the vehicle; accessing, using the one or more computing device processors, a first database comprising historical data associated with at least one of: the vehicle or the operator of the vehicle; initiating first analyzing, using the one or more computing device processors, based on first accessing at least one intelligence model, the first sensor data; determining, using the one or more computing device processors, based on the first analyzing, based on the first accessing the at least one intelligence model, the first sensor data, a first context associated with the operator of the vehicle; determining, using the one or more computing device processors, based on the first analyzing, based on the first accessing the at least one intelligence model, the first sensor data, a second context associated with the vehicle; determining, using the one or more computing device processors, based on the first context associated with the operator of the vehicle and the second context associated with the vehicle, first severity data; initiating generating, using the one or more computing device processors, based on second accessing the at least one intelligence model, a first intervention, wherein the first intervention is based on: the first context associated with the operator of the vehicle, the second context associated with the vehicle, the first severity data associated with at least one of: the first context associated with the operator of the vehicle or the second context associated with the vehicle, and the historical data associated with at least one of: the vehicle or the operator of the vehicle; initiating execution of, using the one or more computing device processors, at a second time after the first time, the first intervention; receiving, using the one or more computing device processors, from the one or more multimodal sensors, at a third time after the first time and the second time, second sensor data; initiating second analyzing, using the one or more computing device processors, based on third accessing the at least one intelligence model, the second sensor data; determining, using the one or more computing device processors, based on the second analyzing, based on the third accessing the at least one intelligence model, the second sensor data, a third context associated with the operator of the vehicle; determining, using the one or more computing device processors, based on the second analyzing, based on the third accessing the at least one intelligence model, the second sensor data, a fourth context associated with the vehicle; determining, using the one or more computing device processors, based on the third context associated with the operator of the vehicle and the fourth context associated with the vehicle, second severity data; determining, using the one or more computing device processors, based on the third context associated with the operator of the vehicle, the fourth context associated with the vehicle, and the second severity data, the first intervention was unsuccessful; determining, using the one or more computing device processors, based on the third context associated with the operator of the vehicle and the fourth context associated with the vehicle, feedback associated with the first intervention for fine-tuning the at least one intelligence model; initiating generating, using the one or more computing device processors, based on fourth accessing the at least one intelligence model, a second intervention, wherein the second intervention is based on: the third context associated with the operator of the vehicle, the fourth context associated with the vehicle, the second severity data, the historical data associated with at least one of: the vehicle or the operator of the vehicle, and the feedback associated with the first intervention; and initiating execution of, using the one or more computing device processors, at a fourth time after the first time, the second time, and the third time, the second intervention.
15. 15 . The method of claim 14 , wherein at least one of: the first accessing the at least one intelligence model comprises transmitting the first sensor data to a first computing system associated with the at least one intelligence model, or the third accessing the at least one intelligence model comprises transmitting the second sensor data to the first computing system associated with the at least one intelligence model.
16. 16 . The method of claim 14 , wherein the initiating execution of, using the one or more computing device processors, at the second time after the first time, the first intervention comprises transmitting the first intervention to a first computing system associated with the vehicle.
17. 17 . The method of claim 14 , wherein the second intervention comprises an escalation or a repetition of the first intervention.
18. 18 . The method of claim 14 , wherein the first intervention comprises at least one of: a lighting adjustment, a sound adjustment, an environment adjustment, a visual message, an audio message, a vibration, a haptic alert, an emergency call, a communication with a second vehicle, or seizing control of the vehicle.
19. 19 . The method of claim 14 , wherein the first intervention comprises at least one of: initiating a conversation with the operator of the vehicle, adjusting a volume level in the vehicle, adjusting a temperature setting in the vehicle, utilizing an audio assistant associated with the vehicle, triggering a vibration on a steering wheel or a seat comprised in the vehicle, initiating an emergency call, connecting with a second vehicle or a second operator associated with the second vehicle for assistance, executing a safe-stop protocol, engaging a driver assistance system, a vehicle to vehicle (V2V) intervention, or a vehicle to everything intervention (V2X).
20. 20 . The method of claim 14 , wherein the one or more computing device processors are comprised in one or more computing systems, wherein the one or more computing systems are located in one or more locations.

Description

TECHNICAL FIELD The present methods and systems are directed to urban intelligence solutions for intervening with a disengaged or distracted operator of a vehicle. BACKGROUND There is a need for an advanced system capable of generating and executing an intervention based on analyzing, using at least one intelligence model, sensor data associated with a vehicle. SUMMARY The disclosed systems and methods may leverage urban intelligence technologies configured to utilize sensor data, including multi-sensor and real-time data, from environments like urban areas and roadways. A disclosed method for utilizing an intelligent driver monitoring system to generate and execute a context-aware intervention comprises: receiving, using one or more computing device processors, from one or more multimodal sensors, at a first time, first sensor data, wherein the one or more multimodal sensors are associated with or comprised in at least one of: a vehicle or an operator of the vehicle; identifying, using the one or more computing device processors, the operator of the vehicle; accessing, using the one or more computing device processors, a first database comprising historical data associated with at least one of: the vehicle or the operator of the vehicle; initiating first analyzing, using the one or more computing device processors, based on first accessing at least one intelligence model, the first sensor data; determining, using the one or more computing device processors, based on the first analyzing, based on the first accessing the at least one intelligence model, the first sensor data, a first context associated with the operator of the vehicle; determining, using the one or more computing device processors, based on the first analyzing, based on the first accessing the at least one intelligence model, the first sensor data, a second context associated with the vehicle; determining, using the one or more computing device processors, based on the first context associated with the operator of the vehicle and the second context associated with the vehicle, first severity data; initiating generating, using the one or more computing device processors, based on second accessing the at least one intelligence model, a first intervention, wherein the first intervention is based on: the first context associated with the operator of the vehicle, the second context associated with the vehicle, the first severity data, and the historical data associated with at least one of: the vehicle or the operator of the vehicle; initiating execution of, using the one or more computing device processors, at a second time after the first time, the first intervention; receiving, using the one or more computing device processors, from the one or more multimodal sensors, at a third time after the first time and the second time, second sensor data; initiating second analyzing, using the one or more computing device processors, based on third accessing the at least one intelligence model, the second sensor data; determining, using the one or more computing device processors, based on the second analyzing, based on the third accessing the at least one intelligence model, the second sensor data, a third context associated with the operator of the vehicle; determining, using the one or more computing device processors, based on the second analyzing, based on the third accessing the at least one intelligence model, the second sensor data, a fourth context associated with the vehicle; determining, using the one or more computing device processors, based on the third context associated with the operator of the vehicle and the fourth context associated with the vehicle, the first intervention was successful; and determining, using the one or more computing device processors, based on the third context associated with the operator of the vehicle and the fourth context associated with the vehicle, feedback associated with the first intervention for fine-tuning the at least one intelligence model. In some cases, the method further comprises: receiving, using the one or more computing device processors, from the operator of the vehicle or a supervisor of the operator of the vehicle, a first authorization or a first restriction associated with generating interventions; and initiating storage of, using the one or more computing device processors, in the first database, the first authorization or the first restriction associated with generating interventions received from the operator of the vehicle or the supervisor of the operator of the vehicle. In other cases, the first context associated with the operator of the vehicle comprises or is associated with at least one of: an inattentiveness of the operator of the vehicle, a distraction affecting the operator of the vehicle, a medical ailment affecting the operator of the vehicle, a first activity associated with the operator of the vehicle, a physical state associated with the operator of the vehicle, or a cogn