US-20260127372-A1 - SYSTEM AND METHOD OF BUILDING A CUSTOMIZED SCENARIO FOR A VEHICLE WITH A LARGE LANGUAGE MODEL BASED ON A SERVICE-ORIENTED ARCHITECTURE

US20260127372A1US 20260127372 A1US20260127372 A1US 20260127372A1US-20260127372-A1

Abstract

A system and a method may include a speech recognition device configured to acquire user speech information, convert the acquired user speech information into user demand information in a text form, classify the converted user demand information, determine whether a type of the user demand information is a customized scenario, and, if the type of the user demand information is determined to be the customized scenario, generate user demand information of the customized scenario. The system may further include a service-oriented architecture (SOA) atomic function library configured to provide status information of a sensor and an actuator of the vehicle. The system may further include a control device configured to analyze the user demand information based on large language models (LLMs) and generate a plan for a customized scenario suitable for a user demand by using user demand information of the customized scenario and the SOA atomic function library.

Inventors

Dong Niu
Feng Yang
Ruzhang Huang

Assignees

HYUNDAI MOTOR COMPANY
KIA CORPORATION

Dates

Publication Date: 20260507
Application Date: 20251029
Priority Date: 20241101

Claims (18)

1 . A system specifically configured to build a customized scenario for a vehicle using large language models (LLMs) based on a service-oriented architecture (SOA), the system comprising: a speech recognition device configured to acquire user speech information, convert the acquired user speech information into user demand information in a text form, classify the converted user demand information, determine whether a type of the user demand information is a customized scenario, and, based on determining that the type of the user demand information is the customized scenario, generate user demand information of the customized scenario; an SOA atomic function library stored in a non-transitory memory, the SOA atomic function library configured to provide status information of a sensor and an actuator of the vehicle; and a control device configured to be communicatively connected to the speech recognition device and the SOA atomic function library, and to analyze the user demand information based on the LLMs and generate a plan for the customized scenario suitable for the user demand information by using user demand information of the customized scenario and the SOA atomic function library.
2 . The system of claim 1 , further comprising: a display device communicatively connected to the control device and configured to display the plan for the generated customized scenario to a user.
3 . The system of claim 2 , further comprising: a face recognition device configured to be communicatively connected to the control device, recognize facial information of a user, determine whether the user is a registered user, and transmit identity information of the determined user to the control device; and a seat occupancy detection device communicatively connected to the control device, and configured to determine a seating position of the user and transmit information including the determined seating position of the user to the control device, wherein the control device is further configured to analyze the user demand information based on the LLMs and generate the plan for the customized scenario suitable for the user demand information by using the user demand information of the customized scenario and the SOA atomic function library, based on determining that the user is the registered user or the user is sitting on a driver seat.
4 . The system of claim 2 , wherein the display device further includes: a confirmation button for indicating that the user is satisfied with the plan for the customized scenario generated by a user selection; and a rebuild button for indicating that the user is not satisfied with the plan for the customized scenario generated through the user selection and for regenerating the plan for the customized scenario.
5 . The system of claim 4 , wherein the control device is further configured to: based on the user not being satisfied with the generated plan for the customized scenario, regenerate the plan for the customized scenario based on feedback information of the user until the user is satisfied; and based on the user being satisfied with the generated plan for the customized scenario, store the generated plan for the customized scenario or store the generated plan for the customized scenario and the corresponding identity information of the user or seating position information.
6 . The system of claim 5 , wherein the control device is further configured to execute the plan for the customized scenario stored based on a condition of the plan for the customized scenario stored being satisfied.
7 . The system of claim 1 , wherein the LLMs are trained and prompt-tuned using predetermined customized scenario information, and wherein the LLMs are installed in the control device within the vehicle or in a cloud.
8 . The system of claim 1 , wherein the speech recognition device further includes: a microphone configured to capture a speech signal of the user; an automatic speech recognition (ASR) module configured to convert the speech signal of the user into text information; a natural language processing (NLP) module configured to process and analyze the text information to understand a structure and meaning of a sentence; and a natural language understanding (NLU) module configured to understand the text information and make a corresponding decision.
9 . The system of claim 3 , wherein the face recognition device includes: a camera configured to acquire a facial image of the user; and a processor configured to convert the acquired facial image into the facial information, extract a feature of the facial information, and compare the extracted feature with a facial feature stored in a database to determine whether the user is the registered user.
10 . A method for building a customized scenario for a vehicle using SOA-based LLMs, the method comprising: obtaining, by a speech recognition device, user speech information; converting, by the speech recognition device, the acquired user speech information into user demand information in a text form; classifying, by the speech recognition device, a type of the converted user demand information; determining, by the speech recognition device, whether the type of the user demand information is a customized scenario; generating, by the speech recognition device, user demand information of the customized scenario based on determining that the type of the user demand information is the customized scenario; and providing, by the speech recognition device, status information of a sensor and an actuator of the vehicle via an SOA atomic function library; analyzing, by a control device, the user demand information based on the LLMs using the user demand information of the customized scenario and the atomic function library; and generating, by the control device, a plan for the customized scenario suited to the user demand information.
11 . The method of claim 10 , further comprising: displaying to the user the plan for the customized scenario generated by a display device.
12 . The method of claim 11 , further comprising: recognizing, by a face recognition device, facial information of the user; determining, by the face recognition device, whether the user is a registered user; transmitting, by the face recognition device, identity information of the determined user to the control device; determining, by a seat occupancy detection device, a seating position of the user; transmitting, by the seat occupancy detection device, seating position information of the determined user to the control device; analyzing, by the control device, based on determining that the user is the registered user or that the user is sitting on a driver seat, the user demand information based on the LLMs by using the user demand information of the customized scenario and the SOA atomic function library; and generating, by the control device, the plan for the customized scenario suited to the user demand information.
13 . The method of claim 11 , further comprising: receiving, by the control device, an indication of satisfaction from the user regarding the generated plan for the customized scenario by selecting a confirmation button on the display device; indicating, by the control device, that the user is not satisfied with the plan for the customized scenario generated by selecting a rebuild button on the display device; and regenerating, by the control device, the plan for the customized scenario.
14 . The method of claim 13 , further comprising: based on the user not being satisfied with the generated plan for the customized scenario, regenerating, by the control device, the plan for the customized scenario based on feedback information of the user until the user is satisfied; and based on the user being satisfied with the generated plan for the customized scenario, storing, by the control device, the generated plan for the customized scenario, or storing the generated plan for the customized scenario and the corresponding identity information of the user or seating position information.
15 . The method of claim 14 , wherein the plan for the customized scenario stored is executed when a condition of the plan for the stored customized scenario is satisfied.
16 . The method of claim 11 , further comprising: training and prompt-tuning the LLMs using predetermined customized scenario information, wherein the LLMs are installed in the control device within the vehicle or in a cloud.
17 . The method of claim 10 , further comprising: capturing, by a microphone, a speech signal of the user; converting, by an automatic speech recognition (ARS) module, the speech signal of the user into text information; processing and analyzing, by a natural language processing (NLP) module, the text information to understand a structure and meaning of a sentence; and understanding, by a natural language understanding (NLU) module, the text information and making a corresponding decision.
18 . The method of claim 12 , further comprising: acquiring, by a camera, a facial image of the user; and converting, by a processor, the acquired facial image into the facial information; extracting, by the processor, a feature of the facial information; and comparing, by the processor, the extracted feature with a facial feature stored in a database to determine whether the user is the registered user.

Description

CROSS-REFERENCE TO RELATED APPLICATION This application claims priority to and the benefit of Chinese Patent Application No. 202411554080.5 filed on Nov. 1, 2024, the entire contents of which are incorporated herein by reference. BACKGROUND (a) Technical Field The present disclosure relates to a smart vehicle and software architecture, and more particularly, to a system and method for building a customized scenario for a vehicle using large language models (LLMs) based on a service-oriented architecture (SOA). (b) Description of the Related Art The content described in this section merely provides background information related to the present disclosure and does not constitute prior art. Currently, vehicle manufacturers are developing smart scenario applications that run on a service-oriented architecture (SOA), and that allow a user to define various functions using sensors and actuators of the vehicle. SOA is a design architecture that composes, i.e., structures or organizes, an application into a variety of independent services that are updatable and extendable independently without affecting the entire system. In this situation, various functions or processes (e.g., driving assistance, entertainment systems, navigation, and the like) of the vehicle may be implemented as independent “atomic” services, i.e., self-contained, reusable components, or specifically configured instances of containerized processes, i.e., containers, which may be combined and customized based on defined parameters such as user demands, i.e., requests, requirements, demands, and the like of users. However, this customization process may lead to some problems. Users must have a certain level of technical knowledge to understand and operate this complex system. Further, manual editing and customization are difficult and time-consuming. SUMMARY The present disclosure provides a system, e.g., a computing system, and method for understanding a user demand using large language models (LLMs) and generating corresponding output based on defined parameters such as data indicative of user demands or requests to control a vehicle. In particular, the disclosed embodiments provide a system and method that control the vehicle to execute different functions, e.g., causing a change on the driving mode of the vehicle, adjusting a seat temperature, changing a music playlist, and the like, in response to receiving simple language commands indicative of requests or demands received from a user. The disclosed embodiments may not only improve user experience, but also enhance safety and efficiency of the vehicle. The disclosed systems and methods accurately understand user demand and eliminate difficulty when setting up customized scenarios by a user. According to an embodiment of the present disclosure, a system is specifically configured to build a customized scenario for a vehicle using large language models (LLMs) based on a service-oriented architecture (SOA). The system may include a speech recognition device configured to acquire user speech information, convert the acquired user speech information into user demand information in a text form, classify the converted user demand information, determine whether a type of the user demand information is a customized scenario, and, based on determining that the type of the user demand information is the customized scenario, generate user demand information of the customized scenario. The system may further include an SOA atomic function library configured to provide status information of a sensor and an actuator of the vehicle. The system may further include a control device configured to be communicatively connected to the speech recognition device and the SOA atomic function library. The control device may be further configured to analyze the user demand information based on the LLMs and generate a plan for the customized scenario suitable for the user demand information by using user demand information of the customized scenario and the SOA atomic function library. The system may further include a display device communicatively connected to the control device and configured to display the plan for the generated customized scenario to a user. The system may further include a face recognition device configured to be communicatively connected to the control device, recognize facial information of a user, determine whether the user is a registered user, and transmit identity information of the determined user to the control device. The system may further include a seat occupancy detection device communicatively connected to the control device. The seat occupancy detection device may be configured to determine a seating position of the user and transmit information including the determined seating position of the user to the control device. The control device may be further configured to analyze the user demand information based on the LLMs and generate the plan for the customized scenario suitable for t