US-12617361-B2 - Apparatus for estimating behavior of vehicle occupant and method for the same

US12617361B2US 12617361 B2US12617361 B2US 12617361B2US-12617361-B2

Abstract

To robustly estimate three-dimensional behaviors of an occupant by fusing, through a particle filter, information obtained through vehicle indoor cameras and through vehicle internal information sensors, an occupant behavior estimation system includes: a camera configured to obtain images of the at least one occupant within the vehicle; sensors configured to obtain information on the vehicle; an image processing device configured to process images obtained from the camera and to obtain key point information of the at least one occupant and object tracking information that is provided by tracking the at least one occupant; and a vehicle safety controller configured to estimate the behaviors of the occupant by using a particle filter based on the information on the vehicle obtained through the sensors, the key point information and the object tracking information, which are obtained from the image processing device.

Inventors

Joon Sang Park

Assignees

HYUNDAI MOTOR COMPANY
KIA CORPORATION

Dates

Publication Date: 20260505
Application Date: 20230315
Priority Date: 20220315

Claims (20)

1 . An occupant behavior estimation system that estimates behaviors of at least one occupant within a vehicle, the occupant behavior estimation system comprising: a camera configured to obtain images of the at least one occupant within the vehicle; sensors configured to obtain information on the vehicle; an image processing device configured to process the images obtained from the camera and to obtain key point information of the at least one occupant and object tracking information that is provided by tracking the at least one occupant; and a vehicle safety controller configured to estimate the behaviors of each of the at least one occupant by using a particle filter based on the information on the vehicle obtained through the sensors, the key point information and the object tracking information comprising information on the at least one occupant, which are obtained from the image processing device, and to control the vehicle based on the estimated behaviors of each of the at least one occupant, wherein the vehicle safety controller is further configured to: determine whether a new occupant is detected based on an existing occupant tracked by the particle filter, and create a particle filter corresponding to the new occupant in response to determining that the new occupant is detected.
2 . The occupant behavior estimation system of claim 1 , wherein the sensors comprise at least one of a seat information sensor or a dynamic sensor, wherein the seat information sensor Is configured to provide occupant sitting information, a seat position, a seat incline angle, and a seat swivel angle, and wherein the dynamic sensor is configured to provide information on an acceleration and an angular acceleration of the vehicle.
3 . The occupant behavior estimation system of claim 2 , wherein the key point information obtained by the image processing device comprises two-dimensional position information of a key point which indicates a two-dimensional position of the key point on an image plane.
4 . The occupant behavior estimation system of claim 3 , wherein the key point information obtained by the image processing device further comprises three-dimensional position information of the key point which indicates a three-dimensional position of the key point on an image plane.
5 . The occupant behavior estimation system of claim 4 , wherein the image processing device obtains the key point information by using an artificial intelligence system which has learned in advance or using a computer vision technique.
6 . The occupant behavior estimation system of claim 4 , wherein the vehicle safety controller comprises a control unit and a particle filter unit, wherein the control unit is configured to: generate information on an object being tracked which comprises information on the existing occupant being tracked by using the particle filter, determine whether detection of the existing occupant fails and whether the new occupant is detected, based on the information on the object being tracked and the object tracking information obtained from the image processing device, instruct, in response to determining that the new occupant is detected, the particle filter unit to create the particle filter corresponding to the new occupant, and instruct, in response to determining that the detection of the existing occupant fails, the particle filter unit to remove a particle filter corresponding to the existing occupant, and wherein the particle filter unit is configured to create, update, or remove a particle filter based on an instruction of the control unit.
7 . The occupant behavior estimation system of claim 6 , wherein the particle filter unit creates the particle filter based on the instruction of the control unit, and then updates the corresponding particle filter, and wherein updating the particle filter comprises propagating particles of the particle filter, updating weightings of the particles, calculating a behavior estimation value based on the weightings, and resampling the particles of the particle filter when a specific condition is satisfied.
8 . The occupant behavior estimation system of claim 7 , wherein the particle filter unit propagates the particles of the particle filter for each occupant based on the information on the acceleration and the angular acceleration of the vehicle obtained from the dynamic sensor and a dynamics model of the occupant.
9 . The occupant behavior estimation system of claim 8 , wherein the particle filter unit propagates the particles of the particle filter for each occupant by additionally using the dynamics model of the occupant only when the acceleration or the angular acceleration of the vehicle obtained from the dynamic sensor is greater than a second predetermined threshold value.
10 . The occupant behavior estimation system of claim 7 , wherein the particle filter unit updates weightings of particles of the particle filter based on at least one of prior probability information, the two-dimensional position information of the key point of the existing occupant, the three-dimensional position information of the key point, or seat information obtained from the seat information sensor.
11 . The occupant behavior estimation system of claim 10 , wherein the updating the weightings of the particles based on the prior probability information comprises: updating the weightings of the particles which indicate the key point corresponding to a physically impossible posture to 0, or updating the weightings of the particles of which a position is out of a vehicle interior to 0, or having range information corresponding to the vehicle interior or photographing range information according to a field of view of the camera, and when three-dimensional coordinates of the key point, which correspond to the particles, are out of a range of the range information or the photographing range information, reducing the weightings of the particles or updating the weightings of the particles to 0, or setting in advance a condition for information on a distance between key points by reflecting data on a statistical body size of a human body, and when the three-dimensional coordinates of the key point, which correspond to the particles, do not satisfy the set condition, reducing the weightings of the corresponding particles or updating the weightings of the corresponding particles to 0.
12 . The occupant behavior estimation system of claim 10 , wherein the updating the weightings of the particles based on the two-dimensional position information of the key point or the three-dimensional position information of the key point comprises calculating a likelihood probability of the particles based on the two-dimensional position information of the key point or the three-dimensional position information of the key point, which corresponds to the particles, and updating particle weighting information based on the calculated likelihood probability.
13 . The occupant behavior estimation system of claim 10 , wherein the updating the weightings of particles based on at least one of the seat information comprises comparing coordinates of first key point of occupant who has sat obtained from the seat information with coordinates of second key point calculated from the particles, and then updating the weightings in inverse proportion to a distance between the coordinates of the first and the second key points.
14 . The occupant behavior estimation system of claim 7 , wherein the vehicle safety controller estimates a position of the key point of the existing occupant as an average value that reflects a weighting of a position of the key point corresponding to the particles, or estimates the position of the key point of the existing occupant as a position of the key point corresponding to a particle having a maximum weighting.
15 . The occupant behavior estimation system of claim 7 , wherein the vehicle safety controller performs particle resampling when the number of effective particles of the particle filter is smaller than a third predetermined threshold value, or performs the particle resampling when the particles of which the number is greater than or equal to a fourth predetermined threshold value are located at a position out of a vehicle interior.
16 . The occupant behavior estimation system of claim 6 , wherein the control unit performs matching such that a sum of an intersection over union (IoU) between a bounding box of the existing occupant being tracked and a bounding box of the detected at least one occupant is maximized, and determines whether the detection of the existing occupant fails and whether the new occupant is detected, based on determining whether a value of the IoU for each occupant is greater than a predetermined threshold value.
17 . The occupant behavior estimation system of claim 6 , wherein the control unit determines whether the new occupant is in a sitting state, wherein the particle filter unit creates the particle filter for the new occupant, initializes particles of the particle filter for the new occupant based on seat information obtained from the seat information sensor in response to determining that the new occupant is in the sitting state, and initializes the particles of the particle filter by using the three-dimensional position information of the new occupant in response to determining that the new occupant is not in the sitting state, that is, in an out of position (OOP) state.
18 . The occupant behavior estimation system of claim 17 , wherein the control unit determines whether the new occupant is in the sitting state, based on the occupant sitting information obtained from the seat information sensor, or obtains a bounding box of a seat based on the seat information obtained from the seat information sensor and determines the new occupant as being in the sitting state when an intersection over union (IoU) between the bounding box of the seat and a bounding box of the new occupant is greater than or equal to a first predetermined threshold value.
19 . The occupant behavior estimation system of claim 6 , wherein, in response to determining that the detection of the existing occupant fails, the control unit records the number of detection failures of the existing occupant, and in response to determining that the number of detection failures is greater than a fifth predetermined threshold value, the control unit removes the particle filter corresponding to the corresponding existing occupant.
20 . An occupant behavior estimation method of an occupant behavior estimation system, which estimates behaviors of each of at least one occupant within a vehicle, the occupant behavior estimation method comprising: obtaining key point information from an image obtained through a camera; detecting at least one occupant based on the key point information, and obtaining object tracking information comprising information on the detected at least one occupant; estimating the behaviors of each of the at least one occupant by using a particle filter based on information on the vehicle obtained through sensors, the key point information, and the object tracking information; controlling the vehicle based on the estimated behaviors of each of the at least one occupant; determining whether a new occupant is detected based on an existing occupant tracked by the particle filter; and creating a particle filter corresponding to the new occupant in response to determining that the new occupant is detected.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS The present application claims the benefit of priority to Korea Patent Application No. 10-2022-0032278, filed on Mar. 15, 2022 in the Korean Intellectual Property Office, the entire contents of which is incorporated herein for all purposes by reference. FIELD Various embodiments relate to an apparatus for estimating vehicle occupant behaviors and a method for the same and more particularly to an apparatus for robustly estimating three-dimensional behaviors of an occupant by fusing, through a particle filter, information obtained through vehicle indoor cameras and through vehicle internal information sensors, and a method for the same. BACKGROUND An autonomous vehicle refers to a vehicle capable of traveling on its own accord without an operation of a driver. The society of automotive Engineers divides the development of an autonomous driving technology into six stages. In the initial stage, a person monitors the driving environment, and the autonomous driving technology performs only a function of assisting the person to drive, such as steering assistance, acceleration/deceleration assistance. However, in the final stage, full automation without human intervention is provided, so that an autonomous driving system is responsible for driving a vehicle while monitoring all road conditions and environments. In such an autonomous vehicle, the autonomous driving system indispensably obtains information on the interior and exterior of the vehicle. Also, in order to secure the safety of the occupant, it is necessary to obtain the three-dimensional behaviors of the occupant. SUMMARY The purpose of the present disclosure is to provide a method for robustly estimating three-dimensional behaviors of an occupant by utilizing a non-contact sensor such as a camera, and an apparatus for the same. Also, the purpose of the present disclosure is to provide a method for robustly estimating three-dimensional behaviors of an occupant by fusing, through a particle filter, information obtained through a vehicle interior camera and a vehicle interior information sensor, and an apparatus for the same. The technical problem to be overcome in this document is not limited to the above-mentioned technical problems. Other technical problems not mentioned can be clearly understood from those described below by a person having ordinary skill in the art. One embodiment of the present disclosure is an occupant behavior estimation system that estimates behaviors of an occupant within a vehicle. The occupant behavior estimation system may include: a camera configured to obtain images of the occupant within the vehicle; sensors configured to obtain information on the vehicle; an image processing device configured to process images obtained from the camera and to obtain key point information of the occupant; and a vehicle safety controller configured to estimate the behaviors of the occupant by using a particle filter based on the information on the vehicle obtained through the sensors and the key point information obtained from the image processing device. Another embodiment of the present disclosure is an occupant behavior estimation system that estimates behaviors of at least one occupant within a vehicle. The occupant behavior estimation system may include: a camera configured to obtain images of the at least one occupant within the vehicle; sensors configured to obtain information on the vehicle; an image processing device configured to process images obtained from the camera and to obtain key point information of the at least one occupant and object tracking information that is provided by tracking the at least one occupant; and a vehicle safety controller configured to estimate the behaviors of the occupant by using a particle filter based on the information on the vehicle obtained through the sensors, the key point information and the object tracking information, which are obtained from the image processing device. Further another embodiment of the present disclosure is an occupant behavior estimation method of an occupant behavior estimation system. The occupant behavior estimation method may include: obtaining key point information from an image obtained through a camera; detecting at least one occupant based on the key point information, and obtaining object tracking information including information on the detected at least one occupant; and determining, based on the detected at least one occupant, whether detection of the existing occupant fails, whether an occupant corresponding to the existing occupant among the detected at least one occupant is present, and whether a new occupant is present; creating, in response to determining that a new occupant is present as a result of the determination, a first particle filter corresponding to the corresponding new occupant and initializing particles of the first particle filter; propagating, in response to determining that an occupant corresponding to