Search

US-12619951-B2 - System and method for finding occluded cargo

US12619951B2US 12619951 B2US12619951 B2US 12619951B2US-12619951-B2

Abstract

The disclosure generally describes a system for monitoring a storage compartment of a vehicle comprising a first sensor configured to detect a loading or unloading event, a second sensor configured to capture data regarding one or more items within the storage compartment of the vehicle, wherein the second sensor captures video data of the one or more items within the storage compartment of the vehicle in response to the first sensor detecting a loading or unloading event, and a virtual imaging system configured to classify each item in the storage compartment, generate a virtual replication of each item within the storage compartment, and transmit the virtual replication to a user device.

Inventors

  • Hiroshi Yasuda
  • Manuel Ludwig Kuehner

Assignees

  • Toyota Research Institute, Inc.

Dates

Publication Date
20260505
Application Date
20220421

Claims (20)

  1. 1 . An item management system for inventorying a storage compartment of a vehicle comprising: one or more processors; and a memory having computer readable instructions stored thereon, which when executed by the one or more processors, cause the one or more processors to: capture, by a camera, image data of a storage compartment of the vehicle, the image data comprising information regarding one or more items within the storage compartment of the vehicle; generate metadata comprising a timestamp of a time when the image data of one or more items within a storage compartment of the vehicle is captured, wherein the timestamp is indicative of when an item of the one or more items is loaded or unloaded from the storage compartment; create a three dimensional virtual replication of the storage compartment of the vehicle from the captured image data for a plurality of loading or unloading events, wherein the three dimensional virtual replication of the storage compartment includes a virtual replication of one or more occluded items in the storage compartment, wherein the virtual replication of the one or more occluded items comprises an image replication of the one or more occluded items overlaid on an image of one or more items in the storage compartment that are occluding the replicated one or more occluded items; create a history of when the one or more items are loaded into or unloaded from the storage compartment as a sequence of image data for the plurality of loading or unloading events arranged according to a scrollable timeline display of the timestamps for the plurality of loading or unloading events to identify, based on the loading or unloading events, the one or more occluded items in the storage compartment; and transmit the three dimensional virtual replication of the storage compartment and the history of when the one or more items are loaded into or unloaded from the storage compartment, based on the captured image data, to a viewing device, wherein the viewing device is configured to display the history to allow a user to scroll through the image data for the plurality of loading or unloading events to locate at least one of the one or more occluded items.
  2. 2 . The system of claim 1 , wherein the viewing device includes a virtual reality device configured to allow a viewer to search through a virtual replication of each of the one or more items in the storage compartment to determine the location of the one or more occluded items.
  3. 3 . The system of claim 1 , wherein the memory includes instructions that when executed further cause the one or more processors to: generate a list of each of the one or more items within the storage compartment of the vehicle during a first time period, wherein the list includes a unique identifier for each of the one or more items within the storage compartment; and transmit the list of each of the one or more items within the storage compartment to the viewing device.
  4. 4 . The system of claim 1 , wherein image data is captured during a plurality of time periods, each time period includes a loading or unloading event of the plurality of loading or unloading events.
  5. 5 . The system of claim 4 , wherein each time period starts when a door of the storage compartment is opened and stops when the door is closed.
  6. 6 . An item management system for monitoring a storage compartment of a vehicle comprising: one or more processors; and a memory having computer readable instructions stored thereon, which when executed by the one or more processors, cause the one or more processors to: determine a plurality of time periods for a plurality loading or unloading events, wherein each time duration starts responsive to detecting that a storage compartment door is opened; for each time period, responsive to determining that the storage compartment door is opened: capture, by a camera, image data of a storage compartment of the vehicle, the image data comprising information regarding one or more items within the storage compartment of the vehicle; generate metadata for each captured image, the metadata comprising a timestamp of when the image data of one or more items within a storage compartment of the vehicle is captured during a respective time period, wherein the timestamp is indicative of when an item of the one or more items is loaded or unloaded from the storage compartment; generate a representation of the storage compartment, using the captured image data for the plurality of time periods, wherein the representation includes one or more occluded items within the storage compartment, wherein the representation of the one or more occluded items comprises an image replication of the one or more occluded items overlaid on an image of one or more items in the storage compartment that are occluding the replicated one or more occluded items; create a history of when the one or more items are loaded into or unloaded from the storage compartment as a sequence of image data for the plurality of loading or unloading events arranged according to a scrollable timeline display of the timestamps for the plurality of time periods to identify, based on loading or unloading events, at least one of the one or more occluded items; and transmit, to a viewing device, the representation of the storage compartment and the history of when the one or more items are loaded into or unloaded from the storage compartment.
  7. 7 . The system of claim 6 , wherein the representation includes a three dimensional virtual replication configured to allow a user to search through the three dimensional virtual replication to locate the one or more occluded items.
  8. 8 . The system of claim 6 , wherein the memory includes instructions that when executed further cause the one or more processors to: assign, based on the captured image data, a unique identifier to each of the one or more items in the storage compartment, during each respective time period; generate a list of each of the one or more items in the storage compartment of the vehicle during each respective period, wherein the list includes the unique identifier for each of the one or more items in the storage compartment; and transmit the list of each of the one or more items to the viewing device.
  9. 9 . The system of claim 6 , wherein each time period stops when the storage compartment door is closed.
  10. 10 . The system of claim 6 , wherein the viewing device includes a virtual reality (VR) device.
  11. 11 . The system of claim 6 , wherein the storage compartment of the vehicle is a trunk of the vehicle.
  12. 12 . A method of monitoring a storage compartment of a vehicle, comprising: capture, by a camera, video data of a storage compartment of the vehicle, the video data comprising information regarding one or more items within the storage compartment of the vehicle; generate metadata comprising a timestamp of a time when the video data is received, wherein the timestamp is indicative of when an item of the one or more items is loaded or unloaded from the storage compartment; creating a three dimensional virtual replication of the storage compartment based on the captured video data of one or more items within a storage compartment of the vehicle for a plurality of loading or unloading events, wherein the three dimensional virtual replication of the storage compartment includes a virtual replication of one or more occluded items, wherein the virtual replication of the one or more occluded items comprises an image replication of the one or more occluded items overlaid on an image of one or more items in the storage compartment that are occluding the replicated one or more occluded items; generating a history of when the one or more items are loaded into or unloaded from the storage compartment as a sequence of received video data for the plurality of loading or unloading events arranged according to a scrollable timeline display of the timestamps for the plurality of loading or unloading events to identify, based on the loading or unloading events, at least one of the one or more occluded items; and transmitting, to a viewing device, the three dimensional virtual replication of the storage compartment and the history of when the one or more items are loaded into or unloaded from the storage compartment.
  13. 13 . The method of claim 12 , further comprising: assigning, based on the received video data, a unique identifier to each of the one or more items in the storage compartment; generating a list of each of the one or more items in the storage compartment of the vehicle; and transmitting the list of each of the one or more items to the viewing device, wherein the list includes the unique identifier to each of the one or more items.
  14. 14 . The method of claim 12 , wherein the viewing device includes a virtual reality device.
  15. 15 . The method of claim 14 , wherein the virtual reality device is configured to allow a viewer to search through the three dimensional virtual replication of each item in the storage compartment to determine a location of the one or more occluded items.
  16. 16 . The method of claim 12 , wherein the video data for each loading or unloading event is received during a respective time duration by one or more cameras, wherein the respective time duration starts when a door of the storage compartment is opened and stops when the door is closed.
  17. 17 . The system of claim 1 , wherein the memory includes instructions that when executed further cause the one or more processors to: for each loading or unloading event, stop capturing image data from the camera responsive to a determination that a door of the storage compartment is closed.
  18. 18 . The system of claim 1 , wherein the memory includes instructions that when executed further cause the one or more processors to: detect, by a computer vision processor, a type of item for each of one or more items within the storage compartment of the vehicle from the captured image data; and assign a unique identifier to each of the one or more items in the storage compartment based on the detected type of item, wherein each of the unique identifiers is overlaid over a respective item of the one or more items in the three dimensional virtual replication when displayed by the viewing device.
  19. 19 . The system of claim 6 , wherein the memory includes instructions that when executed further cause the one or more processors to: detect, by a classifier, a type of item for each of one or more items within the storage compartment of the vehicle during each respective time period; and assign a unique identifier to each of the one or more items in the storage compartment based on the detected type of item, wherein each of the unique identifiers is overlaid over a respective item of the one or more items in the representation of the storage compartment when displayed by the viewing device.
  20. 20 . The method of claim 12 , further comprising: for each loading or unloading event, stopping the camera from capturing image data responsive to a determination that a door of the storage compartment is closed.

Description

TECHNICAL FIELD The present disclosure relates generally to a system and method for monitoring storage compartments, and in particular, some implementations relate to assisting a user in finding the location of items in a storage compartment of a vehicle using one or more sensors. DESCRIPTION OF RELATED ART Typically, when a person loads various items into a storage compartment of a vehicle, he or she must remember what items he or she loaded, and where each item is placed within the storage compartment of the vehicle. However, sometimes people forget which items are stored in the vehicle's storage compartment and where they are stored. This is especially true, when items located in the storage compartment are occluded or blocked by other items, making it difficult to find the blocked items. Therefore there is a need in the art for a system and method for inventorying loaded items, and assisting a person in finding occluded items in the storage compartment of a vehicle. BRIEF SUMMARY OF THE DISCLOSURE The according to various embodiments of the disclosed technology the item management system for inventorying a storage compartment of a vehicle includes a processor, and a memory having computer readable instructions stored thereon, which when executed by the processor, causes the processor to: capture data from a sensor, the data comprising information regarding one or more items within the storage compartment of the vehicle; create a three dimensional virtual replication of the storage compartment of the vehicle from the captured data, wherein the three dimensional virtual replication of the storage compartment includes a virtual replication of one or more occluded items in the storage compartment; and transmit the three dimensional virtual replication of the storage compartment, based on the captured data, to a viewing device. In another embodiment, the system includes a first sensor configured to detect a loading or unloading event, and a second sensor configured to capture data relating to one or more items with the storage compartment of the vehicle, wherein the second sensor captures data of the one or more items in response to the first sensor detecting a loading or unloading event, and transmit the captured data to a virtual imaging system, wherein the virtual imaging system is configured to create a first virtual replication of the storage compartment of the vehicle from the captured data, and transmit a second virtual replication comprising the first virtual replication and the one or more unique identifiers to a viewing device. The disclosed technology further includes a method of monitoring a storage compartment of a vehicle. The method includes receiving video data of one or more items in the storage compartment, wherein the video data is captured during a first time duration by one or more cameras, wherein the first time duration starts when a storage compartment door is opened and stops when the storage compartment door is closed, creating a first virtual replication of the storage compartment based on the captured data, wherein the first virtual storage compartment is a virtual rendering of the physical storage area of the vehicle, classifying each item in the storage compartment, wherein classifying each item includes creating one or more unique identification labels for each item in the storage area and transmitting a second virtual replication that includes the first virtual replication and the one or more unique identification labels to a viewing device. Other features and aspects of the disclosed technology will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the features in accordance with embodiments of the disclosed technology. The summary is not intended to limit the scope of any inventions described herein, which are defined solely by the claims attached hereto. BRIEF DESCRIPTION OF THE DRAWINGS The present disclosure, in accordance with one or more various embodiments, is described in detail with reference to the following figures. The figures are provided for purposes of illustration only and merely depict typical or example embodiments. FIG. 1 illustrates an example of a vehicle with which embodiments of the systems and methods disclosed herein may be implemented. FIG. 2 illustrates an example architecture for detecting the location of occluded items stored within the storage compartment, according to some embodiments. FIG. 3 illustrates an example of an occluded item in a storage compartment, according to some embodiments. FIG. 4 illustrates an example of a method of detecting the location of occluded items, according to some embodiments. FIG. 5 illustrates an example computing component that may be used to implement various features of embodiments described in the present disclosure, according to some embodiments. FIGS. 6A-6B illustrate examples of a user device that may be used to