Search

US-20260123809-A1 - OCCUPANCY MAP SEGMENTATION FOR AUTONOMOUS GUIDED PLATFORM WITH DEEP LEARNING

US20260123809A1US 20260123809 A1US20260123809 A1US 20260123809A1US-20260123809-A1

Abstract

The technology disclosed includes systems and methods for preparing a segmented occupancy grid map based upon image information of an environment in which a robot moves. The image information is captured by at least one visual spectrum-capable camera and at least one depth measuring camera. The system includes logic to receive image information captured by at least one visual spectrum-capable camera and location information captured by at least one depth measuring camera located on a mobile platform. The system includes logic to extract from the image information, features in the environment. The system includes logic to determine a 3D point cloud of points having 3D information. The system includes logic to determine, from the 3D point cloud, an occupancy map of the environment. The system includes logic to segment the occupancy map into a segmented occupancy map of regions that represent rooms and corridors in the environment.

Inventors

  • Zhe Zhang
  • Zhongwei Li
  • Peizhang CHEN
  • Rui Xiang
  • Xu Han

Assignees

  • TRIFO, INC.

Dates

Publication Date
20260507
Application Date
20251229
Priority Date
20211227

Claims (20)

  1. 1 . A method for preparing a segmented occupancy grid map based upon image information of an environment, comprising: receiving image information; determining, by a processor, a 3D point cloud of points including location information from the image information, wherein at least some of the points correspond to one or more features extracted from the image information; determining, by a processor, from a 3D point cloud of points, an occupancy map of the environment; and segmenting, by a processor, the occupancy map into a segmented occupancy map of regions that represent rooms and corridors in the environment, including: determining for non-zero pixels, a distance to a zero pixel, and binning each distance; determining for binned distances, a blob having distances meeting a threshold size; and organizing blobs into regions fitting boundaries of the occupancy map, yielding the segmented occupancy map.
  2. 2 . A method of preparing a segmented occupancy grid map based at least in part upon an occupancy map determined based, at least in part, upon image information of an environment, including: reducing noise in an occupancy map; classifying voxels as (i) free, (ii) occupied, or (iii) unexplored; removing ray areas; removing any obstacles within rooms and any obstacles attached to boundaries; computing for each non-zero pixel, a distance to a closest zero pixel; finding candidate seeds based, at least in part, upon binarizing the distance with a threshold change and finding blobs with a blob size meeting a threshold blob size; dilating the blobs; and removing any noise blobs; watersheding the blobs until one or more boundaries are encountered; merging one or more rooms together; and aligning the occupancy map.
  3. 3 . The method of claim 2 , wherein a voxel as classified further includes a label from a neural network classifier, implementing 3D semantic analysis.
  4. 4 . The method of claim 2 , wherein classifying voxels further includes: setting a binary threshold to find free and occupied voxels; filling holes according to surrounding voxels including: if there are more free points around any voids, the voids will become free; otherwise, smaller voids will become occupied, and larger voids will remain unexplored; and based, at least in part, upon sensory information, repairing defects.
  5. 5 . The method of claim 2 , wherein removing ray areas further includes: finding one or more free edges in the occupancy map; and drawing a line between at least two voxels in nearby edges, if the line is not blocked by, or occupied by, a voxel or a sensor voxel.
  6. 6 . The method of claim 3 , wherein the neural network classifier implements one or more convolutional neural networks (CNN).
  7. 7 . The method of claim 3 , further including employing a trained neural network classifier implementing one or more recursive neural networks (RNN).
  8. 8 . The method of claim 3 , further including employing a trained neural network classifier implementing long short-term memory networks (LSTM) for time-based information.
  9. 9 . The method of claim 3 , wherein the neural network classifier includes 80 levels, from an input to an output.
  10. 10 . The method of claim 3 , wherein the neural network classifier implements a multi-layer convolutional network.
  11. 11 . The method of claim 10 , wherein the multi-layer convolutional network includes 60 convolutional levels.
  12. 12 . The method of claim 3 , wherein the neural network classifier includes: a normal convolutional level and a depth-wise convolutional level.
  13. 13 . A system comprising: one or more processors coupled to a memory storing instructions; which computer instructions, when executed on the one or more processors, implement operations comprising: reducing noise in an occupancy map; classifying voxels as (i) free, (ii) occupied, or (iii) unexplored; removing ray areas; removing any obstacles within rooms and any obstacles attached to boundaries; computing for each non-zero pixel, a distance to a closest zero pixel; finding candidate seeds based, at least in part, upon binarizing the distance with a threshold change and finding blobs with a blob size meeting a threshold blob size; dilating the blobs; and removing any noise blobs; watersheding the blobs until one or more boundaries are encountered; merging one or more rooms together; and aligning the occupancy map.
  14. 14 . A non-transitory computer readable medium comprising stored instructions for preparing a segmented occupancy grid map based at least in part upon an occupancy map, which instructions, when executed by one or more processors, implement actions comprising: reducing noise in an occupancy map; classifying voxels as (i) free, (ii) occupied, or (iii) unexplored; removing ray areas; removing any obstacles within rooms and any obstacles attached to boundaries; computing for each non-zero pixel, a distance to a closest zero pixel; finding candidate seeds based, at least in part, upon binarizing the distance with a threshold change and finding blobs with a blob size meeting a threshold blob size; dilating the blobs; and removing any noise blobs; watersheding the blobs until one or more boundaries are encountered; merging one or more rooms together; and aligning the occupancy map.
  15. 15 . The non-transitory computer readable medium of claim 14 , wherein classifying voxels further includes: setting a binary threshold to find free and occupied voxels; filling holes according to surrounding voxels including: if there are more free points around any voids, the voids will become free; otherwise, smaller voids will become occupied, and larger voids will remain unexplored; and based, at least in part, upon sensory information, repairing defects.
  16. 16 . The non-transitory computer readable medium of claim 14 , wherein removing ray areas further includes: finding one or more free edges in the occupancy map; and drawing a line between at least two voxels in nearby edges, if the line is not blocked by occupied by a voxel or a sensor voxel.
  17. 17 . The non-transitory computer readable medium of claim 14 , wherein the occupancy map is determined based, at least in part, upon image information.
  18. 18 . The non-transitory computer readable medium of claim 14 , wherein occupancy map is determined based, at least in part, upon image information captured by an at least one visual spectrum-capable.
  19. 19 . The non-transitory computer readable medium of claim 14 , wherein occupancy map determined based, at least in part, upon image information captured by an at least one visual spectrum-capable camera and location information captured by an at least one depth measuring camera.
  20. 20 . The non-transitory computer readable medium of claim 14 , wherein the meeting a threshold blob size comprises meeting a blob size threshold of 2000 pixels.

Description

PRIORITY APPLICATION This application is a continuation of U.S. application Ser. No. 18/081,672, titled “Occupancy Map Segmentation for Autonomous Guided Platform With Deep Learning,” filed 14 Dec. 2022, now U.S. Pat. No. 12,514,419, issued 6 Jan. 2026 (Attorney Docket No. TRIF 6005-2), which claims the benefit of Chinese Application No: 202111613025.5, filed 27 Dec. 2021, titled “3D Geometric and Semantic Awareness with Deep Learning for Autonomous Devices”, the entire contents of which are incorporated herein by reference. U.S. application Ser. No. 18/081,672 also claims the benefit of U.S. Provisional Application No. 63/294,907, titled “Occupancy Map Segmentation For Autonomous Guided Platform With Deep Learning,” filed 30 Dec. 2021 (Attorney Docket No. TRIF 6005-1), the entire contents of which are incorporated herein by reference. INCORPORATIONS The following materials are incorporated herein by reference in their entirety for all purposes: U.S. Provisional Application No. 63/294,899, filed 30 Dec. 2021, titled “Autonomous Guided Platform With Deep Learning Environment Recognition And Sensor Calibration” (Attorney Docket No. TRIF 6001-1);U.S. Provisional Application No. 63/294,901, titled “3D Geometric And Semantic Awareness With Deep Learning For Autonomous Guidance,” filed 30 Dec. 2021 (Attorney Docket No. TRIF 6002-1);U.S. Provisional Application No. 63/294,903, titled “Training Of Deep Learning Neural Networks Of Autonomous Guided Platform,” filed 30 Dec. 2021 (Attorney Docket No. TRIF 6003-1);U.S. Provisional Application No. 63/294,904, titled “Preparing Training Data Sets For Deep Learning Neural Networks Of Autonomous Guided Platform,” filed 30 Dec. 2021 (Attorney Docket No. TRIF 6004-1);U.S. Provisional Application No. 63/294,908, titled “Calibration For Multi-Sensory Deep Learning Autonomous Guided Platform,” filed 30 Dec. 2021 (Attorney Docket No. TRIF 6006-1); andU.S. Provisional Application No. 63/294,910, titled “Self Cleaning Docking Station For Autonomous Guided Deep Learning Cleaning Apparatus,” filed 30 Dec. 2021 (Attorney Docket No. TRIF 6007-1). This application is also related to the following contemporaneously filed applications which are incorporated herein by reference in their entirety for all purposes: U.S. Non-Provisional application Ser. No. 18/081,668, titled “Autonomous Guided Platform With Deep Learning Environment Recognition And Sensor Calibration,” filed 14 Dec. 2022 (Attorney Docket No. TRIF 6001-2);U.S. Non-Provisional application Ser. No. 18/081,669, titled “3D Geometric And Semantic Awareness With Deep Learning For Autonomous Guidance,” filed 14 Dec. 2022 (Attorney Docket No. TRIF 6002-2);U.S. Non-Provisional application Ser. No. 18/081,670, titled “Training Of Deep Learning Neural Networks Of Autonomous Guided Platform,” filed 14 Dec. 2022 (Attorney Docket No. TRIF 6003-2);U.S. Non-Provisional application Ser. No. 18/081,671, titled “Preparing Training Data Sets For Deep Learning Neural Networks Of Autonomous Guided Platform,” filed 14 Dec. 2022 (Attorney Docket No. TRIF 6004-2);U.S. Non-Provisional application Ser. No. 18/081,674, titled “Calibration For Multi-Sensory Deep Learning Autonomous Guided Platform,” filed 14 Dec. 2022 (Attorney Docket No. TRIF 6006-2);U.S. Non-Provisional application Ser. No. 18/081,676, titled “Self Cleaning Docking Station For Autonomous Guided Deep Learning Cleaning Apparatus,” filed 14 Dec. 2022 (Attorney Docket No. TRIF 6007-2); andU.S. Design application No. 29/863,047, titled “Self Cleaning Docking Station For Autonomous Guided Deep Learning Cleaning Apparatus,” filed 14 Dec. 2022 (Attorney Docket No. TRIF 6008-1). TECHNICAL FIELD The present disclosure relates to occupancy map segmentation for autonomous guided platform with deep learning techniques for environment recognition and sensor calibration, and more specifically to robots employing occupancy map segmentation for autonomous guided platform with deep learning techniques for environment recognition and sensor calibration. BACKGROUND The subject matter discussed in this section should not be assumed to be prior art merely as a result of its mention in this section. Similarly, a problem mentioned in this section or associated with the subject matter provided as background should not be assumed to have been previously recognized in the prior art. The subject matter in this section merely represents different approaches, which in and of themselves can also correspond to implementations of the claimed technology. Autonomous robots have long been the stuff of science fiction fantasy. One technical challenge in realizing the truly autonomous robot is the need for the robot to be able to identify where they are, where they have been and plan where they are going. Traditional techniques have improved greatly in recent years; however, there remains considerable technical challenge to providing fast accurate and reliable positional awareness to robots and self-guiding mobile platfor