US-12626476-B1 - System to determine humans in a physical space

US12626476B1US 12626476 B1US12626476 B1US 12626476B1US-12626476-B1

Abstract

A physical space includes humans and other objects. Humans may be dynamic in that they move while nonhuman objects may not move over time. An autonomous mobile device (AMD) determines which objects are human or other. Once determined, information about which objects are humans or not is used to provide functions such as facilitating interactions with people, determining accurate occupancy maps that do not incorrectly include the people, and so forth. Objects are represented as clusters, such as a two-dimensional arrangement of squares or a three-dimensional arrangement of cubes. Clusters are processed using one or more classification methods. If the classification indicates a human, one or more extraction techniques may be used to distinguish the portion of the cluster that is associated with the human object and that which is associated with a nonhuman object, such as furniture. The occupancy map, updated with nonhuman objects, may be used for navigation.

Inventors

BOSHEN NIU
Shreekant Gayaka

Assignees

AMAZON TECHNOLOGIES, INC.

Dates

Publication Date: 20260512
Application Date: 20220921

Claims (20)

1 . An autonomous mobile device (AMD) located in a physical space, the AMD comprising: one or more sensors; one or more processors; and one or more memories storing computer-executable instructions which, when executed using the one or more processors, cause the AMD to: determine, using the one or more sensors, sensor data indicative of one or more objects within the physical space; determine, based on the sensor data, cluster data indicative of at least a first cluster representative of a first volume within the physical space that includes at least one object of the one or more objects; determine a geometric center of the first cluster; determine a first distance from the geometric center to a point on the first cluster; determine a second volume based on the first distance; determine a first ratio based on the first volume and the second volume; and determine, based on a comparison of the first ratio to a first threshold, first classification data that is associated with at least a portion of the first cluster and is indicative of the at least one object.
2 . The AMD of claim 1 , wherein the one or more memories store computer-executable instructions which, when executed using the one or more processors, cause the AMD to: determine a first boundary of the first cluster; determine a first point on the first boundary, wherein the first point is a farthest point from the geometric center; determine a second point on the first boundary, wherein the second point is a nearest point to the geometric center; and wherein the point on the first cluster is one of the first point on the first boundary or the second point on the first boundary.
3 . The AMD of claim 1 , wherein the first cluster comprises a plurality of cells that are one or more of: adjacent to, or within a threshold distance of, one another, and further wherein each cell is representative of a particular area within the physical space.
4 . The AMD of claim 1 , wherein the one or more memories store computer-executable instructions which, when executed using the one or more processors, cause the AMD to: determine a first area associated with the first cluster; determine, based on the sensor data, a first location indicative of a human in the physical space, wherein the first location is within the first cluster; determine a second area that is associated with the first location; determine a second ratio based on a portion of the second area that overlaps the first area; and determine, based on a comparison of the second ratio to a second threshold, second classification data.
5 . The AMD of claim 1 , wherein the first cluster comprises a plurality of cells with each cell representative of a particular area within the physical space; and the one or more memories store computer-executable instructions which, when executed using the one or more processors, cause the AMD to: determine, based on the sensor data, semantic segmentation data that is indicative of: a first plurality of cells of the first cluster that are classified as including at least a portion of a human, and a second plurality of cells that are classified as not including at least a portion of a human; determine a first number of cells in the first plurality of cells; determine a total number of cells in the first cluster; determine a second ratio based on the first number of cells and the total number of cells; and determine, based on a comparison of the second ratio to a second threshold, second classification data.
6 . The AMD of claim 5 , wherein the one or more memories store computer-executable instructions which, when executed using the one or more processors, cause the AMD to: responsive to the second ratio being greater than or equal to the second threshold: determine a second cluster comprising the first plurality of cells, wherein the second classification data associates the second cluster with a human, and determine a third cluster comprising the second plurality of cells, wherein the second classification data associates the third cluster with a nonhuman.
7 . The AMD of claim 1 , wherein the one or more memories store computer-executable instructions which, when executed using the one or more processors, cause the AMD to: determine object data based on the first classification data; and perform one or more operations based on the object data.
8 . A method comprising: acquiring sensor data using one or more sensors on an autonomous mobile device (AMD) in a physical space; determining, based on the sensor data, first cluster data representative of a portion of the physical space that includes at least one object; determining, based on the sensor data, a first location indicative of a human in the physical space, wherein the first location is within the portion of the physical space that includes the at least one object; determining a first volume that is associated with the first location; determining a first ratio based on a portion of the first volume that overlaps the portion of the physical space that includes the at least one object; and determining, based on a comparison of the first ratio to a first threshold, first classification data that is associated with the portion of the physical space.
9 . The method of claim 8 , the determining the first cluster data comprising: determining a plurality of cells, wherein each cell is representative of a particular portion of the physical space that includes at least a portion of the at least one object, that are one or more of: adjacent to, or within a threshold distance of, one another.
10 . The method of claim 8 , further comprising: determining a first area of the portion of the physical space; determining a geometric center of the portion of the physical space; determining a first distance from the geometric center to a first point, wherein the first point is at a greatest distance from the geometric center; determining a second area based on the first distance; determining a second ratio based on the first area and the second area; and determining, based on a comparison of the second ratio to a second threshold, second classification data.
11 . The method of claim 8 , further comprising: determining a second volume of the portion of the physical space that includes the at least one object; determining a geometric center of the portion of the physical space that includes the at least one object; determining a first point of the portion of the physical space that includes the at least one object that is a greatest distance from the geometric center; determining a first set of distances from the geometric center to the first point; determining a third volume based on the first set of distances; determining a second ratio based on the second volume and the third volume; and determining, based on a comparison of the second ratio to a second threshold, second classification data.
12 . The method of claim 8 , further comprising: determining a first area that is associated with the first location; determining a second ratio based on a portion of the first area that overlaps the portion of the physical space that includes the at least one object; and determining, based on a comparison of the second ratio to a second threshold, second classification data.
13 . The method of claim 8 , further comprising: determining, based on the sensor data, semantic segmentation data that is indicative of: a first plurality of cells of the portion of the physical space that are classified as containing at least a portion of a human, and a second plurality of cells that are classified as not containing at least a portion of a human; determining a first number of cells in the first plurality of cells; determining a total number of cells in the portion of the physical space; determining a second ratio based on the first number of cells and the total number of cells; and determining, based on a comparison of the second ratio to a second threshold, second classification data.
14 . The method of claim 8 , further comprising: determining that the portion of the physical space is associated with a classification value indicative of a human object, if the first ratio is greater than or equal to the first threshold; and determining that the portion of the physical space is associated with a classification value indicative of a nonhuman object, if the first ratio is less than the first threshold.
15 . The method of claim 8 , further comprising: determining object data based on the first classification data; and performing one or more operations based on the object data.
16 . An autonomous mobile device (AMD) located in a physical space, the AMD comprising: one or more sensors; one or more processors; and one or more memories storing computer-executable instructions which, when executed using the one or more processors, cause the AMD to: determine, using the one or more sensors, sensor data indicative of one or more objects within the physical space; determine, based on the sensor data, first cluster data representative of a portion of the physical space that includes at least one object of the one or more objects; determine, based on the sensor data, semantic segmentation data that is indicative of: a first plurality of cubes in the portion of the physical space that are classified as containing at least a portion of a human, and a second plurality of cubes in the portion of the physical space that are classified as not containing at least a portion of a human; determine a first number of cubes in the first plurality of cubes; determine a total number of cubes in the portion of the physical space; determine a first ratio based on the first number of cubes and the total number of cubes; determine, based on a comparison of the first ratio to a first threshold, first classification data that is associated with the at least the portion of the physical space that includes the at least one object; determine confidence data associated with the first classification data; and confirm or increase the confidence data associated with the first classification data by performing a second type of classification process using the first cluster data, wherein the performing the second type of classification process involves; determining, based on the sensor data, a first location indicative of a human in the physical space, wherein the first location is within the portion of the physical space that includes the at least one object, determining a first volume that is associated with the first location, determining a second ratio based on a portion of the first volume that overlaps the portion of the physical space that includes the at least one object, and determining, based on a comparison of the second ratio to a second threshold, second classification data.
17 . The AMD of claim 16 , wherein the first cluster data is representative of a third plurality of cubes that are one or more of: adjacent to, or within a threshold distance of one another, and further wherein each cube of the third plurality of cubes is representative of a particular portion of the physical space.
18 . The AMD of claim 16 , wherein the one or more memories store computer-executable instructions which, when executed using the one or more processors, cause the AMD to: determine that the portion of the physical space is associated with a classification value indicative of a human object, if the first ratio is greater than or equal to the first threshold; and determine that the portion of the physical space is associated with a classification value indicative of a human object and a classification value indicative of a nonhuman object, if the first ratio is less than the first threshold.
19 . The AMD of claim 16 , wherein the one or more memories store computer-executable instructions which, when executed using the one or more processors, cause the AMD to: determine object data based on the second classification data; and perform one or more operations based on the object data.
20 . The AMD of claim 16 , wherein the one or more memories store computer-executable instructions which, when executed using the one or more processors, cause the AMD to: determine that the portion of the physical space is associated with a classification value indicative of a human object, if the second ratio is greater than or equal to the second threshold; and determine that the first cluster data is associated with a classification value indicative of a nonhuman object, if the second ratio is less than the second threshold.

Description

BACKGROUND An autonomous mobile device (AMD) moves throughout a physical space and may interact with humans. BRIEF DESCRIPTION OF FIGURES The detailed description is set forth with reference to the accompanying figures. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical items or features. The figures are not necessarily drawn to scale, and in some figures, the proportions or other aspects may be exaggerated to facilitate comprehension of particular aspects. FIG. 1 illustrates a system for an autonomous mobile device (AMD) to determine humans in a physical space, according to some implementations. FIG. 2 is a block diagram of the components of the AMD, according to some implementations. FIG. 3 is a block diagram of some components of the AMD, such as network interfaces, sensors, and output devices, according to some implementations. FIG. 4 is a block diagram of an object module to determine object classification such as human and nonhuman, according to some implementations. FIG. 5 is a flow diagram of a first implementation of a process for determining classification of an object represented by a cluster. FIG. 6 is a flow diagram of a second implementation of a process for determining classification of an object represented by a cluster. FIG. 7 is a flow diagram of a third implementation of a process for determining classification of an object represented by a cluster. While implementations are described herein by way of example, those skilled in the art will recognize that the implementations are not limited to the examples or figures described. It should be understood that the figures and detailed description thereto are not intended to limit implementations to the particular form disclosed but, on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope as defined by the appended claims. The headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description or the claims. As used throughout this application, the word “may” is used in a permissive sense (i.e., meaning having the potential to), rather than the mandatory sense (i.e., meaning must). Similarly, the words “include”, “including”, and “includes” mean “including, but not limited to”. DETAILED DESCRIPTION An autonomous mobile device (AMD) such as a robot operates in a physical space that includes users, such as humans. Operations of the robot may include interacting with humans, moving autonomously between locations in the physical space without being driven or controlled by a user, and so forth. During operation, the AMD uses sensor data to determine clusters indicative of one or more objects in the physical space. Each cluster represents an area or volume of the physical space that is determined to include an object. For example, a cluster may comprise a plurality of cells, with each cell representing a particular area of the physical space. In another example, a cluster may comprise a plurality of cells in which each cell represents a particular volume of the physical space. The AMD determines a classification of an object represented by a cluster, such as whether the cluster is representative of a human object or a nonhuman object. Humans may also be designated as dynamic or non-stationary as they may move on their own accord. Nonhumans may also be designated as stationary, as they do not move on their own accord. The clusters and their associated classification data may be used to determine an occupancy map that represents the location of nonhuman (stationary) objects in the physical space. In one implementation, the occupancy map may be used to determine a path plan between locations in the physical space for autonomous movement. In another implementation, a map may represent the current or previous locations of human objects in the physical space. This map may be used during other operations, such as a search for a specified person. An incorrect classification of an object may result in the AMD determining an incorrect occupancy map that represents places where humans were detected as obstacles to movement. Over time, successive misclassifications of objects may result in an incorrect occupancy map of a physical space that depicts objects that are not present. While path planning, the more obstacles there are, the more difficult it is to plan a path. As a result, a cluttered occupancy map may slow down or completely prevent autonomous movement. Described in this disclosure are techniques and systems for classifying an object based on one or more characteristics of a cluster. A cluster may comprise a two-dimensional representation of an object, a two-dimensional (2D) projection of a three-dimensional (3D) object, or a 3D representation of the of the objec