US-12617616-B2 - Artificial intelligence bin recovery technique and downtime mitigation

US12617616B2US 12617616 B2US12617616 B2US 12617616B2US-12617616-B2

Abstract

A system includes one or more robot arms configured to pick and/or place items into totes. The system is designed to reduce overall downtime. In the system, if the robot arm is unable to pick and/or place items into the tote, the system routes the tote to a human operator who rearranges the items in the tote in such a way that the robot arm is more likely to be able to pick the items from the tote. The system further includes an artificial intelligence (AI) system that determines whether totes are able to be picked by the robot arm. The AI system further monitors the operation of the robot arms and human operators for training purposes. The data from picking and rearranging operations are used to train the AI system.

Inventors

Mark R Cass
Amey Sunil Kulkarni
Shelby Lynne Cass
Eric Michael Smith
Paola Andrea Gutierrez Guzman
Ruthvik Vaila

Assignees

BASTIAN SOLUTIONS, LLC

Dates

Publication Date: 20260505
Application Date: 20230801

Claims (17)

1 . A system, comprising: a robotic station including a robot configured to pick from and/or place into a tote one or more items; a robot camera positioned proximal to the robot to capture one or more images of the items in the tote; a computer configured to determine if the tote is pickable by the robot based on the images from the robot camera; an operator station located remote from the robotic station; an operator camera located at the operator station to monitor the operator station; wherein the computer is configured to determine when a human operator at the operator station is idle with the operator camera; wherein the computer is configured to route the tote from the robotic station to the operator station when the human operator is idle and the tote is unpickable by the robot; wherein the operator station is configured to facilitate manual rearrangement of the items in the tote by the human operator; and wherein the computer is configured to route the tote back from the operator station to the robotic station after determining the items in the tote are pickable with the operator camera.
2 . The system of claim 1 , further comprising: wherein the computer is configured to monitor tote rearrangement via the operator camera; an input/output (I/O) device including an indicator of proper tote arrangement; and wherein the I/O device is configured to provide instructions for proper tote arrangement to the human operator.
3 . The system of claim 1 , wherein: the computer includes an artificial intelligence (AI) system; the AI system is configured to determine if the tote is pickable by the robot based on the images from the robot camera; and the AI system is configured to learn operations from captured image data.
4 . The system of claim 3 , wherein the AI system is configured to learn how to pick from and/or place into the tote the items based on the captured image data.
5 . The system of claim 1 , wherein: the computer is configured to rank totes based on probability of successfully picking items from the totes; and the computer is configured to route totes to the robot based on the rankings.
6 . The system of claim 1 , further comprising: a network; and wherein the computer is configured to distribute previously captured imaging to the robot via the network.
7 . The system of claim 1 , further comprising: a second camera aimed towards the same items in the tote.
8 . The system of claim 1 , wherein: the computer is configured to determine the items in the tote at the operator station are pickable with the operator camera; and the computer is configured to activate an indicator when the items in the tote at the operator station are pickable.
9 . The system of claim 1 , wherein: the computer includes an artificial intelligence (AI) system; the AI system is configured to learn from the manual rearrangement of the items in the tote by the human operator captured by the operator camera; and the computer via learning by the AI system from the manual rearrangement is configured to change operation of the robot to enhance picking of the items.
10 . A method, comprising: capturing an image of a tote with a robot camera positioned proximal to a robot at a robotic station; determining the tote is unpickable by the robot based on the image with a computer; monitoring a human operator at an operator station with an operator camera; determining the human operator is idle at the operator station via the computer as a result of the monitoring; routing the tote from the robotic station to the operator station in response to the determining the tote in unpickable and the determining the human operator is idle; wherein the operator station is located remote from the robotic station; wherein the operator station is configured to facilitate manual rearrangement of the tote by the human operator; and routing the tote from the operator station to the robotic station after the human operator manually rearranges the tote to be pickable.
11 . The method of claim 10 , further comprising: wherein the tote contains one or more items; and picking at least one of the items in the tote with the robot.
12 . The method of claim 10 , further comprising: determining a chance of success of the robot rearranging items in the tote is above a threshold with the computer; and attempting to rearrange the items in the tote using the robot before the routing the tote to the human operator.
13 . The method of claim 10 , further comprising: determining the tote ranks ahead other totes with the computer based on probability of picking success; and routing the tote to the robot in response to the determining the tote ranks ahead other totes.
14 . A method, comprising: determining a first tote is unpickable by a first robot at a robotic station with a robot camera; routing the first tote to an operator station in response to the determining the first tote is unpickable; capturing one or more images of a human operator manually rearranging the first tote by hand with an operator camera at the operator station; routing the first tote from the operator station to the robotic station after the human operator manually rearranges the first tote to be pickable; distributing the images to one or more computers using a network; training at least one artificial intelligence (AI) system on at least one of the computers to perform a manipulation of one or more items using the images; and rearranging the items in a second tote with a second robot based on the training.
15 . The method of claim 14 , wherein the rearranging of the items includes picking and/or placing the items in the second tote with the second robot.
16 . The method of claim 14 , further comprising routing the first tote to the human operator who is idle.
17 . The method of claim 14 , further comprising: labeling the images associated with the manipulation with the AI system.

Description

BACKGROUND Facilities with various purposes utilize both human and robotic operators to perform tasks. For example, a facility utilizes both human and robotic operators to perform tasks related to storage, inventory management, manufacturing, fulfilling orders, and/or other purposes. It can be dangerous for human operators to interact with robotic operators during normal operations. The speed and efficiency of operations within such facilities may be reduced in order to operate safely. Thus, there is a need for improvement in this field. SUMMARY System uptime is always a concern with automated material handling systems like robotic equipment. Unfortunately, robotic equipment is sometimes unable to pick a bin and/or other product carrier. For example, sometimes the bins get jostled and/or otherwise disorganized so that there are no pick locations for the robot. When such a fault occurs, a human usually has to enter the work area of the robotic equipment to remove and/or rearrange the bin or take some other physical corrective action to address the situation. As should be appreciated, having a human enter the work area creates production downtime, which can become costly. Furthermore, having a human enter the work area can be quite dangerous, and as a result, safety precautions must be taken which can lead to further downtime. A unique automated downtime mitigation system has been developed to handle such faults as well as other issues during the material handling process. The system is configured to automatically determine a bin status and act in a predetermined manner based on the determined bin status. For example, the system is configured to automatically route unpickable bins to a human operator for reorganization. In another example, the system is configured to automatically route unpickable bins to a robot for reorganization. In yet another example, the system is configured to determine the probability of a successful bin pick and rank the bins based on the probability. For example, the system then routes bins with the highest pick probability to the robot and routes bins with the lowest pick probability to a human and/or robot for reorganization. Furthermore, the system is configured to handle the faults without a pause and/or stop in the material handling process such that system downtime is mitigated. The automated downtime mitigation system includes an artificial intelligence (AI) system configured to determine the bin status. In one example, the AI system is configured to work with one or more cameras. For example, the cameras are configured to monitor one or more bins as they approach the pick location. In one embodiment, the system determines that the bin is pickable and continues through the material handling system without issue. In another embodiment, the system routes the bin to a human operator for remediation and/or reorganization. After the operator reorganizes the bin, the bin is placed back into the material handling system for picking. In one example, the system includes a physical indicator of proper bin arrangement. For example, the system includes an indicator light and/or alert message. As should be appreciated, the system is configured to automatically route the unpickable bin to a human operator without stopping and/or slowing the material handling system. Thus, system downtime is mitigated. The AI system is further configured to capture and label imaging data during bin picking and rearrangement for use in AI training, AI testing, and system optimization. In one embodiment, the AI system records data while monitoring the robotic arm. For example, the AI system records data about the movement of a robotic arm during a successful pick operation and labels the movement as successful. In another embodiment, the AI system monitors human operators picking or reorganizing items in a bin. For example, the AI system records and learns each step that a human operator takes to orient an item in a bin. Machine learning is used to train the system based on data recorded during bin picking operations. For example, a neural network is used to train the AI system using data from successful and/or failed pick operations. Testing may be performed on the AI system, and recorded testing data may be used for additional training of the AI system. For example, the AI system performs testing by attempting to pick randomized items from different bins using the robotic arms. The recorded data may be distributed among multiple robots. For example, the system is configured to propagate image data to all robots via a wireless communication system to provide AI training for new and/or replacement robots. In another embodiment, the data may be transferred to a centralized computer to train the AI system. In another embodiment, the system monitors operator/robot downtime. Based on the status of an operator/robot (e.g., idle and/or active), the system routes bins for reorganization to idle operators/robots to m