EP-3849754-B1 - MANIPULATING FRACTURABLE AND DEFORMABLE MATERIALS USING ARTICULATED MANIPULATORS

EP3849754B1EP 3849754 B1EP3849754 B1EP 3849754B1EP-3849754-B1

Inventors

JOHNSON, David, M.S.
WAGNER, Syler
LINES, Steven
HEBERT, Mitchell

Dates

Publication Date: 20260506
Application Date: 20190913

Claims (20)

A method comprising: determining a set of trajectories of an end-effector that collect an amount of material from a collection of materials by employing a representation of the collection of material generated by observations of an environment by at least one sensor system of a machine having an articulated robot arm, the collected material deforming or fracturing during collection, the robot arm being configured to move the end-effector able to manipulate the collection of materials, the representation including one or more of a surface of the material, a point cloud, object locations, sizes, shapes, orientations, material type, condition of the material, and a depth image, the amount of material being one or more of a given mass, volume, or quantity of material, based on a volume of intersection or an estimate of the volume of the intersection between the end-effector and the material volume, and based on at least one physical reaction of the material with the end-effector; selecting a given trajectory of the trajectories based on at least one criterion; and sending the given trajectory to the machine configured to move the end-effector, causing the machine to execute the trajectory using measurements of the at least one sensor system.
The method of Claim 1, wherein executing the trajectory includes moving at least one of the end-effector and components of the machine within a threshold of the given trajectory.
The method of Claim 1, wherein determining the trajectories includes determining a trajectory free from collision with an object in an environment based on at least one of the following: a physical model of the environment, and output from one or more of the sensor systems.
The method of Claim 1, wherein selecting the given trajectory includes employing a criterion based on at least one of the following: execution time, intersection volume, physical reaction of the material with the end-effector, proximity to collision with other objects, proximity to hardware physical constraints including at least one of joint limits acceleration limits, velocity limits, and jerk limits, path length, required torque, required energy or effort, and additional computational effort.
The method of Claim 1, further comprising determining at least one of an end-effector trajectory, a joint position, and a joint torque based on an estimated property of the material, the property including at least one of stiffness, viscosity, viscoelasticity, fracture toughness, material density, serving mass, serving volume, friction coefficient, and material grain size.
The method of Claim 1, wherein selecting the given trajectory further includes, choosing the given trajectory based on the criterion having a numerical score, the numerical score including one or more of shortest joint-space distance, shortest end-effector distance, minimum energy, smallest maximum torque, shortest time, the amount of material, and largest clearance between obstacles.
The method of Claim 1, further comprising: during execution of the given trajectory, compensating for errors in trajectory tracking based on: tracked position, tracked velocity, and tracked acceleration of at least one actuator of the robotic arm, the trajectory, the tracked velocity, and the tracked acceleration, being of at least one portion of the robot or a frame of the robot arm that can be transformed to one or more of the tracked position, tracked velocity, and tracked acceleration, or measured or inferred force or torque on at least one joint of the robotic arm, on an end-effector of the robot arm, or on a frame of the robot arm that can be transformed to measured or inferred torque; maintaining at least one criterion, including one or more of a volume of material intersection and physical reaction of the material.
The method of Claim 1, further comprising: determining the amount of material to be collected based on one or more of: direct computation of an intersection volume of an end-effector coupled with the machine for the given trajectory with the collection of materials and the physical reaction to the end-effector, an approximation of at least one of the volume and mass of the captured material established by a neural network or statistical model, and a heuristic function providing the amount of material to be collected by the end-effector for the given trajectory.
The method of Claim 1, wherein selecting the given trajectory further includes: determining the amount of material to be collected in the end-effector by at least one of the trajectories in the set of trajectories, and selecting based on the amount of material to be collected; or generating the trajectory by employing a generation function based on the amount of material to be collected.
The method of Claim 1, further comprising: determining a trajectory generation function by at least one of: an approximation function, adversarial training, adversarial training including using a generative adversarial network (GAN), or reinforcement learning; wherein the approximation function is generated by a neural network.
The method of Claim 1, further comprising estimating a path of dispersing of the collected material into a predefined configuration and location based on at least one of an estimated viscosity, a viscoelasticity, a serving mass, a serving volume, a fracture toughness, and a density.
The method of Claim 1, further comprising: determining a destination for material collected by the robotic arm, and a destination configuration for the material collected, the destination configuration being one or more of a desired density distribution, voxel representation of the material, material surface, or a probability distribution of any of the prior representations.
The method of Claim 1, further comprising: determining, as part of the trajectory, removing the material from the end effector of the robotic arm.
The method of Claim 1, further comprising: creating a torque or effort profile for given material characteristics based on one or more of a tool geometry, an intersection profile with the material, non-linear effects of velocity and acceleration, and acceleration dependent forces.
The method of Claim 1, further comprising: adjusting, by at least one closed loop controller, a position, a velocity, an acceleration, or a torque of at least one actuator of the robotic arm, based on a camera, depth sensor input, robot position, or a torque sensor, causing the amount of collected material to be within a threshold of a goal amount; employing an estimator to determine an intermediate system representation based on the sensor data, and determine a control output based on the intermediate system representation; choosing an active controller from a set of closed loop controller based on which best matches the current conditions based on a quality metric; wherein each of the closed loop controllers is configured to employ a set of operation conditions, wherein each of the closed loop controllers is configured to be a neural network trained based on collected data or simulated data, the neural network being adversarily trained, and wherein each of the at least one closed loop controllers may be tuned by reinforcement learning or optimization based on experiments.
A system comprising: an end-effector; an articulated robot arm; a machine having the articulated robot arm and at least one sensor system, wherein the robot arm is configured to move the end-effector able to manipulate the collection of materials; a processor; and a memory with computer code instructions stored thereon, the processor and the memory, with the computer code instructions, being configured to cause the system to: determine a set of trajectories of the end-effector that collect an amount of material from a collection of materials by employing a representation of the collection of material generated by observations of an environment by the at least one sensor system, the collected material deforming or fracturing during collection, the representation including one or more of a surface of the material, a point cloud, object locations, sizes, shapes, orientations, material type, condition of the material, and a depth image, the amount of material being one or more of a given mass, volume, or quantity of material, based on a volume of intersection or an estimate of the volume of the intersection between the end-effector and the material volume, and based on at least one physical reaction of the material with the end-effector; select a given trajectory of the trajectories based on at least one criterion; and send the given trajectory to the machine configured to move the articulated robot arm and the end-effector, causing the machine to execute the trajectory using measurements of the at least one sensor system.
The system of Claim 16, wherein executing the trajectory includes moving the robotic arm within a threshold of the given trajectory.
The system of Claim 16, wherein determining the trajectories includes determining a trajectory free from collision with an object in an environment based on at least one of the following: a physical model of the environment, or output from one or more of the sensor systems.
The system of Claim 16, wherein selecting the given trajectory includes employing a criterion based on at least one of the following: execution time, intersection volume, proximity to collision with other objects, proximity to hardware physical constraints including at least one of joint limits, acceleration limits, velocity limits, and jerk limits, path length, required torque, required energy or effort, and additional computational effort.
The system of Claim 16, further comprising determining at least one of an end-effector trajectory, a joint position, and a joint torque based on an estimated property of the material, the property including at least one of stiffness, viscosity, viscoelasticity, fracture toughness, material density, serving mass, serving volume, friction coefficient, and material grain size.

Description

BACKGROUND Traditionally, the food industry employs human labor to manipulate ingredients with the purpose of either assembling a meal such as a salad or a bowl, or packing a box of ingredients such as those used in grocery shopping, or preparing the raw ingredients. Robots have not yet been able to assemble complete meals from prepared ingredients in a foodservice setting such as a restaurant, largely because the ingredients are arranged unpredictably and change shape in difficult-to-predict ways rendering traditional methods to move material ineffective without extensive modifications to existing kitchens. Additionally, traditional material handling methods are ill-suited to moving cooked foods without altering their texture and taste-profile. These difficulties arise because the friction, stiction, and viscosity of commonly consumed foods cause auger, conveyor, and suction mechanisms to become clogged and soiled, while these mechanisms simultaneously impart forces on the foodstuffs which alter their texture, consistency, and taste-profile in unappetizing ways. US 2018/147718 A1 is directed to methods, computer program products, and computer systems for instructing a robot to prepare a food dish by replacing the human chef's movements and actions. Monitoring a human chef is carried out in a standardized robotic kitchen, and involves using sensors and computers to watch, monitor, record and interpret the motions and actions of the human chef, in order to develop a robot-executable set of commands robust to variations and changes in the environment, capable of allowing a robotic or automated system in a robotic kitchen to prepare the same dish to the standards and quality as the dish prepared by the human chef. US 2015/114236 A1 relates to an apparatus for preparing dishes using recipe steps and cooking conditions coded as a computer program recipe executed by a kitchen top robotic cooking machine supplied with the required ingredients with minimal user intervention while adjusting recipes to tastes of the user. The robot head assembly is able to transfer ingredients, identify and measure as well as take inventory periodically. CN 107092209 relates to a device for cooking rice having a fuselage with a water supply installation, rice-washing means, storage bowl device, point bowl device, and a steaming plant. The device automatically and continuously cooks many portions of rice and can simultaneously boil one or more parts of rice. WO 2018/ 133861 relates to a standardized, automatic cooking apparatus and a method of using it for cooking. The automatic cooking apparatus comprises at least one holding device configured to receive and store unit material, wherein the unit material have corresponding material information; at least one cooking device configured to receive and process unit material from the at least one holding device; a material transfer device configured to transfer the unit material between the at least one holding device and the at least one cooking device; a control device configured to acquire material information of the unit material and control the at least one cooking device to process the received unit material. SUMMARY Autonomous manipulation of objects and materials in unstructured environments is an unsolved problem in robotics. Substantial progress has been made in grasping a priori unknown rigid objects and placing them in a specific location (e.g. "pick and place"). However, materials that deform during manipulation, or require fracturing or breaking to separate a piece to manipulate, are substantially more difficult to manipulate. A successful pick and place operation of an un-occluded object, typically requires one, challenging and expensive computation, which is to find the configuration that satisfies a set of constraints enabling the robot to firmly grasp the object. However, successfully picking or scooping a given quantity of a deformable material from a container also requires additional set of constraints such as separating a desired quantity of material from the bulk, and keeping the object from falling out while in transition, which then necessitates finding an entire trajectory that satisfies a set of constraints. The need for the entire trajectory to satisfy certain constraints makes the problem of manipulating deformable material significantly more difficult. Deformable and fracturable materials are commonly encountered in food & beverage industry, construction, and manufacturing. Example materials include granular materials (grains, croutons, corn kernels, soil, snow, construction debris, leafy greens, etc.), soft solid materials (ice cream, butter, guacamole, mashed potatoes, etc.), collections (cubed chicken, steamed carrots, green beans, sliced cucumbers, cherry tomatoes, etc.), and deformable items (tomatoes, peppers, chicken breasts, hamburgers, etc.), etc. These materials are grouped into classes, which may contain seemingly unrelated materials, having similar