EP-4737977-A1 - MANIPULATION SYSTEM AND MANIPULATION METHOD

EP4737977A1EP 4737977 A1EP4737977 A1EP 4737977A1EP-4737977-A1

Abstract

A manipulation system (310) includes a control apparatus (330). The control apparatus (330) includes: a movement amount acquisition unit (56) that acquires a movement amount of the manipulator; a position identification unit (58) that identifies a position of the sample and a position of the manipulator based on an image captured by the imaging apparatus (24); and a calculation unit (376) that calculates a target position or target movement amount of the manipulator for performing a predetermined operation on the sample based on the position of the sample and the position or movement amount of the manipulator.

Inventors

AOYAMA, Tadayoshi
SAKAMOTO, KAZUYA
MORI, RYOYA
KOBAYASHI, TAISUKE

Assignees

NATIONAL UNIVERSITY CORPORATION TOKAI NATIONAL HIGHER EDUCATION AND RESEARCH SYSTEM
Inter-University Research Institute Corporation Research Organization of Information and Systems

Dates

Publication Date: 20260506
Application Date: 20240619

Claims (18)

A manipulation system comprising: a manipulator for manipulating a sample; a manipulator drive mechanism for moving the manipulator; an imaging apparatus that images the sample and the manipulator through an objective lens; and a control apparatus, wherein the control apparatus includes: a movement amount acquisition unit that acquires a movement amount of the manipulator; a position identification unit that identifies a position of the sample and a position of the manipulator based on an image captured by the imaging apparatus; and a calculation unit that calculates a target position or target movement amount of the manipulator for performing a predetermined operation on the sample based on the position of the sample and the position or movement amount of the manipulator.
The manipulation system according to Claim 1, wherein the calculation unit calculates the target position or the target movement amount based on time-series data in which the position of the sample and the position or movement amount of the manipulator are associated with each other in time series.
The manipulation system according to Claim 1, wherein the calculation unit calculates the target position or the target movement amount by using a trained model that has undergone machine-learning using teaching data in which the position of the sample and the position or movement amount of the manipulator are associated with each other in time series, which are acquired when a skilled operator performs the predetermined operation.
The manipulation system according to any one of Claims 1 through 3, wherein the calculation unit calculates the target movement amount based on the position of the sample and the movement amount of the manipulator.
The manipulation system according to Claim 4, wherein the calculation unit includes: a first calculation unit that uses the position of the sample and the movement amount of the manipulator as input and calculates a first movement amount for performing a predetermined operation on the sample; a second calculation unit that uses the position of the manipulator as input and calculates a second movement amount of the manipulator for guiding the manipulator to a reference trajectory set around the sample; and a third calculation unit that uses the first and second movement amounts so as to calculate the target movement amount of the manipulator.
The manipulation system according to Claim 5, wherein the reference trajectory is set based on a probability density function that represents distribution of positions of the manipulator acquired when a skilled operator performs the predetermined operation.
The manipulation system according to Claim 5 or 6, wherein the third calculation unit calculates the target movement amount by weight-averaging the first movement amount and the second movement amount by using a weight coefficient according to a distance from the reference trajectory to the position of the manipulator.
The manipulation system according to any one of Claims 5 through 7, further comprising: a display apparatus that displays operation support information that is based on the target movement amount and the reference trajectory.
The manipulation system according to any one of Claims 5 through 7, wherein the calculation unit further calculates the reliability level of the first movement amount, the manipulation system further comprising: a display apparatus that displays operation support information that is based on the target movement amount and the reliability level.
The manipulation system according to any one of Claims 1 through 3, further comprising: a display apparatus that displays operation support information that is based on the target position and the target movement amount.
The manipulation system according to any one of Claims 1 through 3, wherein the calculation unit further calculates the reliability level of the target position or the target movement amount, the manipulation system further comprising: a display apparatus that displays operation support information that is based on the target position or the target movement amount and on the reliability level.
The manipulation system according to any one of Claims 1 through 11, further comprising: a force sensation presentation apparatus that is configured to present a force sensation according to the target position or the target movement amount to a user.
The manipulation system according to any one of Claims 5 through 9, further comprising: a force sensation presentation apparatus that is configured to present a force sensation according to the target movement amount to a user, wherein force sensation presentation apparatus performs: presenting a force sensation with a magnitude of force of a constant value regardless of the magnitude of the target movement amount if the distance from the reference trajectory to the current position of the manipulator is equal to or greater than a predetermined value; and presenting a force sensation with a magnitude of force that is variable according to the magnitude of the target movement amount if the distance from the reference trajectory to the current position of the manipulator is less than the predetermined value.
The manipulation system according to Claim 9, further comprising: a force sensation presentation apparatus that is configured to present a force sensation a user, wherein the force sensation presentation apparatus performs: presenting a force sensation with a magnitude of force that is variable according to the magnitude of the target movement amount if the target movement amount is smaller than the reliability level; and presenting a force sensation with a magnitude of force of a constant value regardless of the magnitude of the target movement amount if the target movement amount is larger than the reliability level.
The manipulation system according to Claim 11, further comprising: a force sensation presentation apparatus that is configured to present a force sensation a user, wherein the force sensation presentation apparatus performs: presenting a force sensation with a magnitude of force of a constant value regardless of the distance from the current position to the target position if the current position is outside a confidence circle having the target position as a center thereof and the reliability level as a radius thereof, and presenting a force sensation with a magnitude of force that is variable according to the distance from the current position to the target position if the current position is inside the confidence circle.
The manipulation system according to any one of Claims 1 through 15, wherein the manipulator drive mechanism moves the manipulator based on the target position or the target movement amount.
A manipulation method comprising: acquiring an image of a sample and a manipulator captured through an objective lens; acquiring a movement amount of the manipulator; identifying a position of the sample and a position of the manipulator based on the image acquired; calculating a target position or target movement amount of the manipulator for performing a predetermined operation on the sample based on the position of the sample and the position or movement amount of the manipulator.
A program comprising computer-implemented modules including: a module that acquires an image of a sample and a manipulator captured through an objective lens; a module that acquires a movement amount of the manipulator; a module that identifies a position of the sample and a position of the manipulator based on the image acquired; and a module that calculates a target position or target movement amount of the manipulator for performing a predetermined operation on the sample based on the position of the sample and the position or movement amount of the manipulator.

Description

TECHNICAL FIELD The present disclosure relates to manipulation systems and manipulation methods. BACKGROUND ART Manipulation systems for manipulating samples such as cells by using a manipulator are known. Since cells are miniscule, cells are manipulated while the positions of the cells and the manipulator are observed under a microscope. For example, a technology has been proposed that is for improving the operability of a sample by generating force information indicating a magnitude of a force sensation presented to a user based on an image captured of the sample and presenting the force sensation to the user via a force sensation presentation apparatus. RELATED-ART LITERATURE PATENT LITERATURE Patent Literature 1: WO 2023/027095 SUMMARY OF INVENTION TECHNICAL PROBLEM In intracytoplasmic sperm injection, in which a sperm is injected into an egg cell using a pipette, manipulation to rotate the cell in three dimensions is required as a preliminary preparation for inserting the pipette into the cells while avoiding the vicinity of the polar body of the cell. Skilled cell manipulation operators can skillfully rotate cells while minimizing cell damage based on two-dimensional visual information from a microscope. However, for beginners in cell manipulation, it is not easy to rotate and manipulate cells properly based only on visual information, and it usually takes a long time (for example, one year or more) to master the skill. The present disclosure addresses the issue described above, and an illustrative purpose is to provide a technology for improving the operation accuracy when performing a predetermined operation on a sample using a manipulator. SOLUTION TO PROBLEM A manipulation system according to one embodiment of the present disclosure includes: a manipulator for manipulating a sample; a manipulator drive mechanism for moving the manipulator; an imaging apparatus that images the sample and the manipulator through an objective lens; and a control apparatus. The control apparatus includes: a movement amount acquisition unit that acquires a movement amount of the manipulator; a position identification unit that identifies a position of the sample and a position of the manipulator based on an image captured by the imaging apparatus; and a calculation unit that calculates a target position or target movement amount of the manipulator for performing a predetermined operation on the sample based on the position of the sample and the position or movement amount of the manipulator. Another embodiment of the present disclosure relates to a manipulation method. This manipulation method includes: acquiring an image of a sample and a manipulator captured through an objective lens; acquiring a movement amount of the manipulator; identifying a position of the sample and a position of the manipulator based on the image acquired; and calculating a target position or target movement amount of the manipulator for performing a predetermined operation on the sample based on the position of the sample and the position or movement amount of the manipulator. Optional combinations of the aforementioned constituting elements, and constituting elements and implementations of the disclosure in the form of methods, systems, programs, etc., may also be practiced as additional modes of the present disclosure. ADVANTAGEOUS EFFECTS OF INVENTION According to the present disclosure, the operation accuracy when performing a predetermined operation on a sample using a manipulator can be improved. BRIEF DESCRIPTION OF DRAWINGS Fig. 1 is a diagram schematically showing a configuration of the manipulation system according to the first embodiment.Fig. 2 is a block diagram schematically showing a functional configuration of an image recognition device according to the first embodiment.Figs. 3(a) to 3(c) are diagrams schematically showing how a sample is manipulated by a manipulator.Fig. 4(a) is a diagram showing an example of an operation trajectory of the manipulator when a skilled operator performs a predetermined operation, and Fig. 4(b) is a diagram showing an example of a reference trajectory set around the sample.Figs. 5(a) and 5(b) are diagrams schematically showing a calculation method for the second movement amount.Figs. 6(a) and 6(b) are diagrams schematically showing an example of setting a weight coefficient for calculating a target movement amount.Fig. 7 is a diagram showing an example of operation support information displayed on a display apparatus.Fig. 8 is a diagram showing an example of the operation support information displayed on the display apparatus.Figs. 9(a) to 9(c) are diagrams showing an example of operation trajectories of the manipulator when a beginner performs a predetermined operation.Fig. 10 is a flowchart showing an example of a manipulation method according to the first embodiment.Fig. 11 is a block diagram schematically showing a functional configuration of a control apparatus according to the second embod