IROS2019国际学术会议论文集 0609

Abstract Extensive research efforts have been made toward automating the microinjection of biological cells by leveraging micro robotic technologies However to best knowledge of the authors there is no report on the automation of the time consuming process moving the injection tools a micropipette a grippers etc and cells into the field of view FOV of microscope from the macro FOV outside the microscopic FOV This paper presents a novel macro micro conversion strategy and a grid detection and positioning algorithm to automate the time consuming step of moving the injection tools and cells to the microscopic FOV The proposed solution can free the technician from the laborious hand eye coordination operations for moving the injection tools and cells to the target position within the microscopic FOV Furthermore this paper proposes an auto focusing algorithm to automate the operation step moving down the gripper from the air outside the culture media and then precisely clamping a cell in the liquid environment for injection In the proposed solution the active window based auto focusing algorithm is developed to solve the challenging problem the image information is lost due to the viscous effect taking place when the gripper jaw touches the water surface The proposed solutions are tested and validated by the microinjection experiments of zebrafish embryos using the in house develop micro robotic system The technologies and strategies proposed in this paper significantly improve the automation level of the cell microinjections and can be easily extended to any other micromanipulation of biological cells I INTRODUCTION In experimental biology and drug development cell microinjection has been widely accepted as the technology used for delivering exogenous materials such as DNAs RNAs sperms proteins and drug compounds into biological cells using a sharp tipped micropipette installed on a micro manipulator 1 7 To reduce the human involvement in tedious micro operations and to improve the consistence speed accuracy and success rate of microinjection micro robotic technologies have been developed for automating the microinjection operations Many processes for micromanipulation have been automated such as vitrification of mammalian embryos 8 automatic suction of sperm by micropipette 9 identification and position of specific structures of cell 10 14 perception and control of needle penetration force 15 19 adjustment of cell posture 20 22 In the robotic solutions to the automation of microinjection in This work was supported in part by 2018 Inovative Methodology Project 2018IM010400 H Zhang L Su corresponding author and H Wei Y Yu are with the College of Mechanical Engineering and Applied Electronics Technology Be ijing University of Technology Beijing China email huipeng zhang ema sly hongmiao wei yqyu X Zhang corresponding author is with the Department of Engineering Aarhus University Aarhus Denmark e mail xuzh eng au dk Figure 1 The schematic illustration of microinjection system The blue box illustrates the operations that need to be automated zoom in the current literatures the two time consuming operations still need to be conducted manually as shown in Fig 1 One operation denoted by the yellow arrow is moving the injection tools and cells into the FOV of microscope from the macro FOV The other operation marked by the red arrow is moving the injection tools into the culture media to the target location close to the cells In cell microinjection the microscope is an indispensable device for imaging the microscale size of biological cells microgrippers injection micropipette tips etc The real situation is that the FOV of the microscope is very limited and hence it is necessary to move the target tool parts and cells from the outside into the microscopic FOV using manually hand eye coordination back and forth for couple of times before performing the microinjection This operation is time consuming and laborious and there is no published research work toward automating this manual operation The first major contribution of this paper is to propose the macro micro conversion strategy and the grid detection and positioning algorithm to automatically move the operation tools and cells to a predetermined position in the microscopic FOV from the macro FOV After the injection micropipette microgripper and cells are moved into the FOV of the microscope the next step is to move the tools into culture media so that the tools and cells are to be aligned onto the common focal plane for microinjection Several alignment technologies have been reported in the literature In the first method the auto focusing alignment method was proposed to determine the focal plane of cells and the micropipette and then the micropipette is moved to the same plane of the embryonic cells 23 26 The second approach is to estimate the focal plane by measuring the width of the inner space of a holding pipette 10 The width of the Automated Macro Micro Manipulation for Robotic Microinjection with Computer Vision Huipeng Zhang Liying Su Hongmiao Wei Yueqing Yu and Xuping Zhang Member IEEE 2019 IEEE RSJ International Conference on Intelligent Robots and Systems IROS Macau China November 4 8 2019 978 1 7281 4003 2 19 31 00 2019 IEEE652 inner space of the holding pipette obtained after the image processing of pipette at different positions is different The largest width of the inner space is considered to be taken at the focal plane position The third alignment method is contact detection This method was introduced to determine the z axis position of the end effector the injection micropipette by measuring the horizontal motion displacement of the pipette under bending deformation when it touches the substrate of the glass slide 8 27 All the aforementioned three alignment methods were implemented based on the assumption that the tools are already moved into the culture media where the cells are located Under microscopic imaging condition moving the tools into the culture media from the air environment meets a challenging issue viscous effect of the microgripper jaws namely image information loss occurring when the gripper tip enters the liquid environment The second major contribution of this paper is to propose the active window based auto focusing algorithm to automate the operation of moving down the tools to the culture media and aligning the tools with the cells onto the common focal plane The rest of this paper is organized as follows Section II gives the description of robotic cell microinjection process and problem analysis Section III presents the procedure and principle of macro micro conversion strategy and a grid detection and positioning algorithm for the automation of moving the injection tools and cells to the microscopic FOV Section IV addresses the automation of moving the tools to culture medium from the air environment and aligning the tools with cells using the proposed active window based auto focusing algorithm Experimental testing and results are detailed and discussed in Section V and conclusions are made in Section VI II CELL MICROINJECTION PROCESS AND PROBLEM DESCRIPTION In this work the microinjection of zebrafish embryos is used as the representative case of the cell microinjection for technical presentation as the zebrafish is the one of the most well established research models in biotechnology and drug development A System setup The system as shown in Fig 1 and Fig 2 employs a motorized inverted microscopy Ml 11 Mshot and two cameras for imaging One camera camera1 a cellphone camera MD50 Mshot in this preliminary testing setup is directly connected to the microscope to capture images in the microscopic FOV The other one camera2 HD98 Gucee is placed on the top side of the system to provide the global macro view image Images are acquired through 4X objects lens with 9 5 V lightening voltage and processed by a 3 6 GHz host computer A pair of 3 DOF micromanipulators CFT 8301D2 are utilized to control the position of the microgripper and the micropipette and a motorized XY stage FL35ST28 0504BF12 is used to hold and move a petri dish containing culture medium and zebrafish embryos Figure 2 Picture of the system A magnified view of the micro operation workspace is shown in the blue circle Microgripper and micropipette are the main tools in the system as shown in the picture B Automation and Problem Definition The operations to be automated are illustrated in Fig 1 The yellow arrow represents the first operation to be automated moving the injection tools and cells to the microscopic FOV The red arrow represents the second operation moving down the gripper from the air outside the culture medium and then clamping a cell in the liquid environment for injection At the initial setup of the system the injection tools and cells are not usually located within the FOV of microscope as shown in Fig 3 a Moving the injection tools and cells into the FOV of the microscope from the macro FOV is a very time consuming process The culture media droplets where the cells are located are located right above the microscope objective lens It is difficult if not impossible to accurately obtain the distance among the occluded lens the injection tools and cells from the macroscopic view Therefore technician needs to manually move the injection tools and cells to the microscopic FOV from the macro FOV The macro micro conversion strategy is proposed in this paper to make full use of the complementarity of images taken by two cameras set at different positions so that the location information acquisition of the injection tool and cells can be achieved in a large dynamic spatial range as shown in Fig 1 One camera camera2 is used to capture a large range of imaging information throughout the workspace from a global perspective Figure 3 The circular photo was taken by the camera directly connected to the microscope a The microgripper and cells are not in the micro FOV at the initial setup of the system and there is nothing in the photo under micro FOV b A large area of black is present in the area where the viscous effect occurs when the gripper tip touches the culture nedia surface 653 The other camera camera1 is connected to the microscope to obtain the positional imaging information of the target in a small range from the micro scale perspective The original position and target positions of the injection tool are first obtained from the imaging by the camera2 and the tools are quickly moved to the target position this process will be described in detail in the following sections Then through processing and analyzing the photos taken by camera1 the injection tools are moved to the microscopic field along the serpentine path based on the proposed grid detection and positioning algorithm Moving a microgripper or an injection tool from the air into the liquid environment to immobilize or inject the embryonic cells needs to handle two problems The first problem is the viscous effect When the jaws of the microgripper come into contact with the water surface a large black area appears in the contact portion in the image resulting in the loss of image information as shown in the Fig 3 b This is a challenging problem for robotic operations based on visual feedback There is currently no research effort on handling this problem The second problem is to determine if the jaw of microgripper and the embryo are located in the same plane When the object in the FOV of the microscope moves only in the direction of the optical axis of the microscope the auto focusing algorithm can accurately obtain the depth information based on the sharpness of the image The autofocusing algorithms traditionally employed are based on fixed windows that assume that the positions of objects within the FOV do not change in the XY plane However based on experimental observations when the jaws of the microgripper immobilize the cells in the liquid the position of the jaws and the cells in the FOV always changes slightly as the jaws move downward This change occurs for the following reasons 1 The vibration of the clamp takes place during the movement 2 When the jaw moves in the liquid it will drive the cells to move at the same time To solve the two problems the phased control strategy and the active window based auto focusing algorithm are proposed The phased control strategy is used to move the jaws down into the culture media droplets but above the orthographic projection of the cells along the Z axis The active window based auto focusing algorithm is employed to adjust and move the jaw of microgripper into the same plane as the central horizontal plane of the cell III MOVE THE INJECTION TOOLS AND CELLS INTO THE MICROSCOPE FOV A The control flow As shown in the Fig 2 the two cameras are placed in different positions for obtaining macro and microscopic views of the system The injection tools a micropipette a gripper and cells are not in the microscopic FOV at system initial state The macro micro conversion strategy is proposed to automate the process and consists of the following steps 1 determine the relative position of the jaw of microgripper and the microscope lens under the macro FOV An indirect position conversion method is used to obtain the position of the jaw of the microgripper and a template matching algorithm is used to obtain the position of the microscope objective lens After obtaining the relative position of the two targets the microgripper is moved to the lower left corner of the square Figure 4 Schematic diagram of the microgripper The origin black point is set at the upper left corner of the markers Figure 5 Process of obtaining the position of the jaws a The red box is t he marker placed b The markers are separated from the image c Morpho logical close operation d The result of algorithm frame where the objective lens is located 2 start transferring to the processing and analysis of the image taken by the camera1 The microgripper is moved along a serpentine path into the micro FOV and its position is obtained using a grid detection and positioning algorithm the principles of the above methods are presented in next subsection Similarly the other micromanipulation tools micropipette and materials cells can also be automatically moved to FOV by employing the proposed macro micro conversion strategy B Localization algorithm under macro FOV and grid detection positioning algorithm The method of indirect position conversion is used to determine the position of the jaw of microgripper Because in the macro FOV the jaw is extremely small and is directly exposed by the microscope light source so that it is difficult to accurately position it We placed two markers at the driver behind the jaws to aid positioning The markers are indicated in red for easy identification as shown in Fig 4 and Fig 5 a The image taken by the camera2 is analyzed The RGB value of the marker s color is different from the values in other locations of the image Therefore the markers can be segmented from the background as shown in Fig 5 b A morphological opening operation is performed on the image to remove small dots in the image and the two largest contours in the processed image are the markers as shown in Fig 5 c The relative positional relationship between the markers and the jaw is established to obtain the position of the jaw as shown in Fig 4 The position of the jaw is calculated using the position of the marker as 654 1 y y y 2 Where is the center of gripper x0 y0 is the upper left corner of the square where the marker is located a and b are scale factor a is 2 3 and b is 2 w and h are the width and height of the markers The position of the jaws is obtained from the position of the markers as shown in Fig 5 d The normalized correlation coefficient template matching algorithm is adopted to determine the position of the micro scope objective lens under macro FOV This algorithm compensates the intensity variances among different images Therefore it is more suitable for situations where the reflection caused by the direct light source of the microscope is severe The definition of this algorithm as follow 3 Where and T x y is the pixel intensity of the point at the x y in the template picture I x y is the pixel intensity of the point at the x y in the image being searched w and h represent the width and height of the image and R represents the degree of similarity between the area to be searched and the template image The higher the similarity between the template image and the image to be searched the closer the value of R is to 1 The result of positioning microscope objective lens is shown in Fig 6 The jaw of microgripper will move from point A to point B The area of the microscope lens template image used by the system for the template matching algorithm is larger than the actual lens because after moving the cells into the microscope field the lens is easily blocked by the culture media in which the cells are located resulting in the loss of positioning information of lens In order to improve the robustness of the template matching algorithm a larger template image is adopted It improves the success rate of positioning although the position at which the microgripper reaches is farther than the microscopic FOV and the path of movement is longer The grid detection and positioning algorithm is used to determine where the microgripper enters the micro FOV In this algorithm the microgripper will move in a serpentine manner until it enters the micro FOV as shown in Fig 7 The image is divided into detection grids as shown in Fig 8 a In this project the image is divided into 12 areas using five verti cal lines and four horizontal lines and each line is 20 pixels wide The pixel intensity of the grid lines in the newly acquir ed image is compared with those on the previous frame for every moving step of the microgripper The degree of change in pixel intensity is compa