IROS2019国际学术会议论文集 0599

Abstract The research on rare cells makes a significant contribution to biology research and medical treatment for the application of diagnostic operation as well as prognoses treatment Therefore sortingthemfromheterogeneous mixtures is crucial and valuable Traditional cell sorting methods featured with poor purity and recovery rate as well as limited flexibility which are not ideal approaches for rare type In this paper we proposed a cell screening method based on automatedmicroroboticaspiration and placementstrategy under fluorescence microscope An innovative autofocusing visualfeedback AVF methodisproposedforprecise three dimensional 3D locating of target cells For depth detection multiple depth from defocus MDFD method is adopted to solve symmetry problem and attain an average accuracy of 97 07 For planar locating Markov random field MRF based locating method is utilized to separate and locate the overlapped cells The end actuator locating and real time tracking are performed relying on normalized cross correlation NCC method Experiential results show that our system collects rare cells 100 cells ml 1 at a speed of 5 cells min 1with 90 purity and 75 recovery rate which is valuable for biological and medical application I INTRODUCTION Rare cells refer to low abundance cells with concentration of less than 1 000 cells ml 1in fluid 1 Researches on precisely and efficiently sorting specified number of cells from complex and heterogeneous mixtures are critical in biology and medicine In actual diagnostic operation cell sorting usually serves as the preparatory work For example breast prostate ovarian colon and other cancers can be prognosed by concentration of rare circulating tumor cells CTC s form peripheral blood of cancer patient 23 Furthermore rapidandaccuratecellsortingmakes customized medicine treatment a reality 4 Various methods have been applied for sorting cells from different complex and heterogeneous mixtures Conventional cells sorting methods such as density gradient centrifugation DGC 5 andimmunomagnetic assisted cellsorting IMACS allow the high speed sorting with high throughput volume 6 However the consequent disadvantages are obvious such as poor purity and recovery Sorted cells containing impurities areunaccommodatedforprognosticandmedicaluse Afterwards several modified methods were proposed such as fluorescence activatedcellsorting FACS 7 and magnetically activated cell separation MACS 8 As the first commercial cell sorter FACS was invented in 1969 and Research supported by the National Natural Science Foundation of China under grants 61520106011 and 61603044 All authors are with the Intelligent Robotics Institute School of Mechatronical Engineering Beijing Institute of Technology 5 South Zhongguancun Street Haidian District 100081 Beijing CHINA e mail wanghuaping now has become the benchmark of the filed after continuous improvement 9 FACS has a high sorting speed up to 50 000 cells per second and can ensure purity of the sample after extraction 10 Whereas its high processing costs and high operation pressures reduce the viability of cells In recent years cell sorting methods combing microfluidic with dielectrophoresis reduced the equipment size simplified the sorting steps and reduced the processing costs However when it is applied to rare cells sorting i e extracting only a few positive samples from thousands of negative samples the limitations are obvious such as poor flexibility and cost of effectiveness Bio micromanipulationasastraightforwardwayto perform cell manipulations such as injection 11 and aspiration 12 makes rare cells sorting possible Generally bio micromanipulationsisperformedmanuallyor semi automatically with low operation repeatability and low success rate 13 To solve these problems micro visual servo control is necessary which can highly increase accuracy and automation of the manipulation Researchers developed a computervision basedsortertoautomaticallyisolate individual cells 14 However without the depth of targets visual servo system was only applicable in planar or thin layer of cell suspension To obtain depth dual camera was equipped without enough depth of field DOF most microscopes have a shallow DOF To solve this constraint microrobotic manipulations attracted considerable attentions in recent year By integrating high efficiency visual algorithms the robotic system can successfully obtain cell s three dimensional 3D position information and fulfill subsequently automated cell sorting Wang et al demonstrated an automated robot assisted cell sorting methodology integrated with visual close loop feedback under microscope 15 However low contrast between cells and background increases the difficulty of detection and the overlap of multiple cells reduces the precisionofthelocating Tosolvetheseproblems fluorescence observation in the fluorescence microscopy was presented for the clear superiority over bright field It enhances the contrast between cells and background and targets are easily screened out of distractions 16 In addition fluorochrome is non invasive and different live cells can be labeled by different fluorescent dyes for identification 17 However the time of fluorescence observation is limited because of fluorescence quenching 18 Therefore fast visual processing algorithms are highly desirable In this paper we propose an autofocusing visual feedback AVF method for automated sorting of rare cells in fluorescence microscopy The automated system consists of a fluorescence microscope and a three axis translation robot with a micropipette as end actuator for sorting cells by aspiration AVF algorithm is integrated into automated system which consists of depth detection and planar locating The Automated Sorting of Rare Cells Based onAutofocusing Visual Feedback in Fluorescence Microscopy Kailun Bai Huaping Wang QingShi Zhiqiang Zheng JuanCui TaoSun QiangHuang Fellow IEEE Paolo Dario Fellow IEEE Toshio Fukuda Life Fellow IEEE 2019 IEEE RSJ International Conference on Intelligent Robots and Systems IROS Macau China November 4 8 2019 978 1 7281 4003 2 19 31 00 2019 IEEE1567 depth detection is achieved by the improved multiple depth from defocus MDFD algorithm derives from traditional depth from defocus DFD The planar locating relies on markov random field MRF based segmentation algorithm and normalized cross correlation NCC algorithm Assisted by AVF real time 3D locating of micropipette and cells are achieved Through the movement and aspiration of the micropipette accurate and rapid sort of rare cells from heterogeneousmixturesisperformed Experiments demonstrate this system has high efficiency in sorting cells with high purity and recovery rate which is meaningful for biological and medical research II OVERVIEW OF ROBOTIC SYSTEM A System framework As shown in Fig 1 the microrobotic manipulation system consists of a linear translation stage M 461 XYZ Newport Inc and three micro stepping motors NSA 12 Newport Inc with high accuracy resolution 0 2m and wide range of motion 11mmforeachaxis Aglassmicropipette B100 50 10 Sutter Instrument Inc is heat pulled by micropipette puller PC 10 Narishige Inc with an inner diameter of 19m The micropipette is fixed on the end actuator connecting to microinjection pump Legato 111 KD Scientific Inc through plastic tube An inverted optical microscope OM IX83 Olympus Inc is connected to a CCD camera DP22 Olympus Inc with maximum 2 76 megapixels The lamp U HGLGPS Olympus Inc and filter block U FNU FNB FNG Olympus Inc ensure multi colored fluorescence observation NIH 3T3 cells are chosen as sample since the stable activity and morphology in vitro culture A motorized X Y translational stage ProScan Prior Scientific Inc can change field of view FOV rapidly Cells are marked by two fluorescent dyes DiO and DiI and will give out different colors of fluorescent light Main computer configured with CPU Core i7 Intel Inc and GPU TITAN X NVIDIA Inc is utilized for visual process and automated control B Overview of automated sorting process The whole process of automated sorting is a visual feedback is shown in Fig 2 Under the fluorescence microscope the fluorescent cells and micropipette are clear with high contrast The camera transmits the collected information to the main computer simultaneously In visual processing the AVF algorithm consists of four parts In preprocessing target cells stand out and impurities are filtered out Then MDFD algorithm is applied to detect depth of target cells In planar locating overlappedcellsarelocatedbyMRF based algorithm Besides NCC algorithm is used to locate and track themicropipette stip Maincomputerreceivesvisual feedback to control the robot movement and the aspiration of target cells Finally the microrobotic system can achieve rare cells sorting automatically and rapidly which saving plenty of labor costs and can make contribution to biological research and medical treatment III AUTOFOCUSING VISUAL FEEDBACK A MDFD for depth detection Depth from focus DFF depth from defocus DFD stereo vision and structure from motion SFM are the mainstream methods of depth detection 19 For stereopsis and SFM binocular camera and motion camera are equipped which is not feasible for microscope system due to the space limitation and the shallow DOF In contrast DFD and DFF are more suitable for microscope s monocular vision system and more robust than stereo vision 20 However in DFF depth is obtained by searching lots of focused images which is time consuming On the contrary DFD method simply analyses two defocused image to get depth improving the detection efficiency greatly However low accuracy and symmetry problem exist in traditional two image DFD Deriving from traditional DFD multiple depth from defocus MDFD method are presented with optimized performance by processing additional images The convex lens imaging model shown in Fig 3 a where dnis the object depth to the current IP When IP and FP coincide a point object is a focused dot keeping IP away from FP it becomes a defocused blur circle When cell rather than a Fig 1 Automated microrobotic manipulation system Fig 2 Schematic of automated sorting 1568 point object in the imaging model as shown in Fig 3 b blur diameter acts like a linear shift Adjusting IP1to IP2with a pre set depth interval d the cell blur diameter changes from D1 to D2 According to geometry we have 1 1 21 DD dd DD 1 where D is the focused diameter of cell This is explained by point spread function PSF h x y with two dimensional Gaussian 22 22 1 exp 22 xy h x y 2 where spread parameter is positive correlated to Dn As shown in Fig 3 c the observed image g x y is the result of the convolution between the focused image im x y and PSF g x yh x yim x y 3 where g x y shows the distribution curves of pixel brightness Two curves are crossed and the length between two intersections is focused diameter of cell D D1and D2can be calculated by their enclosed area with coordinate axis about 99 of the area contained Bring D D1 D2and d0into Eq 1 depth of object can be obtained However symmetrical problem exists As shown in Fig 3 b in symmetrical position IP1and IP 1 on either side of the plane objects have the same blur diameter D 1and D1 which cannot be distinguished by two image DFD method To solve this problem MDFD shift IP with the samedand obtains additional blur diameters D3 The algorithm screens out the wrong data by analysis and object optimized depth is obtained B MRF based segmentation In microscopic scene separating the overlapped cells can improve the precision of the planar locating As shown in Fig 4 a image segmentation is regarded as classification Similar pixels are marked with same labels to form sub regions MRF was introduced to image segmentation by Geman et al in 1984 The MRF based image segmentation has been widely applied in different macroscopic senses such as magnetic resonance imaging MRI scans 21 computed tomography CT images 22 and remote sensing images In MRF every pixel is not independent as shown in Fig 4 b they are relevant to their neighboring pixels and pixels with the same labels Thus labels of pixels can be optimized by iteration According to the maximum a posteriori estimation MAP we have argmax W WP W P S W 4 Fig 3 Theory of MDFD a Convex lens imaging model b Schematic of MDFD c Result of PSF Fig 4 MRF based segmentation a Label distribution in segmentation b Neighbouring system c Process of ICM algorithm 1569 where W is label set S is pixel set P W is prior probability and P S W is likelihood function P W describes the relationbetweenneighboringpixels Accordingto Hammersley Clifford theorem 23 we have 1 1 U W P Wz e 5 where z is normalized function and 1 U W is energy function P S W describes the relation of pixels under different labels which obeys Gaussian distribution As shown in Fig 3 the label of pixel is more likely in accordance with the label whose likelihood function has bigger function response Then we have 2 2 1 exp 22 rr r R r r s P S W 6 Therefore MAP problem can be expressed as 12 argmax W WU WU W S 7 where 2 U W S is another energy function minimum energy corresponds to optimal label Then iterative condition mode ICM algorithm is introduced to achieve image segmentation The whole process is shown in Fig 4 c First setting the number of class and initialize all labels randomly Then updating label set by MAP optimization iteratively untilconvergenceconditionissatisfied Convergence condition is defined as 1kk EE 8 where k Eis current global energy Finally the MRF ICM based segmentation algorithm marked optimal labels for each pixelaccordingtoMAP whichenablesmoothest segmentation and noise elimination IV EXPERIMENTAL RESULT A Experiment setup Setting DiI red dyed NIH 3T3 cells as rare targets with concentrationof100cells ml 1 with is mixedbyabundant DiO green dyed NIH 3T3 cells in a proportion of 1 100 The Micropipette is also pre dyed with DiI The Overall process of automatic microrobotic manipulation system is shown in Fig 5 In AVF the initial segmentation makes micropipette and target cells stand out by setting color threshold MDFD detects the depth of target cells Moving micropipette to the same X Y plane of cells and focusing The overlapped cells are distinguished through MRF based segmentation and located through Hough circle algorithm The micropipette tip is located and tracked using normalized cross correlation NCC method Then the robot executes manipulation until all target cells are sorted out in the FOV X Y translation stage will bring a new group of cells into FOV and another round of process well be carried out until the collected cells are enough B Depth detection with MDFD As shown in Fig 6 recording the initial depth image Then setting depth interval10 dmto attain two additional images Drawing pixel brightness distribution curves and the wrong data depth 2 can be screened out Bring parameters into Eq 1 to get the depth 1 5 23dm and focus image However the actual value of depth1 is 6 00m where deviation exists in MDFD To evaluate the deviation we introduce a formula Fig 5 Overall process of automated cell sorting Fig 6 Experiment of MDFD to obtain cell depth 1570 100 dd dev Dd 9 where d is the experimental depth value d is the actual depth value and D is introduced as correction factor In this way the dev of depth in Fig 6 is 3 7 Noticed that the depth interval d and the number of additional images n are two crucial factors to determine the accuracy and efficiency of MDFD algorithm to research the optimal combination we set two experiments First for depth interval experiment setting 6 groups of depthinterval values d 2 m 4 6 8 10 12 For eachd setting 6 different initial depths For each interval testing 10 cells The total amount of data is 360 The result is shown in Fig 7 with the increasing of d the average deviation increases first and then decreases When d0 8m there is a minimum value 3 29 Second for additional images experiment setting 5 groups of additional image numbers n 2 3 4 5 6 at the same time keeping d 8 m Other operations are the same as the first experiment The result is shown in Fig 8 the average deviation decreases as the number increases It showing that more additional number of images can improve precision However according to the gradient when the number is greater than 3 the precision increasing gently Meanwhile excessive images will bring extra time consumption Finally we choose d 8m and n 3 as the optimal parameter with average deviation of 2 93 The whole process takes 920ms on average C Planar locating and tracking For cells locating under microscopic scene they are likely to get overlapped because of surface adhesion Thus separating them into individuals is crucial As shown in Fig 9 Fig 7 Depth interval experiment Fig 8 Additional images experiment Fig 9 MRF based cell segmentation and locating Fig 10 Micropipette locating and tracking Fig 11 Automated rare cells sorting 1571 two target cells are overlapped and their edges are sticking together The MRF based segmentation algorithm classifies the original image into 3 parts where gray region represents the overlapped edges Fig 9b Then Otsu adaptive threshold is used to filter theoverlapped edges with two separated cells retained Fig 9c Morphological close operation is used to fill small hole of the cell Fig 9d and morphological open operation is used to create smooth edges Fig 9e Hough circle transformation is used to match the round edges with circles Fig 9f The locations of cells are marked by centers of circles For micropipette locating NCC method is used as shown in Fig 10 The matching template at the tip of the micropipette is generated after installation The template searches on the image by single pixel to get normalized cross correlation window The tip of micropipette can be located by the best match Once the tip is located its moving will be tracked by searching area of last known location of previous frame which is time