机器人资料-论文-050-Visual Tracking and Servoing System Design for Circling.pdf

visualtrackingandservoingsystemdesignforcircling atargetofanairvehiclesimulatedinvirtualreality cheng-minghuang1,jong-hannjean3,yu-shanchengl,andli-chenfu1,2 departmentofelectricalengineering . departmentofcomputer2scieern departmentofelectricalengineerngi andinformationengineering2 nationaltaiwanuniversitynationaltaiwanuniversityst.johnsandst.marysinstituteoftechnology taipei,taiwan,r.0.c.taipei,taiwan,r.0.c.tamsui,taipei,taiwan,r.0.c. .tw abstract-thispaperpresentsthedevelopmentofavisual trackingandvisualservosystemforareconnoiteringair vehicle.theairvehicleismountedwithacameraplatform underneath.itisproposedtoperformanaerialreconnoitering missionforcontinuouslyviewinganinterestingtarget.the visionsystemiscomposedofahybridvisualtrackingsystem andanimagebasedfuzzylogicvisualservosystem.the designedhybridvisualtrackingsystemcantrackarbitrary- shapedmovingobject.also,thefollowedvisualservosystem istocontroltheairvehiclehoveringthetargetautonomously. theoverallsystemissimulatedandexperimentedinavirtual realityenvironmenttovalidatetheresearchwork. indexterms-visualtracking,visualservo. i.introduction apparently,unmannedairvehicle(uav)hasgained increasingattentionrecently.itissmallandsafetocarryout variousdangerousmissionswithoutapilotaboard.theuav canbecontrolledbyoperatorsremotelyorcanperformsome specificfunctions1-3autonomously,suchasmotion planning,navigationcontrol,sensorcontrol,dataanalysis, communication,etc.theimagingsensorislikelytobethe mostcommonsensingdevicemountedonuav.itis generallydevisedfortwopurposes,namely,flyingguidance andinformationacquisition.thecameraplatformisjustlike theeyeoftheairvehiclesothattheflyingpathoftheair vehiclecanbeproperlyspecifiedortheautopilotnavigation 4,5canbeappliedthroughuseofsomevisualservo mechanism.theotheraspectofthistaskdomainisaboutthe imageprocessingandcomputervision,suchasfeature identification,visualtracking,datacompression,andvisual representation,visualservoing,etc6,7. inthispaper,wedesignanintegratedvisionsystem applyingvisualtrackingandthevisualservoingtechniquefor navigationoftheairvehicle.thevisualtrackingsub-systemis designedtoextracttheinterestingtargetpositionfromthe capturedimagesequence.weadoptthearbitrary-shaped visualtrackingmethodologyproposedinourrecentwork8. thistrackingmethodcombinesthecontourmatchingand templatematchingasahybridsearchfortheunderlying trackingalgorithm.thetrackingresultisthenfedbacktothe autopilotoftheairvehiclesothatthevisualservosub-system canberealized.sincetherealdynamicsoftheairvehicleis verycomplex,wewillreduceitfirstandapplythehumanlike fuzzylogiccontroltotheautopilot.finally,thedeveloped imagebasedfuzzylogicnavigationcontrolleracquires appealingperformanceandconsiderablerobustnessapparent fromseveralsimulations. inourautopilotcontroldesign,therearetwoscenarios. first,wefixthecameraplatformunderneaththeairvehicle, withoutanydegreeoffreedom,toviewtheground straightdown.next,werelaxthisrestrictionsothatthecamera platformcanhave2dof(pitchandyaw)movement,to functionlikethemilitarylight-of-sight(los)whichcanlock onthetarget.however,inbothscenarios,theuavalways needstoadjustitsposetocompletetheautonomous reconnaissance. ontheotherhand,virtualreality(vr)isakindof advancedtechnologywhichfacilitatesscientistsandengineers tosimulateandtoverifytheirworksinaninteractive applicationenvironment.uavexperimentsposereally challengingenvironmentsforvariousrelatedconduct researches16.owingtothiscondition,beforeperforming thatrealexperiment,weproposeanexperimentwhich integratesthevisualtrackingalgorithmandthevr presentationtechniquesothatauavequippedwitha vision systemcanbetestedinlab. thispaperisorganizedasfollows.thehybridvisual trackingalgorithmisdescribedinsection2,wherean arbitrary-shapedvisualtrackingalgorithmforextractionthe positionoftheinterestingtargetisprovided.insection3the visionbasedfuzzylogicautopilotcontrollerisproposedfor thevisualservosub-systemoftheairvehicle.section4and section5presentsseveralnumericalsimulationsandthe virtualrealityexperimenttovalidatethedevelopedsystem. finally,conclusionisgiveninsection6. ii.visualtrackingsub-system beforedesigningtheaerialhoveringnavigationcontrol, wehavetogettheinterestingtargetpositionfromthe camerassensedimage.inthissection,webrieflypresentthe hybridtrackingalgorithm8,whichcantrackanarbitrary- shapedobjectin3dspace.weutilizeittofindoutthetarget andtoperformvisualtrackingduringtheflying. a.contourmatching differentfromthetraditionalsnake-basedtracking algorithms9-11,“snake“isonlyusedtoextracttheoutline ofthetargetratherthantoperformtrackinginoursystem. sincesnake-basedtrackingalgorithmsshouldbe underthe assumptionofaslowlymovingobject,itwilllikelyfailin trackingtheobjectwithlargemovement.thissub-section presentsacontourmatchingmethodwhichmakesuseofthe extractedcontourmodelfortrackingofanarbitrary-shaped object. weadoptthesimplepoint-basedsnakescheme9-11and discretize10ittoreducethecomputationalcomplexityand thecomputationaltime.also,wehaveaddedanadditional term8describingthedistancebetweenthecentroidpoint andeachcontrolpointtospeeduptheconvergingrate.from thesnakedeformationinfig.1,wecanseethesnakecontour maygrowoutwardorinwardtoextracttheoutlineofthe humanhead.italsoshowsthatthismethodremainstowork evenwhenthereisrelativemotionorchangeoforientation betweentheobjectandthecamera. (a) (b) (c)(d) (e)(f9h) 01)0j)(k)(1) fig.1.thedeformationofthesnakecounter(dashedline)toextractthehead counterwithvariousposes. afterwehaveacquiredtheoutlineofthetargetobjectas thetemplate,wewillsearchthisobjectviacontourmatching intheedgeimageofthefollowingcapturedimageframe.we needtosumthetotalgradientvaluespixelbypixelintheedge imagealongapre-extractedsnakeoutline.oncethe summationovertheabovecontourisdone,wewillshiftthe centerofthecontourmodeltothenextpixelandthen computethesummationalongtheperimeteroftheoutline again.aftertraversingthroughallthesearchingarea,wecan obtainthelargestsummationwhichwillcorrespondtothe distributionofedgepixelswiththehighestsimilaritytothe objectscontour. thecontourmatchingmethoddoesnotlikesnake deformationwhichconsumesagreatdealofcomputational time.however,westillhavetoupdatethereferencedoutline bysnakedeformationonceafterseveralimageframes.the contourmatchingmethodalsohasalimitationagainstlow- contrastenvironment,justlikewhatthesnake-basedtracking algorithmsnormallyencounter. b.templatematching uptonow,thereareseveralkindsoftechniquedeveloped torealizethetemplatematchingmethod,suchasncc (normalizedcross-correlation),sad(sumofabsolute difference),andssd(sumofsquareddifference),etc.thisis themostgeneralmethodtosearchfortheregionmostsimilar tothereferencetemplate.templatematching,whichisbased onthegray-leveltextureinformationoftheimagefeature,is differentfromthecontourmatching.however,theformerwill takeuplongcomputationaltime.toavoidthat,weusethe mmx(multimediaextension)instructionsprovidedbythe technologyoftheintelscputoperformthesadoperation. thesimd(singleinstructionmultipledata)characteristicsof mmxinstructionscamprovidemorepowerfulcalculation capability,andhencewecanspeedupthetemplatematching searchdrastically. sincethetargetmaychangeitsposeintheimage sequenceduetorelativemovementbetweenthetargetandthe camera,wewillpre-establishthetemplatedatabasewith severalreferencedtemplatesacquiredfromvariousviews. oncetheinterestingtargetappears,the visualtrackingsub- systemcandetectitandlockonit,andthenthe visualservo autopilotwillbeturnedon. c.hybridvisualtrackingalgorithm asmentionedabove,thetemplatematchingbasedon textureisefficientinvisualtrackinginthelow-contrast environment,wherethecontourmatchingmayfail.butwhen thereissomeocclusiontothetarget,theacquiredimage textureofthetargetintheimagesequencewillbeaffected. templatematchingmethodmaylosetrackinginthissituation; however,contourmatchingmaystillworksolongasthe affectedshapeisstillconsiderablyrecognizableandisquite similartothereferencedoutline.ourhumanbeingsalso searchforanddistinguishaninterestingtargetinan environmentbyutilizingitstextureandshapeusually.hence, weintegratethetemplatematchingandthecontourmatching intoahybridtrackingmethodologytoovercometheforegoing drawbacks. assumethecontourmatchingscoreoftheimagespace position(ix,iy)issc(ix,iy),thetemplatematchingscoreis st(ix,iy),andthemixedscoreofthehybridtrackingmethod is 1s1 s(ix,iy)=asc(ix,iy)+8st(ix,iy)(1) wherea,8e0,1areweightingswitha+,=1.wewantto estimatethetargetposition(i$,i*)withthemaxvalueof st(ix,iy) ineachimageframe.thekeypointofthehybrid trackingmethodistoconstructanuserdefinedlook-uptable. thistableliststheweightingsaboutthecontourmatchingand templatematchingsearchresults.thetablemayneedtobe adjustedindifferentsituationorenvironment. eachcapturedimagesizefromccdcamerais320*240 pixels.themicroprocessorusedhereispentium41.5ghz with256mbram,andthematroximaginglibraryis appliedforsomebasicimageprocessing.thetotalprocessing timeofthevisualtrackingimageprocessingisabout30to40 ms. iii.visualservoautopilotcontrol inthissection,weassumetheimageprocessingforvisual trackingproposedinsectioniihasbeenperfectlyapplied. theinterestingtargetpositionhasbeenideallyextractedfrom theimage. a.problemformulation acameraplatformismountedunderneaththeairvehicle. ourgoalistokeeptheairvehiclereconnoiteringthe interestingobjectallthetimewhileflying.sincethecamera mayhavefinitefield-of-view,theairvehiclemusthavesome controlstrategytoadjusttheflyingtrajectoryautonomously. first,weformulatetheoverallproblemandthecoordinate relations12fortheaerialreconnaissanceasfollows.usethe pin-holemodelofthecamera,andlet(xc,yc, z,) bethe interestingobjectcoordinateasapointinthecamera3d space.also,thecameralensistheorigininthecameraspace. assumethecameraconstantfareknown,andthedepthzc canbeyieldfromthelaserrangefinderorgps.moreover,the depthzcalsocouldbeestimatedfromstereoimagesduring flying.accordingtotheperspectiveprojection,the correspondingobjectcoordinate(u,v)intheimageplaneare ixcyczcit=uzclfvzclczct . (2) thecameraplatformhas2dofmotionwithainpitch directionandi?inyawdirection.moreover,this2dof cameraplatformismountedunderneaththeairvehicle,and wecantransformthecameraspacecoordinate(xc,yc,zc) intobodycoordinate(xb,ybzb)oftheairvehicle lxbybzbt =rb(a,i3)lxc yczct+tb(3) withtherotationmatrixrb(a,8)andafixedtranslation amounttb.fromthegyroandgpssensorontheairvehicle, wecanknowtheorientationinformationrwandthecurrent positiontwintheworld.hence,theairvehiclecanbe describedintheworldcoordinateas xwywzwt=rw(gyro)xbybzb +tw(gps) .(4) fig.2.theoverallcoordinaterelationshipoftheaerialreconnaissance. here,weconsiderourairvehicleasaflyingblimp.the simplifiedkinematicblimpmodel17is dyb co cavsxiyb dtz, 9,. ,.r., 46w., .w.s;ge_f.,. mi= 4._._, s m_ fffih.bie .nnmp ,. (a)(b) fig.6.(a)theairvehiclemovingtrajectory(solidline)intheworld coordinatexwywplanewhentheobjectismoving(dashed trajectory). (b)themovingtrajectoryofthetargetviewedintheimage coordinatewhentheobjectismoving. assumethecameraplatforminthemovingrangecan keepthetargetinthecenteroftheimage(u,v-)=(0,0),and thevisualtrackingandvisualservocontrolofthemotorsis ideallycorrect.ifthemotorsislocatedontheboundary,the projectionofthetargetintheimageplanemaynotbekeptin thecenter.inordertoperformourreconnoiteringmission,we stillhavetoadjusttheflyingposeoftheairvehicle.however, theairvehicledoesnthavetoaffordtherotationvelocityas hardasthepreviousworkwhenthe2dofcameraplatform hasthevisualservotrackingability.lettheallowedmoving rangeofthemotorsis (180-a-)a(180+a)inpitchdirection,(6) (45-,6lo),6(45+,6lo)inyawdirection,(7) asshowninfig.7. yi a.x z. inpitchdirectimo inyawdirection fig.7.theshadedportionsindicatetheallowedmovingrangeofthemotors ofthe2dofcameraplatform. whenthevisualtrackingandvisualservoofthemotors canworkperfectly,i.e.themotorsareinthepermissive movingrangeandthetargetpositionintheimageis(0,0),we donthavetoadjusttheairplaneflyingpose(=0). otherwise,weutilizetheimagebasedfuzzylogiccontroller proposedinsectioniiitodesigntheautopilotnavigation. noticethatthedesiredtargetimageposition(ui*,v*)issetas (0,0)currently. finally,wesimulatethesituationwhenthe2dofcamera platformcanmoveinacertainrange.assumethemotorscan moveintherange150oa210and35.,855,i.e. a=30and,6=10.figure8isthesimulationresult.the goalisfulfilledintheimagedomainfig.8.(b),andthe movingtrajectoryofthetwomotorsinfig.8.(c)and(d)are bothboundedintheallowedrange.sincethecameraplatform hasthevisualtrackingandvisualservoabilitymentionedin sectionii,theairvehicleflyingtrajectorydoesnthavetobe hardrestrictedaroundthedesiredcircleinworld xvyv plane. fromfig.8.(a),wecanseethattheairvehiclehovering trajectoryislikeapentagon.thereasonisthatwedefinesthe motorinpitchdirectioncanmovein2a=60range,and thisrangeofangleisapproximatetotheangelofapentagon (360/5=72).ontheotherhand,themovingrangeinpitch direction/%mayaffectthehoveringdistancefarornear betweenthetargetandairvehicle. a h . .=w (degree)(a) (degree) (d) ,m1!1a,4- ua ar (b) fig.8.(a)theairvehiclemovingtrajectoryintheworldcoordinate (sec.) (c) jc. xwywplanewhenthecameraplatformcanmove. (b)themovingtrajectoryofthetargetviewedintheimage coordinatewhenthecameraplatformcanmove. (c)the movingtrajectoryoftheangleainpitchdirection. (d)themovingtrajectoryoftheanglej8inyawdirection. v.experimentinvirtualreality thescenarioisanairvehicleflyinginthevirtualreality withavirtual2dofcameraplatformunderneath,andthe targetisfixedinthevirtualgroundcoveredwithrealaerial image.thevisualtrackeroftheairvehicleinthevirtualworld willbereplacedbythecameramountedonamotionplatform intherealworld,andthevrimagewillbepresentedfromthe viewpointoftheairvehicleunderneath.hence,thevisual trackingsystemprocessestheaerialimageintherealworld, andthentheresultisprovidedfortheimagebasedfuzzy controller.theairvehicleadjustsitsposeandmotioninvr, andthecorrespondingcapturedaerialimagewillbepresented again.theexperimentistestedintheclosed-loopsothatthe visualservosystemcansuccessfullyfulfillitsendowed functions. figure9showstheprojectedimagefromtheairvehicle underneathintherealworld,andwecanseetheperformance ofthehoveringreconnaissance. je)(i)g9)(n) fig.9.theprojectedvrscenefromtheairvehicleunderneath.thesmall rectangleisvisualtrackingcameraviewedregion.thecross-markis thesearchedtarget.thedrawnarrowmeanstheairvehicleflying directionthatiscorrectlycirclingthetarget. vi.conclusion inthispaper,weproposeanintegratedvisionsystem designincludingthevisualtrackingsub-systemandvisual servosub-system.thevisualtrackingsub-systemisproposed forextractionofaninterestingimagefeature.itutilizesthe hybridtrackingalgorithmtoextractandtotrackanarbitrary- shapedobjectinsequenceofimageframeswiththe processingintervalsaboutreal-time(30-40ms.).ithasbeen examinedinseveraltrackingscenarioandhaswide applications.thevisualservosub-systemisappliedto autopilotnavigationoftheairvehicle.theimagebasedfuzzy logiccontrollerisdevelopedforcirclinganinterestingtarget. wehaveverifiedtheproposedfuzzycontrollerwithseveral differentsimulationstoshowitswellperformanceandreliable stability.furthermore,wehaveexperimentedthecasewith the2dofcameraplatforminthevirtualreality.hence,the aerialreconnaissancecanbeachieved withtheintegrated visionsystem. althoughweonlyconsideredtheairvehicleasa simplifiedflyingblips,theproposedimagebasedfuzzylogic controllerstillcanbeappliedtomorecomplexcasesandbe implementedintherealflyingcontrolwithsome modifications.thesimulationaboutacompleteaircraftmodel isunderconstruction.wewillutilizetheimageinformationto controlthethrust,aileron,elevator,andrudderinorderto overcomethehardersituation.ourfutureworkwillalso implementandembedthewholevisionsystemontoauav andfurtherinvestigateitsapplications. references 1r.schroer,“uavs:thefuture,“ieeeaerospaceandelectronic systemsmagazine,vol. 18,pp.61-63,july2003. 2r.w.beard,t.w.mclain,m.a.goodrich,ande.p.anderson, “coordinatedtargetassignmentandinterceptforunmannedair vehicles,“ieeetransactionsonroboticsandautomation,vol. 18,no. 6,2002. 3i.k.nikolos,k.p.valavanis,n.c.tsourveloudis,a.n.kostaras, “evolutionaryalgorithmbasedoffline/onlinepathplannerforuav navigation,“ieeetransactionsonsystems,man,andcybernetics- partb:cybernetics,vol.33,no.6,2003. 4 j.e.kortelingandw.vanderborg,“partialcameraautomationinan unmannedairvehicle,“ieeetransactionsonsystems,man,and cybernetics-parta:systemsandhumans,vol.27,no.2,1997. 5h.zhangandj.p.ostrowski,“visualservoingwithdynamics:control ofanunmannedblimp,“ieeeinterna