外文翻译原文-远程视频监控

ASystemforVideoSurveillanceandMonitoringRobertT.Collins,AlanJ.LiptonandTakeoKanadeRoboticsInstitute,CarnegieMellonUniversity,Pittsburgh,PAE-MAIL:frcollins,ajl,PHONE:412-268-1450HOMEPAGE:/vsamAbstractTheRoboticsInstituteatCarnegieMellonUniversity(CMU)andtheSarnoffCorporationaredevelopingasystemforautonomousVideoSurveillanceandMonitoring.Thetechnicalobjectiveistousemultiple,cooperativevideosensorstoprovidecontinuouscoverageofpeopleandvehiclesinclutteredenvironments.Thispaperpresentsanoverviewofthesystemandsignificantresultsachievedtodate.1IntroductionTheDARPAImageUnderstanding(IU)programisfundingbasicresearchintheareaofVideoSurveillanceandMonitoring(VSAM)toprovidebattlefieldawareness.ThethrustofCMUsVSAMresearchistodevelopauto-matedvideounderstandingalgorithmsthatallowanetworkofactivevideosensorstoautomaticallymonitorobjectsandeventswithinacomplex,urbanenvironment.Wehavedevelopedvideounderstandingtechnologythatcanau-tomaticallydetectandtrackmultiplepeopleandvehicleswithinclutteredscenes,andtomonitortheiractivitiesoverlongperiodsoftime.Humanandvehicletargetsareseamlesslytrackedthroughtheenvironmentusinganetworkofactivesensorstocooperativelytracktargetsoverareasthatcannotbeviewedcontinuouslybyasinglesensoralone.Eachsensortransmitssymboliceventsandrepresentativeimagerybacktoacentraloperatorcontrolstation,whichprovidesavisualsummaryofactivitiesdetectedoverabroadarea.Theuserinteractswiththesystemusinganintuitivemap-basedinterface.Forexample,theusercanspecifythatobjectsenteringaregionofinterestshouldtriggeranalert,relievingtheburdenofcontinuallywatchingthatarea.Thesystemautomaticallyallocatessensorstooptimizesystemperformancewhilefulfillingusercommands.Althoughdevelopedwithinacontextofprovidingbattlefieldawareness,webelievethistechnologyhasgreatpoten-tialforapplicationsinremotemonitoringofnuclearfacilities.Sampletasksthatcouldbeautomatedareverificationthatroutinemaintainanceactivitiesarebeingperformedaccordingtoschedule,loggingandtrackingvisitorsandpersonnelastheyenterandmovethroughthesite,andprovidingsecurityagainstunauthorizedintrusion.Otherapplicationsinmilitaryandlawenforcementscenariosincludeprovidingperimetersecurityfortroops,monitoringpeacetreatiesorrefugeemovementsusingunmannedairvehicles,providingsecurityforembassiesorairports,andstakingoutsuspecteddrugorterroristhide-outsbycollectingtime-stampedpicturesofeveryoneenteringandexitingthebuilding.ThefollowingsectionspresentanoverviewofvideosurveillancealgorithmsdevelopedatCMUoverthelasttwoyears(Section2)andtheirincorporationintoaprototypesystemforremotesurveillanceandmonitoring(Section3).ThisworkisfundedbytheDARPAIUprogramunderVSAMcontractnumberDAAB07-97-C-J031.1Collins-12VideoUnderstandingTechnologiesKeepingtrackofpeople,vehicles,andtheirinteractionsinacomplexenvironmentisadifficulttask.TheroleofVSAMvideounderstandingtechnologyinachievingthisgoalistoautomatically“parse”peopleandvehiclesfromrawvideo,determinetheirgeolocations,andautomaticallyinsertthemintoadynamicscenevisualization.Wehavedevelopedrobustroutinesfordetectingmovingobjects(Section2.1)andtrackingthemthroughavideosequence(Section2.2)usingacombinationoftemporaldifferencingandtemplatetracking.Detectedobjectsareclassifiedintosemanticcategoriessuchashuman,humangroup,car,andtruckusingshapeandcoloranalysis,andtheselabelsareusedtoimprovetrackingusingtemporalconsistencyconstraints(Section2.3).Furtherclassificationofhumanactivity,suchaswalkingandrunning,hasalsobeenachieved(Section2.4).Geolocationsoflabeledentitiesaredeterminedfromtheirimagecoordinatesusingeitherwide-baselinestereofromtwoormoreoverlappingcameraviews,orintersectionofviewingrayswithaterrainmodelfrommonocularviews(Section2.5).Thecomputedgeolocationsareusedtoprovidehigher-leveltrackingcapabilities,suchastaskingmultiplesensorswithvariablepan,tiltandzoomtocooperativelytrackanobjectthroughthescene(Section2.6).2.1MovingTargetDetectionTheinitialstageofthesurveillanceproblemistheextractionofmovingtargetsfromavideostream.Therearethreeconventionalapproachestoautomatedmovingtargetdetection:temporaldifferencing(two-frameorthree-frame)Andersonetal.,1985;backgroundsubtractionHaritaogluetal.,1998,Wrenetal.,1997;andopticalflow(seeBarronetal.,1994foranexcellentdiscussion).Temporaldifferencingisveryadaptivetodynamicenvironments,butgenerallydoesapoorjobofextractingallrelevantfeaturepixels.Backgroundsubtractionprovidesthemostcompletefeaturedata,butisextremelysensitivetodynamicscenechangesduetolightingandextraneousevents.Opticalflowcanbeusedtodetectindependentlymovingtargetsinthepresenceofcameramotion;however,mostopticalflowcomputationmethodsareverycomplexandareinapplicabletoreal-timealgorithmswithoutspecializedhardware.TheapproachpresentedhereissimilartothattakeninGrimsonandViola,1997,andisanattempttomakebackgroundsubtractionmorerobusttoenvironmentaldynamism.Thekeyideaistomaintainanevolvingstatisticalmodelofthebackgroundtoprovideamechanismthatadaptstoslowchangesintheenvironment.Foreachpixelvaluepninthenthframe,arunningaveragepnandaformofstandarddeviationpnaremaintainedbytemporalfiltering,implementedas:pn+1=pn+1+(1?)pnn+1=jpn+1?pn+1j+(1?)n(1)where=f,fistheframerateandisatimeconstantspecifyinghowfast(responsive)thebackgroundmodelshouldbetointensitychanges.theinfluenceofoldobservationsdecaysexponentiallyovertime,andthusthebackgroundgraduallyadaptstoreflectcurrentenvironmentalconditions.Ifapixelhasavaluewhichismorethan2frompn,thenitisconsideredaforegroundpixel.Atthispointamultiplehypothesisapproachisusedfordeterminingitsbehavior.Anewsetofstatistics(p0;0)isinitializedforthispixelandtheoriginalsetisremembered.If,aftertimet=3,thepixelvaluehasnotreturnedtoitsoriginalstatisticalvalue,thenewstatisticsarechosenasreplacementsfortheold.Foreground(moving)pixelsareaggregatedusingaconnectedcomponentapproachsothatindividualtarget“blobs”canbeextracted.Transientmovingobjectscauseshorttermchangestotheimagestreamthatarenotincludedinthebackgroundmodel,butarecontinuallytracked,whereasmorepermanentchangesare(afteratimeincrementof3haselapsed)absorbedintothebackground(seeFigure1).Themovingtargetdetectionalgorithmdescribedaboveispronetothreetypesoferror:incompleteextractionof2Collins-1(A)(B)Figure1:Exampleofmovingtargetdetectionbydynamicbackgroundsubtraction.Figure2:Targetpre-processing.Amovingtargetregionismorphologicallydilated(twice),erodedandthenitsborderisextracted.amovingobject;erroneousextractionofnon-movingpixels;andlegitimateextractionofillegitimatetargets(suchastreesblowinginthewind).Incompletetargetsarepartiallyreconstructedbyblobclusteringandmorphologicaldilation(Figure2).Erroneouslyextracted“noise”isremovedusingasizefilterwherebyblobsbelowacertaincriticalsizeareignored.Illegitimatetargetsmustberemovedbyothermeanssuchastemporalconsistencyanddomainknowledge.Thisisthepurviewofthetargettrackingalgorithm.2.2TargetTrackingTobegintobuildatemporalmodelofactivity,individualobjectsmustbetrackedovertime.Thefirststepinthisprocessistotaketheblobsgeneratedbymotiondetectionandmatchthembetweenframesofavideosequence.ManysystemsfortargettrackingarebasedonKalmanfilters.However,asIsardandBlakepointout,theseareoflimitedusebecausetheyarebasedonunimodalGaussiandensitiesthatcannotsupportsimultaneousalternativemotionhypothesesIsardandBlake,1996.IsardandBlakepresentanewstochasticalgorithmcalledCONDEN-SATIONthatdoeshandlealternativehypotheses.WorkontheproblemofmultipledataassociationinradartrackingcontextsisalsorelevantBar-ShalomandFortmann,1988.Weemployamuchsimplerapproachbasedonaframe-to-framematchingcostfunction.Arecordofeachblobiskeptwiththefollowinginformation:imagetrajectory(positionp(t)andvelocityv(t)asfunctionsoftime)oftheobjectcentroid,blob“appearance”intheformofanimagetemplate,3Collins-1blobsizesinpixels,colorhistogramhoftheblob.ThepositionandvelocityofeachblobTiisdeterminedfromthelasttimesteptlastandusedtopredictanewimagepositionatthecurrenttimetnow:pi(tnow)pi(tlast)+vi(tlast)(tnow?tlast)(2)UsingthisinformationamatchingcostisdeterminedbetweenaknowntargetTiandacandidatemovingblobRjC(Ti;Rj)=f(jpi?pjj;jsi?sjj;jhi?hjj):(3)Targetsthatare“closeenough”incostspaceareconsideredtobepotentialmatches.Tolendmorerobustnesstochangesinappearanceandocclusions,thefulltrackingalgorithmusesacombinationofcostandadaptivetemplatematching,asdescribedindetailinLiptonetal.,1998.RecentresultsfromthesystemareshowninFigure3.Figure3:Recentresultsofmovingentitydetectionandtrackingshowingdetectedobjectsandtrajectoriesoverlaidonoriginalvideoimagery.Notethattrackingpersistsevenwhentargetsaretemporarilyoccludedormotionless.2.3TargetClassificationTheultimategoaloftheVSAMeffortistobeabletoidentifyindividualentities(suchasthe“FedExtruck”,the“4:15pmbustoOakland”and“FredSmith”)anddeterminewhattheyaredoing.Asafirststep,entitiesareclassifiedintospecificclassgroupingssuchas“humans”and“vehicles”.Currently,weareexperimentingwithaneuralnetworkapproach(Figure4).Theneuralnetworkisastandardthree-layernetworkwhichusesabackpropagationalgorithmforhierarchicallearning.Inputstothenetworkare4Collins-1amixtureofimage-basedandscene-basedentityparameters:dispersedness(perimeter2/area(pixels);imagearea(pixels);aspectratio(height/width);andcamerazoomfactor.Usingasetofmotionregionsautomaticallyextractedbutlabeledbyhand,thenetworkistrainedtooutputoneofthreeclasses:human;vehicle;orhumangroup(twoormorehumanswalkingclosetogether).Whenteachingthenetworkthataninputentityisahuman,alloutputsaresetto0.0exceptfor“human”,whichissetto1.0.Otherclassesaretrainedsimilarly.Iftheinputdoesnotfitanyoftheclasses,suchasatreeblowinginthewind,alloutputsaresetto0.0.InputLayer(4)HiddenLayer(16)OutputLayer(3)TeachpatternAreaDispersednessVehicleSinglehuman1.00.0Multiplehuman0.0AspectratioRejecttargetsinglehuman0.00.00.0ZoommagnificationTargetCameraFigure4:Neuralnetworkapproachtotargetclassification.Resultsfromtheneuralnetworkareinterpretedasfollows:if(outputTHRESHOLD)classification=maximumNNoutputelseclassification=REJECTTheresultsforthisclassificationschemearesummarizedinTable1.Thisclassificationapproachiseffectiveforsingleimages.However,oneoftheadvantagesofvideoisitstemporalcomponent.Toexploitthis,classificationisperformedoneveryentityateveryframeandtheresultsofclassificationarekeptinahistogramwiththeithbucketcontainingthenumberoftimestheobjectwasclassifiedasclassi.Ateachtimestep,theclasslabelthathasbeenoutputmostoftenforeachobjectischosenitsmostlikelyclassification.2.4ActivityAnalysisAfterclassifyinganobject,wewanttodeterminewhatitisdoing.Understandinghumanactivityisoneofthemostdifficultopenproblemsintheareaofautomatedvideosurveillance.DetectingandanalyzinghumanmotioninrealtimefromvideoimageryhasonlyrecentlybecomeviablewithalgorithmslikePfinderWrenetal.,1997andW4Haritaogluetal.,1998.Thesealgorithmsrepresentagoodfirststeptotheproblemofrecognizingandanalyzinghumans,buttheystillhavesomedrawbacks.Ingeneral,theyworkbydetectingfeatures(suchashands,feetandhead),trackingthem,andfittingthemtosomeapriorihumanmodelsuchasthecardboardmodelofJuetalJuetal.,1996.Thereforethehumansubjectmustdominatetheimageframesothattheindividualbodycomponentscanbereliablydetected.5Collins-1ClassSamples%CorrectlyClassifiedHuman43099.5Humangroup9688.5Vehicle50899.4Falsealarms4864.5Total108296.9Table1:Resultsforneuralnetworkclassificationalgorithm.Weusea“star”skeletonizationprocedureforanalyzingthemotionofhumansthatarerelativelysmallintheimage.DetailscanbefoundinFujiyoshiandLipton,1998.Thekeyideaisthatasimpleformofskeletonizationthatonlyextractsthebroadinternalmotionfeaturesofatargetcanbeemployedtoanalyzeitsmotion.Thismethodprovidesasimple,real-time,robustwayofdetectingextremalpointsontheboundaryofthetargettoproducea“star”skeleton.The“star”skeletonconsistsofthecentroidofanentityandallofthelocalextremalpointswhichcanberecoveredwhentraversingtheboundaryoftheentitysimage(Figure5a).0,enddistanceborderpositiondistancedistanceiDFTLPFInverseDFT0endabcdeabcdestarskeletonoftheshaped(i)d(i)centroid-+0xyx,yccl,lxy(a)(b)x,ycc(A)(B)Figure5:(A)Thestarskeletonisformedby“unwrapping”aregionboundaryasadistancefunctionfromthecentroid.Thisfunctionisthensmoothedandextremalpointsareextracted.(B)Determinationofskeletonfeaturesmeasuringgaitandposture.istheangletheleftmostlegmakeswiththevertical,andistheanglethetorsomakeswiththevertical.Usingonlymeasurementsbasedonthe“star”skeleton,itispossibletodeterminethegaitandpostureofamovinghumanbeing.Figure5bshowshowtwoanglesnandnareextractedfromtheskeleton.Thevaluenrepresentstheangleofthetorsowithrespecttovertical,whilenrepresentstheangleoftheleftmostleginthefigure.Figure6showsskeletonmotionfortypicalsequencesofwalkingandrunninghumans,alongwiththevaluesofnandn.Thesedatawereacquiredinreal-timefromavideostreamwithframerate8Hz.Comparingtheaveragevaluesninfigures6(e)-(f)showthatthepostureofarunningtargetcaneasilybedistinguishedfromthatofawalkingoneusingtheangleofthetorsosegmentasaguide.Also,thefrequencyofcyclicmotionofthelegsegmentsprovidescuestodistinguishingrunningfromwalking.6Collins-1frame111213141516171819200.125sec(a)skeletonmotionofawalkingperson12345678910(b)skeletonmotionofarunningperson1112131415161718192012345678910frameradframe-1-0.500.510510152025-1-0.500.510510152025rad(d)legangleofarunningperson(c)legangleofawalkingperson00.4051015202500.40510152025frameframerad|rad|(f)torsoangleofarunningperson(e)torsoangleofawalkingpersonFigure6:Skeletonmotionsequences.Clearly,theperiodicmotionofnprovidescuestothetargetsmotionasdoesthemeanvalueofn.2.5Model-basedGeolocationThevideounderstandingtechniquesdescribedsofarhaveoperatedpurelyinimagespace.Alargeleapintermsofdescriptivepowercanbemadebytransformingimageblobsandmeasurementsinto3Dscene-basedobjectsanddescriptors.Inparticular,determinationofobjectlocationinthesceneallowsustoinfertheproperspatialrelation-shipsbetweensetsofobjects,andbetweenobjectsandscenefeaturessuchasroadsandbuildings.Furthermore,webelievethekeytocoherentlyintegratingalargenumberoftargethypothesesfrommultiplewidely-spacedsensorsiscomputationoftargetspatialgeolocation.Inregionswheremultiplesensorviewpointsoverlap,objectlocationscanbedeterminedveryaccuratelybywide-baselinestereotriangulation.However,regionsofthescenethatcanbesimultaneouslyviewedbymultiplesensorsarelikelytobeasmallpercentageofthetotalareaofregardinrealoutdoorsurveillanceapplications,whereitisdesirabletomaximizecoverageofalargeareagivenfinitesensorresources.Determiningtargetlocationsfromasinglesensorrequiresdomainconstraints,inthiscasetheassumptionthattheobjectisincontactwiththeterrain.Thiscontactlocationisestimatedbypassingaviewingraythroughthebottomoftheobjectintheimageandintersectingitwithamodelrepresentingtheterrain(seeFigure7a).Sequencesoflocationestimatesovertimearethenassembledintoconsistentobjecttrajectories.Previoususesoftherayintersectiontechniqueforobjectlocalizationinsurveillanceresearchhavebeenrestrictedtosmallareasofplanarterrain,wheretherelationbetweenimagepixelsandterrainlocationsisasimple2Dho-mographyBradshawetal.,1997,FlinchbaughandBannon,1994,Kolleretal.,1993.Thishasthebenefitthatnocameracalibrationisrequiredtodeterminetheback-projectionofanimagepointontothesceneplane,providedthemappingsofatleastfourcoplanarscenepointsareknownbeforehand.However,largeoutdoorsceneareasmay7Collins-1Elev(X0+kU,Y0+kV)Z0+kW11X0,Y0,Z0X0121087469351213ProjectionX0,Y0Ray:(X0,Y0)+k(U,V)Ray:(X0,Y0,Z0)+k(U,V,W)VerticalX(A)(B)Figure7:(A)Estimatingobjectgeolocationsbyintersectingtargetviewingrayswithaterrainmodel.(B)ABresenham-liketraversalalgorithmdetermineswhichDEMcellcontainsthefirstintersectionofaviewingrayandtheterrain.containsignificantlyvariedterrain.Tohandlethissituation,weperformgeolocationusingrayintersectionwithafullterrainmodelprovided,forexample,byadigitalelevationmap(DEM).Givenacalibratedsensor,andanimagepixelcorrespondingtotheassumedcontactpointbetweenanobjectandtheterrain,aviewingray(x0+ku;y0+kv;z0+kw)isconstructed,where(x0;y0;z0)isthe3Dsensorlocation,(u;v;w)isaunitvectordesignatingthedirectionoftheviewingrayemanatingfromthesensor,andk0isanarbitrarydistance.Generalmethodsfordeterminingwhereaviewingrayfirstintersectsa3Dscene(forexample,raytracing)canbequiteinvolved.However,whenscenestructureisstoredasaDEM,asimplegeometrictraversalalgorithmsuggestsitself,basedonthewell-knownBresenhamalgorithmfordrawingdigitallinesegments.ConsidertheverticalprojectionoftheviewingrayontotheDEMgrid(seeFigure7b).Startingatthegridcell(x0;y0)containingthesensor,eachcell(x;y)thattheraypassesthroughisexaminedinturn,progressingoutward,untiltheelevationstoredinthatDEMcellexceedsthez-componentofthe3Dviewingrayatthatlocation.Thez-componentoftheviewrayatlocation(x;y)iscomputedaseitherz0+(x?x0)uworz0+(y?y0)vw(4)dependingonwhichdirectioncosine,uorv,islarger.ThisapproachtoviewingrayintersectionlocalizesobjectstoliewithintheboundariesofasingleDEMgridcell.Amoreprecisesub-celllocationestimatecanthenbeobtainedbyinterpolation.Ifmultipleintersectionswiththeterrainbeyondthefirstarerequired,thisalgorithmcanbeusedtogeneratetheminorderofincreasingdistancefromthesensor,outtosomecut-offdistance.SeeCollinsetal.,1998formoredetails.8Collins-12.6Multi-SensorCooperationInmostcomplexoutdoorscenes,itisimpossibleforasinglesensortomaintainitsviewofanobjectforlongperiodsoftime.Objectsbecomeoccludedbyenvironmentalfeaturessuchastreesandbuildings,andsensorshavelimitedeffectivefieldsofregard.Apromisingsolutiontothisproblemistouseanetworkofvideosensorstocooperativelytrackanobjectthroughthescene.Trackedobjectsarethenhanded-offbetweencamerastogreatlyextendthetotaleffectiveareaofsurveillancecoverage.Therehasbeenlittleworkdoneonautonomouslycoordinatingmultipleactivevideosensorstocooperativelytrackamovingtarget.OneapproachispresentedbyMatsuyamaforacontrolledindoorenvironmentwherefourcameraslockontoontoaparticularobjectmovingacrossthefloorMatsuyama,1998.Weapproachtheproblemmoregenerallybyusingtheobjects3Dgeolocationascomputedinthelastsectiontodeterminewhereeachsensorshouldlook.Thepan,tiltandzoomoftheclosestse