数字视频特效处理lec07-stitching图像拼接

ImagestitchingDigitalVisualEffectsYung-YuChuangwithslidesbyRichardSzeliski,SteveSeitz,MatthewBrownandVaclavHlavacImagestitchingStitching=alignment+blendinggeometricalregistrationphotometricregistrationApplicationsofimagestitchingVideostabilizationVideosummarizationVideocompressionVideomattingPanoramacreationVideosummarizationVideocompressionObjectremovalinputvideoObjectremovalremoveforegroundObjectremovalestimatebackgroundObjectremovalbackgroundestimationPanoramacreationWhypanorama?Areyougettingthewholepicture?CompactCameraFOV=50x35°Whypanorama?Areyougettingthewholepicture?CompactCameraFOV=50x35°HumanFOV=200x135°Whypanorama?Areyougettingthewholepicture?CompactCameraFOV=50x35°HumanFOV=200x135°PanoramicMosaic=360x180°PanoramaexamplesSimilartoHDR,itisatopicofcomputationalphotography,seekingwaystobuildabettercamerausingeitherhardwareorsoftware.MostconsumercamerashaveapanoramamodeMars:http://www.panoramas.dk/fullscreen3/f2_mars97.htmlEarth:http://www.panoramas.dk/new-year-2006/taipei.html

/

.tw/Whatcanbegloballyaligned?Inimagestitching,weseekforamatrixtogloballywarponeimageintoanother.Areanytwoimagesofthesamescenecanbealignedthisway?ImagescapturedwiththesamecenterofprojectionAplanarsceneorfar-awayscene

ApencilofrayscontainsallviewsrealcamerasyntheticcameraCangenerateanysyntheticcameraviewaslongasithasthesamecenterofprojection!MosaicasanimagereprojectionmosaicprojectionplaneTheimagesarereprojectedontoacommonplaneThemosaicisformedonthisplaneMosaicisasyntheticwide-anglecameraChangingcameracenterDoesitstillwork?syntheticPPPP1PP2WhatcannotThescenewithdepthvariationsandthecamerahasmovement

Planarscene(orafarawayone)PP3isaprojectionplaneofbothcentersofprojection,soweareOK!ThisishowbigaerialphotographsaremadePP1PP3PP2MotionmodelsParametricmodelsastheassumptionsontherelationbetweentwoimages.2DMotionmodelsMotionmodelsTranslation2unknownsAffine6unknownsPerspective8unknowns3D

rotation3unknownsAcasestudy:cylindricalpanoramaWhatifyouwanta360

fieldofview?mosaicprojectioncylinderCylindricalpanoramasStepsReprojecteachimageontoacylinderBlendOutputtheresultingmosaicapplet/courses/cs178/applets/projection.htmlCylindricalpanoramaTakepicturesonatripod(orhandheld)WarptocylindricalcoordinateComputepairwisealignmentsFixuptheend-to-endalignmentBlendingCroptheresultandimportintoaviewerItisrequiredtodoradialdistortioncorrectionforbetterstitchingresults!TakingpicturesKaidanpanoramictripodheadTranslationmodelWhereshouldthesyntheticcamerabeTheprojectionplaneofsomecameraOntoacylinderrealcamerasyntheticcameraCylindricalprojectionAdoptedfrom/tutorials/image-projections.htmCylindricalprojectionCylindricalprojectionAdoptedfrom/tutorials/image-projections.htmCylindricalprojectionunwrappedcylinderxyfxθzCylindricalprojectionunwrappedcylinderxyθxyfCylindricalprojectionunwrappedcylinderxyzxyfs=fgiveslessdistortionf=180(pixels)Cylindricalreprojectionf=380f=280Image384x300top-downviewFocallength–thedirtysecret…AsimplemethodforestimatingfOr,youcanuseothersoftware,suchasAutoStich,tohelp.dfwpInputimagesCylindricalwarpingBlendingWhyblending:parallax,lensdistortion,scenemotion,exposuredifferenceBlendingBlendingBlendingGradient-domainstitchingGradient-domainstitchingPanoramaweavingAssemblingthepanoramaStitchpairstogether,blend,thencropProblem:DriftErroraccumulationsmallerrorsaccumulateovertimeProblem:DriftSolutionaddanothercopyoffirstimageattheendthereareabunchofwaystosolvethisproblemadddisplacementof(y1

–yn)/(n-1)toeachimageafterthefirstcomputeaglobalwarp:y’=y+axrunabigoptimizationproblem,incorporatingthisconstraintbestsolution,butmorecomplicatedknownas“bundleadjustment”(x1,y1)copyoffirstimage(xn,yn)End-to-endalignmentandcropRectanglingpanoramasvideoRectanglingpanoramasRectanglingpanoramasViewer:panorama++++example:/education/courses/cse590ss/01wi/projects/project1/students/dougz/index.htmlViewer:texturemappedmodelexample:http://www.panoramas.dk/CylindricalpanoramaTakepicturesonatripod(orhandheld)WarptocylindricalcoordinateComputepairwisealignmentsFixuptheend-to-endalignmentBlendingCroptheresultandimportintoaviewerDeterminepairwisealignment?Feature-basedmethods:onlyusefeaturepointstoestimateparametersWewillstudythe“Recognisingpanorama”paperpublishedinICCV2003RunSIFT(orotherfeaturealgorithms)foreachimage,findfeaturematches.Determinepairwisealignmentp’=Mp,whereMisatransformationmatrix,pandp’arefeaturematchesItispossibletousemorecomplicatedmodelssuchasaffineorperspectiveForexample,assumeMisa2x2matrixFindMwiththeleastsquareerrorDeterminepairwisealignmentOverdeterminedsystemNormalequationGivenanoverdeterminedsystem

thenormalequationisthatwhichminimizesthesumofthesquaredifferencesbetweenleftandrightsides

Why?Normalequationnxm,nequations,mvariablesNormalequationNormalequation→NormalequationNormalequationDeterminepairwisealignmentp’=Mp,whereMisatransformationmatrix,pandp’arefeaturematchesFortranslationmodel,itiseasier.Whatifthematchisfalse?Avoidimpactofoutliers.RANSACRANSAC=RandomSampleConsensusAnalgorithmforrobustfittingofmodelsinthepresenceofmanydataoutliersComparetorobuststatisticsGivenNdatapointsxi,assumethatmajorityofthemaregeneratedfromamodelwithparameters

,trytorecover

.RANSACalgorithmRunktimes:(1)drawnsamplesrandomly(2)fitparameters

withthesensamples(3)foreachofotherN-npoints,calculateitsdistancetothefittedmodel,countthenumberofinlierpoints,cOutput

withthelargestcHowmanytimes?Howbig?SmallerisbetterHowtodefine?Dependsontheproblem.Howtodeterminekp:probabilityofrealinliersP:probabilityofsuccessafterktrialsnsamplesareallinliersafailurefailureafterktrialsnpk30.53560.69760.5