微软亚洲研究员图像识别网格化网络

BeyondBeyondGridHanHu(胡瀚)MicrosoftResearchAsiaJointworkwithJifengDai,YichenWei,ZhengZhang,JiayuanGu*,HaozhiQi*,YuwenYiLi*andGuodongBeyondGridBeyondGrid•••RelationNetworksforObjectDetection(CVPR2018oral)LearningRegionFeaturesforObjectDetection(ECCV2018)ImageImageObjectpersonInstanceInstancePose,ImageImageObjectpersonInstanceInstancePose,PersonID,CNNforImagespecificfortasksorsharedfortasksorImageCNNforImagespecificfortasksorsharedfortasksorImageGoogleNet,ResNetObjectPool+CNNforObjectFundamentalinCNNforObjectFundamentalinGoogleNet,ResNetObjectConvolutionConvolutionOutputFeature×𝐻×𝑊𝑐′,𝑐,∆ℎ,∆𝑤∙=InputFeature(𝐶𝑖𝑛×𝐻×ConvolutionConvolutionlayerisgrid/coordinateOutputConvolutionConvolutionlayerisgrid/coordinateOutputFeature×𝐻×𝑊𝑐′,𝑐,∆ℎ,∆𝑤∙=෍InputFeature(𝐶𝑖𝑛×𝐻×RoIPooling+FC(conv)••Itmodelspart-partRoIPooling+FC(conv)••Itmodelspart-partItiscoordinateOutputRegion(𝑁×Region(𝑁×𝐶𝑖𝑛×3×InputImage(𝐶𝑖𝑛×𝐻×FCTwoTwoImportantIssuesinComputerWithinCategoryVariation:AppearanceWithinCategoryVariation:AppearanceViewpointIntra-class(ExamplesaretakenfromLiFei-fei’scourseModelingofModelingofObject-ObjectItismucheasiertodetecttheifweknowthereisaCNNsCNNsPartlySolvetheProblem:CategoryItshouldhavenoGeometricMemorizeobjectappearanceundervariousgeometricCNNsCNNsPartlySolvetheProblem:Object-LargereceptivefieldtoincludecontextBetterBetterBeyondGridKeyIdea:Grid/CoordinateAlignment->Non-Grid/Semanticobject-objectBeyondGridKeyIdea:Grid/CoordinateAlignment->Non-Grid/Semanticobject-objectpart-partfeaturetoregionfeatureDCNfirstlearndeformationinconv&rand3rdinCOCORelationNetworksfirstmoduleforobject-objectDeformableDeformableConvolutionalJifengDai*,HaozhiQi*,YuwenXiong*,YiLi*,GuodongZhang*,HanHuandYichenWei(*EqualICCV’2017CoordinateAlignmenttoSemanticdeformableregularOutputCoordinateAlignmenttoSemanticdeformableregularOutputOutputInputInputIllustrationofDeformableOffsetsaredeterminedbyinputIllustrationofDeformableOffsetsaredeterminedbyinputdeformableBilinearEnablesbackpropagationgothroughbothoffsetbranchandinputfeaturemapsBilinearEnablesbackpropagationgothroughbothoffsetbranchandinputfeaturemapsdeformableSamplelocationandbilinearBackpropagationDeformableConvolutionisSemi-DeformableConvolutionisSemi-deformableOutputInputEfficientregularconvforwardinputcolumnEfficientregularconvforwardinputcolumnbackwardinputcolumnweightMatrixMatrixEfficientconvforwardinputcolumnEfficientconvforwardinputcolumnbackwardinputcolumnweightoffsetMatrixMatrixInput/outputInput/outputfeaturemapsarebothof…layerlayerlayerDeformable(BetterSemantic+(BetterCoordinateRoIgulardeformableRoIoutputfeatureinputfeatureInitialstateofsamplingconcurrent1KaimingHeetal.MaskR-CNN.xDeformable(BetterSemantic+(BetterCoordinateRoIgulardeformableRoIoutputfeatureinputfeatureInitialstateofsamplingconcurrent1KaimingHeetal.MaskR-CNN.xDeformableDeformableSameinput&outputastheplainRegularconvolution->deformableRegularRoIpooling->deformableRoIEnd-to-endtrainablewithoutadditionalObjectR-RoIRoIPSRoIPositionScoreR-RoIRoIPSRoIPositionScore:DeformableConvolution/RoILearnedLearnedSamplingSamplingLocationsSamplingLocationsofDeformableSamplingSamplingLocationsofDeformablePartPartOffsetsinDeformableRoIDeformableConvolutiononDeformableConvolutiononVOC&(DeepLabisaSOTAbaselineforsemanticsegmentation,whileclass-awareRPN,FasterR-CNNandR-FCNareSOTAbaselinesforobjectDeformableConvNetsonCOCO(Test-+3.3+3.1+4.7FASTERR-CNN,2FC(RESNET-CLASS-AWARERPN(RESNET-mAPDeformableConvNetsonCOCO(Test-+3.3+3.1+4.7FASTERR-CNN,2FC(RESNET-CLASS-AWARERPN(RESNET-mAP ModelComplexityModelComplexityandRuntimeonVOC&简单、粗暴（网格对齐）简单、温柔（语义对齐）、有LearningRegionLearningRegionFeaturesforObjectJiayuanGu1*,HanHu2,LiweiWang1,YichenWei2andJifeng1Peking2MicrosoftResearchAsiaRegionFeatureRegionFeatureObjectRoIRegionRoIRegionOtherOtherSpatialPyramidEnablesverylight-weighthead(per-regionBetterCoordinateAlignment+BetterSemantic…UnifiedUnifiedAGeneralExistingregionfeatureextractionAGeneralExistingregionfeatureextractionmethodsusedifferentdesignsof𝜔2(𝑏,Other…Other…Learning𝜔Learning𝜔withoutanyHandcraftedBetterencodingcontextFullyLearnableGeometric𝑊𝑏𝑜𝑥∙ ,𝑊𝑖𝑚FullyLearnableGeometric𝑊𝑏𝑜𝑥∙ ,𝑊𝑖𝑚∙<>Appearance𝑊𝑎𝑝𝑝∙Final𝜔∝exp(𝐺𝑒𝑜+SamplingforSamplingforHigherSamplingoutsideRoIregionhasalmostnoaccuracydropbuthigherExperimentsExperimentsonCOCOWhatisWhatis从从图像特征过度到区域特征的统一数学RelationRelationNetworksforObjectHanHu1*,JiayuanGu2*,ZhengZhang1*,JifengDai1,andYichen1MicrosoftResearchAsia2PekingUniversityRelationModelinginpart-partEffectiveRelationModelinginpart-partEffectiveandEasytoRequirenorelationTranslationalObject-objectrelation:wellrecognizedpart-partobject-objectObject-objectrelation:wellrecognizedpart-partobject-object?WellRecognizedWellRecognizedItismucheasiertodetecttheifweknowthereisaRarelyStudiedinDeepLearningRarelyStudiedinDeepLearningIrregularitiesofAtarbitraryimageOfdifferentWithindifferentOfvaryingnumberacrossdifferentimagesGoal:designGoal:designasimplemoduletoobject-objectEffectiveandEasytoRequirenorelationIn-place,ObjectRelationObjectRelationExtensionofObjectRelationExtensionofattentionobject-object(2D(1DLeftfigurecreditbyA.VaswanietRelationbetweenTwoAnovelgeometricappearanceRelationbetweenTwoAnovelgeometricappearanceapp.+geometricdotsmallmax{0,𝑊𝐺∙(𝑞)4dboundingregressioninstandardattentioninobjectrelationRelation𝑜𝑢𝑡(𝑛)appearance+Relation𝑜𝑢𝑡(𝑛)appearance+geometric𝑓𝑜𝑢𝑡(𝑛)=෍ω(𝑚,𝑛)𝑛𝑚)objectMulti-Branchbranch(person-branch(playground-…Multi-Branchbranch(person-branch(playground-…branch(duplicate…ObjectRelationresidualoutput=ObjectRelationresidualoutput=input+…KaimingHe,XiangyuZhang,ShaoqingRen,JianSun.DeepResidualLearningforImageRecognition.CVPR,ObjectRelationEffectiveandObjectRelationEffectiveandEasytoRequirenorelationTranslational…Application:Application:ObjectRegion-basedObjectR-regionRegion-basedObjectR-regionR.Girshick.FastR-CNN.ICCV,S.Renetal.FasterR-CNN.NIPS,Ourmethod:insertingobjectrelationmodulesindependenthandcaftOurmethod:insertingobjectrelationmodulesindependenthandcaftexR-learnable××InstanceDuplicateLearnableDuplicate√LearnableDuplicate√originalTheFirstFullyEnd-to-EndObject×TheFirstFullyEnd-to-EndObject××backpropagation××*Faster××*FasterR-CNNwithResNet-50modelare+2.3mAPbyinserting2with+3%××××*FasterR-CNNwithResNet-50modelareMoremodules:8××*Faster××*FasterR-CNNwithResNet-50modelareImportanceofrelativegeometric××*Faster××*FasterR-CNNwithResNet-50modelareImportanceofmulti-branch××*Faster××*FasterR-CNNwithResNet-50modelareImportanceofresidualDuplicateRemovalNoticeablybetterthanDuplicateRemovalNoticeablybetterthanSlightlybetterthanSoftNMS[N.Bodlaetal,2017]TrainableFullyEnd-to-EndObjectBenefitfromFullyEnd-to-EndObjectBenefitfromfullyend-to-endTrainableUsingStronger+3.0UsingStronger+3.0+2.0+1.0*FasterR-CNNwithResNet-101modelareused(