2019年国际大数据竞赛第2名-homon队

1、The First IKCEST “The Belt and Road” InternationalBig Data ContestHomon3rd Place Semi-Final02Introduction03Model Architecture04Model Construction05Feature Engineering06StackingCONTENT07Summary and Discussion01TeamThe TeamYuting Huang The Australian National UniversityMathematical Science InstituteCh

2、eng Xue The Australian National UniversityArtificial Intelligence GroupYiji ZhaoYongkai Zhang Haomin WenBeijing Jiaotong UniversityInstitute of network science and intelligent systems02Introduction03Model Architecture04Model Construction05Feature Engineering06StackingCONTENT07Summary and Discussion0

3、1TeamEvaluation MetricTaskP1P2P3P9？Remote Image 0.jpg 1.jpg 2.jpg 3.jpg 4.jpg 5.jpg 0067afb4b21cdc99 20190223&16,20190226&15,1e9d3f65a2a7050c 20181009&10,20181015&100fd618da74272249 20190314&18|19,20190315&10918d53a74edb196f 20181205&23,20181207&23 . 9930aafadee4401 20181203&23,20181204&230fd618da74

4、272249 20190314&18|19,20190315&10918d53a74edb196f 20181205&23,20181207&230fd618da7427224s 20190314&18|19,20190315&10918d53a74edb196p 20181205&23,20181207&23918d53a74sdb196p 20181265&23,20181207&24uidtimeVisit RecordsOverall IdeaImageVisit NetSequenceMulti-modal RepresentationMulti-modalResidential A

5、reaSchool Industrial ParkRailway StationAirportParkShopping AreaAdministrative DistrictHospitalCategoriesClassification02Introduction03Model Architecture04Model Construction05Feature Engineering06StackingCONTENT07Summary and Discussion01TeamBest Model ArchitectureInputModelTaskFCConcatMulti-classCla

6、ssificationImageCNN ModelVisitCNN ModelTarget EncodingFCHand-CraftedFCDeep Fusion Network02Introduction03Model Architecture04Model Construction05Feature Engineering06StackingCONTENT07Summary and Discussion01TeamFirst TrailImageAuto-KerasTransfer Learning 100 * 100 * 3CNN modelVisitNLP modelDoc0.330.

7、470.280.43Doc2vecFastTextNone0.540.650fd618da74272249 20190314&18|19,20190315&10918d53a74edb196f 20181205&23,20181207&23 . 9930aafadee4401 20181203&23,20181204&230fd618da74272249 20190314&18|19,20190315&10Dropout(0.5)ReLUDense(1024)ConcatSoftmaxImageVisitImage ModelVisit NLP ModelFastText EmbeddingS

8、E-ResNet-50Deep Fusion Network-1Acc: 0.685Dense(9)imageSE-ResNeXt-50XceptionShake_Pyramid 100 * 100 * 3CNN modelPretreatmentImage EnhancementDefogSamplingCroppingCutoutDark ChannelCNNOver-SamplingUnder-Sampling0.420.410.51SE-ResNeXt-1010.44Inception v40.38Image CNNVisit CNN visitCNN modelSE-ResNet-5

9、00.521D-CNN0.6413D-ResNeXt-500.645Shake-Pyramid0.632DPN-260.646DPN-26-Dropout0.6680fd618da74272249 20190314&18|19,20190315&10 . 9930aafadee4401 20181203&23,20181204&231D2DRecurrence Plot3DDPN+DropoutImageconv1conv2conv3conv4conv5Dropout(0.5)Dropout(0.5)Dropout(0.5)Avg PoolDropout(0.5)ReLUDense(1024)

10、ConcatSoftmaxImageVisitAcc: 0.6630.724Dense(9)VisitReLUDense(1024)Deep Fusion Network-2Image ModelVisit NLP ModelFastText EmbeddingSE-ResNeXt-50SE-ResNeXt-101XceptionShake-PyramidVisit CNN Model1D-CNN3D-ResNeXt-50Shake-PyramidDPN-26-DropoutDFN-2: Confusion MatrixTarget-EncodingIndividual Visiting Pr

11、obability0.000.010.000.000.670.000.020.000.30123456789Prob. of User visit Un categorial 20.020.270.000.000.010.000.000.000.700.020.930.000.000.030.000.020.000.000.130.150.320.100.020.130.110.020.020.020.280.000.000.680.000.020.001.00.041.200.000.000.040.000.020.000.70 PlaceUserClass 9Class 2Class 5A

12、void Leakage: 5-fold validationAvoid Dominating Users: NormalizationSum/AvgSum/AvgAggregation ImageVisitImage ModelVisit ModelFastText EmbeddingSE-ResNeXt-50Acc: 0.8310.855VisitSE-ResNeXt-101XceptionShake-PyramidVisit CNN Model1D-CNN3D-ResNeXt-50Shake-PyramidDPN-26-DropoutTarget EncodingDropout(0.5)

13、ReLUDense(2048)ConcatSoftmaxDense(9)ReLUDense(2048)Deep Fusion Network-3DFN-3 with target encodingWhats Next ?class 1 and 2 account for over half of the dataFind Features to improve 1st and 2nd class02Introduction03Model Architecture04Model Construction05Feature Engineering06StackingCONTENT07Summary

14、 and Discussion01TeamHand-Craft FeaturesWorkingWeekdays8:0017:00Visit FeaturesStaying FeaturesRestingNight and Weekends19:006:00Weekly CountWeekdays CountDistribution FeaturesWhy ?Chinese new yearNational dayWeekly Distribution2nd Class has the most dramatic change19th weekDifferent Responses Weekly

15、 DistributionWeekdaysWeekdays distributionSchool，Indu. Park，Admin. District and Hospital see significant decrease in weekends.People visited Residential Areas tend to stay there for a long time.Distribution of Days that People Staying at the Same PlaceMost of the people visited Railway Station, Airp

16、ort and Park visit less than 5 times.People tend to visit the same residential place for many daysWhile people who visits Railway Station, Airport and Park dont usually visit the same place again.Distribution of DaysHand-Craft FeaturesVisiting FeaturesStaying Hours and Distribution Distribution24 ho

17、ursDistribution of the time entering and leaving the placeDistribution of Staying hoursDistribution at each time pointDistribution of Staying hoursRatio of visit between working hours and non-working hoursRatio of visit between day and nightRatio of visit in the small hoursRatio of visit at the week

18、endsRatio of staying one or two hoursRatio of staying more than five hours at workRatio of staying more than five hours in the small hoursSkewness, mean, variance of hourly, daily and weekly distribution VisitingStayingAverage Staying hour in working hoursAverage Staying hour at the weekendsAverage

19、Staying hour in the small hoursAcc: 0.7257 x 26ImageVisitImage ModelVisit ModelTarget EncodingSE-ResNeXt-50DFN-4 with target encoding and hand-crafted featuresVisitSE-ResNeXt-101XceptionShake-PyramidVisit CNN Model1D-CNN3D-ResNeXt-50Shake-PyramidDPN-26-DropoutHand CraftDropout(0.5)ReLUBatchNormaliza

20、tionDense(2048)ReLUBatchNormalizationDense(2048)ConcatSoftmaxAcc: 0.865Deep Fusion Network-4Tri-Model with target encoding and hand crafted featuresWe successfully improved model performance on the 1st class and the 2nd.DFN-40.070.020.020.02Deep model optimizationModel TricksModel Parameter Xavier i

21、nitWarmup trainingNo bias decayLabel smoothingRandom erasingCutout for imageLinear scaling learning rateCosine learning rate decayHe, et al. Bag of tricks for image classification with convolutional neural networks. CVPR 2019: 558-567.Data: Preliminary+ Semi-FinalData Split: 0.9 train, 0.1 valSample

22、r: Stratified SamplerOptimizer: AdamMinibatch: 128Learning rate: 3e-4Other AttemptsLinear ModeTree ModeNN ModeLinearSVCLRLightGBMXGboostMLPXDeepFMImage PCAVisit PCAHard-CraftTarget EncodingCNN FeaturesFeatureModeAlgorithmPerformance0.74060.84530.86650.85880.83690.848302Introduction03Model Architectu

23、re04Model Construction05Feature Engineering06StackingCONTENT07Summary and Discussion01TeamlearnlearnlearnlearnlearnlearnlearnlearnlearnlearnlearnlearnTraining DataTraining DataTest DataPredictsTest DataLayer1Layer2Fold1Feature Layer1Fold2Fold3Fold4Fold10Model2Model3.Model10New FeatureModel11Fold1-10

24、Model110 folds10 seeds avgModel4.PredictsPredictsPredictsPredictsPredicts.learnlearnlearnlearnlearnlearnlearnlearnlearnlearnPredictPredictPredictPredictPredictInputNew InputPredictPredictPredictPredictPredictPredictaveragePredictlearnlearnlearnStacking StructureFeature Layer2StackingStacking (10 Fol

25、d)Second LayerFirst LayerAvgInputsLightGBMLinearSVCLogistic Regression Image PCAVisit PCAHand-CraftTar. EncodingCNN FeaturesDeep FusionNetAveragepredict0.10.10.20.10.150.10.10.050.1LightGBMLinearSVCLogistic Regression VisitImageAcc: 0.871Final ModelStackingSecond LayerFirst LayerInputsLightGBM(10 Fo

26、ld)Hand-CraftTar. EncodingCNN FeaturesDeep FusionNetpredict0.10.10.20.10.150.10.10.050.1LightGBMVisitImageTar. EncodingAcc: 0.8729Hand-Craft02Introduction03Model Structure04Model Construction05Feature Engineering06StackingCONTENT07Summary and Discussion01TeamFirst AttemptDeepFuNet (DFN)BehaviorVisit

27、ing FeatureStacking0.650.6850.8270.8550.865 (DFN)0.867 (LGB)0.8729 (3rd)(Preliminary)(Semi-Final)DPN+DropoutTarget EncodingCombined SamplingImg-CNN EnsembleMulti-ModelEnsembleStackingVisiting CountStaying timeDistributionBi-ModalitySummaryExpectationP1P2P3PnimgCNNTimeNode RepresentationNode Classifi

28、cationGNN-Based ModelGNN ModelGraph1 Yamada Y, Iwamura M, Akiba T, et al. Shakedrop regularization for deep residual learningJ. arXiv preprint arXiv:1802.02375, 2018.2 Szegedy C, Ioffe S, Vanhoucke V, et al. Inception-v4, inception-resnet and the impact of residual connections on learningC/Thirty-Fi

29、rst AAAI Conference on Artificial Intelligence. 2017.3 Chen Y, Li J, Xiao H, et al. Dual path networksC/Advances in Neural Information Processing Systems. 2017: 4467-4475.4 Keesara S, Saravanaraj S, Sun L, et al. Access network dual path connectivity: U.S. Patent 9,813,257P. 2017-11-7.5 Gastaldi X.

30、Shake-shake regularizationJ. arXiv preprint arXiv:1705.07485, 2017.6 Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networksC/Proceedings of the thirteenth international conference on artificial intelligence and statistics. 2010: 249-256.7 Goyal P, Dollr P, Girshick R, e