基于mapreduce的关联规则数据挖掘在电信网络运营分析中的应用

基于MAPREDUCE的关联规则数据挖掘在电信网络运营分析中的应用/PAPEREDUCN基于MAPREDUCE的关联规则数据挖掘在电信网络运营分析中的应用苗宇，侯春萍北京邮电大学大学电子工程学院，北京市100876摘要本文基于电信运营商建设大数据共享平台的应用需求和现状，研究相关技术，提出了利用开源框架HADOOP对海量信令数据进行挖掘，以弥补传统作业系统在这方面的缺陷与不足，设计并实现了基于MAPREDUCE的并行化APRIORI算法，通过关联规则挖掘发现移动通信网络业务量和业务质量的数据突变状况。实验结果表明，该方法能够有效地从海量数据中发现数据突变的相关业务因素和场景，为上层应用有针对性解决数据突变问题提供依据。关键词数据挖掘关联规则海量数据HADOOP运营分析中图分类号TP391APPLICATIONOFMAPREDUCEBASEDASSOCIATIONRULEDATAMININGONOPERATIONOFTELECOMMUNICATIONNETWORKMIAOYU,HOUCHUNPINGDEPARTMENTOFELECTRONICENGINEERING,BEIJINGUNIVERSITYOFPOSTSANDTELECOMMUNICATIONS,BEIJING100876ABSTRACTTHISPAPERBASEDONTHEREQUIREMENTSANDSTATUSOFCONSTRUCTINGDATASHARINGPLATFORMFORTELECOMOPERATORS,PROPOSEDUSINGTHEHADOOPOPENSOURCEFRAMEWORKFORMASSIVESIGNALINGDATAMININGANALYSISTOMAKEUPFORTHEDEFECTSANDDEFICIENCIESOFTRADITIONALMONITORINGSYSTEMAHADOOPBASEDSIGNALINGMININGPLATFORMANDAMAPREDUCEBASEDAPRIORIALGORITHMISDESIGNEDANDIMPLEMENTEDTODISCOVERTHEASSOCIATIONRULESONNETWORKSERVICEQUALITYANDDATAMUTATIONTHEEXPERIMENTALRESULTSSHOWTHAT,THEMETHODCANEFFECTIVELYDISCOVERAVARIETYOFBUSINESSFACTORSANDSCENARIOSWHICHLEADTOTHEMUTATION,ANDPROVIDESUPPORTFORTHEUPPERAPPLICATIONTOSOLVETHEPROBLEMKEYWORDSDATAMININGASSOCIATIONRULEBIGDATAHADOOPOPERATIONANALYSIS1/PAPEREDUCN0INTRODUCTIONWITHTHERAPIDDEVELOPMENTOFMOBILECOMMUNICATION,COEXISTINGOF2G/3G/4GSYSTEM,APPEARENCEOFCOMLPEXTYPESOFTELECOMMUNICATIONEQUIPMENTS,ANDDISPERSIVEMODEOFCONSTRACTION,THETRADITIONALNETWORKMANAGEMENTSYSTEMOFOPERATORSAREFACINGGREATCHALLENGESNETWORKMANAGEMENTSYSTEMOUGHTTOBECLOUDBASED,INTEGRATED,INTELLIGENT,AUTOMATICINFUTUREONEOFTHESIGNIFICANTDIRECTIONSFORTHEEVOLUTIONOFNMSIS3LAYEREDARCHITECTUREINCLUDEDNETWORKELEMENTANDEQUIPMENTLAYER,ACQUISITIONLAYERANDDATASHARINGLAYERAMONGTHEM,THEDATASHARINGLAYERISRESPONSIBLEFORINTEGRATINGVARIOUSTYPESOFDATAANDPROVIDINGUNIFIEDDATASERVICEFORALLOFUPPERLAYERAPPLICATIONSTHEREFOREOPERATORSWILLHAVEANATURALADVANTAGEOFCOLLECTINGBIGDATA1HOWEVER,ASTHEDRAMATICINCREASINGINDATASIZE,THETRADITIONALMETHODOFSIGNALINGMONITORINGISNOLONGERAPPLICABLEUNDERTHENEWDATASHARINGPLATFORMMEANWHILE,HUGEAMOUNTSOFDATANEEDTOREFINEANDSUBLIMATEINTOUSEFULINFOMATIONTOBEPROVIDEDTOANALYSTSORDECISIONMAKERSTHUSITISNECESSARYTOCOLLECTANDMINEBIGDATATOGETVALUABLEINFORMATIONTHEAPACHEHADOOPSOFTWARELIBRARYISAFRAMEWORKTHATALLOWSFORTHEDISTRIBUTEDPROCESSINGOFLARGEDATASETSACROSSCLUSTERSOFCOMPUTERSUSINGSIMPLEPROGRAMMINGMODELSITPROVIDESASOLUTIONFORDEALWITHMASSIVEDATAINTHISPAPER,WEPROPOSEDABIGDATAPROCESSINGFRAMEWORKBASEDONRANDHADOOP,THENOPERATEDDATAMININGFORDATAMUTATIONONGNSIGNALINGASSOCIATIONRULEDATAMININGALGORITHMAPRIORIANDITSPARALLELIZATIONINMAPREDUCEFRAMEWORKAREDESCRIBEDINSECTION1SECTION2PROPOSEDSOMEPREPROCESSINGMETHODSONGNINTERFACESIGNALINGASTHEDATASOURCEINSECTION3,WEBUILTUPADATAMININGPLATFORMONHADOOPANDACHIEVEASSOCIATIONRULESMININGTOFINDDATAMUTATIONBYANALYZINGRESULTS,WEDISCOVEREDTHAT1MAPREDUCEBASEDASSOCIATIONRULEDATAMININGALGORITHM11FUNDAMENTALPRINCIPLEOFASSOCIATIONRULEASSOCIATIONRULE2LEARNINGISAPOPULARANDWELLRESEARCHEDMETHODFORDISCOVERINGINTERESTINGRELATIONSBETWEENVARIABLESINLARGEDATABASESBYRAGRAWALIN1993ASSOCIATIONRULEGENERATIONISUSUALLYSPLITUPINTOTWOSEPARATESTEPSFIRST,MINIMUMSUPPORTISAPPLIEDTOFINDALLFREQUENTITEMSETSINADATABASESECOND,THESEFREQUENTITEMSETSANDTHEMINIMUMCONFIDENCECONSTRAINTAREUSEDTOFORMRULESASSUMETHATII1,I2,IKISACOLLECTIONOFITEMS,CALLEDITEMSETAKITEMSETISASETOFITEMSWHICHCONTAINSKITEMSTASKRELATEDDATASETDISACOLLECTIONOFDATABASETRANSACTIONSWHEREEACHTRANSACTIONTCOMPOSEDBYITEM,ANDTIEACHTRANSACTIONHASANIDENTIFIER,2/PAPEREDUCNCALLEDTIDASSUMETHATXISANITEMSETANDTRANSACTIONTCONTAINSX,IFFXTASSOCIATIONRULESARESHAPEDLIKETHEIMPLICATIONXY,XI,YI,ANDXY6TOMAKEXYFOUNDEDINTHETRANSACTIONSETDSHOULDHAVESUPPORTANDCONFIDENCESUPPORTMEANSTHEPERCENTAGEOFTRANSACTIONSINDCONTAINXY,THATISPROBABILITYPXYCONFIDENCEREPRESENTSTHATTHEPERCENTAGEOFTRANSACTIONSCONTAINXINDCONTAINY,THATISPROBABILITYPX|YIFANITEMSETMEETSTHEMINIMUNSUPPORTTHRESHOLDMINSUPPORT,THISITEMSETISAFREQUENTITEMSETARULEWHICHMEETINGTHEMINIMUMSUPPORTTHRESHOLDMINSUPPORTANDTHEMINIMUMCONFIDENCETHRESHOLDMINCONFIDENCTATTHESAMETIMEISCALLEDASTRONGRULESTRONGRULESARENOTALWAYSINTERESTED,SUPPORTCONFIDENCEFRAMEWORKISNOTENOUGHTOFILEROUTTHEASSOCIATIONRULESTHATARENOTINTERESTEDTODEALWITHTHISPROBLEM,CORRELATIONMEASURECOULDBEUSEDTOEXPANDTHELIFTISASIMPLECORRELATIONMEASUREITEMSETXISINDEPENDENTOFY,IFPXXPXPX,OTHERWISEXISDEPENDENTOFYLISTISDEFINEDASFOLLOWSLIFTX,YPXYPXPY1IFTHEVALUEISLESSTHAN1,THENXANDYAREINVERSELYCORRELATED,ITMEANSONESAPPERANCECOULDLEADTOANOTHERDOESNOTAPPEARIFGREATERTHAN1,XANDYAREPOSITIVECORRELATED,ITMEANSTHATONESAPPEARANCECONTAINSANOTHERSAPPEARANCEIFEQUALTO1,XANDYAREINDEPENDENT12APRIORIALGORITHMASSOCIATIONRULESAREPRESENTINTHEREACTIONBETWEENTWOORMORECORRELATIONATTRIBUTES,WHICHAREDESIGNEDTOPREDICTTHEVALUEOFSOMEPROPERTIESFROMOTHERATTRIBUTEVALUESATTHESAMETIMEITCANBEPROMOTEDTOREFLECTTHEKNOWLEDGEOFDEPENDENCEORRELATIONBETWEENEVENTSTHEMOSTFAMOUSASSOCIATIONRULEDATAMININGALGORITHMISCALLEDAPRIORI3WHICHWASPROPOSEDBYRAGRAWALAPRIORIALGORITHMUSESAPROPERTYCALLEDAPRIORIPROPERTYTOSCANTHEDATABASESFORMULTIPLETIMESFREQUENT1ITEMSETL1ISGENERATEDFROMTHEFIRSTTRIPTOTHEDATABASEBEFORETHEKTHSCAN,THECANDIDATEITEMSETCKWILLBEGENERATEDFROMLK1THENCALCULATETHESUPPORTOFEACHITEMINCKBYTHEKTHSCANNINGLKISANITEMSETWHICHISCOMPOSEDBYTHEITEMSMEETINGHEMINSUPPORTTHEWHOLEPROCESSTERMINATESWHENCKORLKTHEAPRIORIPROPERTYANDPSEUDOCODEASFIG1THEOREM1APRIORIPROPERTYANYSUBSETOFFREQUENTITEMSETMUSTBEFREQUENT13MAPREDUCEBASEDASSOCIATIONRULEMAPREDUCE4USINGTHEIDEAOF”DIVIDEANDCONQUER”TOOPERATEONLARGEDATASETS,AJOBISDISTRIBUTEDTOEACHNODEUNDERTHEMANAGEMENTOFAMASTERNODETOCOMPLETE,ANDTHENTHROUGHTHEINTEGRATIONOFTHEINTERMEDIATERESULTSOFTHESUBNODETOGETTHEFINALRESULTABOVE3/PAPEREDUCNDATATRANSACTIONDATABASEDRESULTFREQUENTITEMSETOFSIZEK1CKCANIDATEITEMSETOFSIZEK2LKFREQUENTITEMSETOFSIZEK3L1FREQUENTITEMS4INTK15WHILELK6DO6CK1LK/LK7FOREACHTDDO8CCOUNTCCK19LK1CCK1|CCOUNTMINSUPPORT10END11K12END13RETURNKLK1APRIORIALGORITHMPROCESSISHIGHLYABSTRACTEDINTOTWOFUNCTIONS,MAPANDREDUCETHEMAPFUNCTIONISINCHARGEOFSPLITINGTASKINTOMULTIPLESUBTASKS,ANDTHEREDUCEFUNCTIONRESPONSIBLEFORAGGREGATINGTHERESULTSOFSUBTASKSASFOROTHERKINDSOFCOMPLEXPROBLEMSINPARALLELPROGRAMMING,SUCHASDISTRIBUTEDSTORAGE,JOBSCHEDULING,LOADBALANCING,FAULTTOLERANCE,NETWORKCOMMUNICATIONS,RESPONSIBLEFORPROCESSINGBYTHEMAPREDUCEFRAMEWORKTHEMAPFUNCTIONANDTHEREDUCEFUNCTIONARETHECOREPARTOFMAPREDUCEFRAMESPECIFICFUNCTIONSOFTHESETWOFUNCTIONACHIEVEBYTHEUSERACCORDINGTOTHEIRDEMANDJUSTFOLLOWTHEUSERDEFINEDRULESTOPUTKEY,VALUEINTOANDGETANOTHERKEY,VALUEORAGROUPINTHEMAPPHASE,DATASTOREDINHDFSISDIVIDEDINTOFIXEDSIZESPLIT64MDEFAULTHADOOPCREATEAMAPPERFOREACHSPLITTORUNTHEUSERDEFINEDMAPFUNCTIONMAPFUNCTIONGETAK1,V1ASINPUT,ANDOUTPUTTHEINTERMEDIATERESULTSK2,V2NEXT,THEINTERMEDIATERESULTSARESORTEDBYTHEKEYOFTHEM,RESULTSWHICHHAVETHESAMEKEYAREGROUPEDTOGETHERTOANEWLIST,NAMELYK2,LISTVINTHEREDUCEPHASE,REDUCERREVEIVESOUTPUTDATAFROMDIFFERENTMAPPERANDSORTEDTHENCALLTHEUSERDEFINEDREDUCEFUNCTIONTOPROCESSK2,LISTVTOOBTAINK3,V3FINALLY,RESULTSAREWROTETOHDFSASFILEFIG2ISTHEPSEUDOOFMAPREDUCEBASEDAPRIPRIALGORITHM5THATRUNSONMAP/REDUCEFRAMEWORKPRUNECK1FUNCTIONISTOREMOVETHENONFREQUENTITEMSETCK1BYELIMINATINGNONFREQUENTITEMSETSCKWITHAPRIORIPROPERTY4FIG/PAPEREDUCNDATATRANSACTIONDATABASEDRESULTFREQUENTITEMSETOFSIZEK1MAPTRANSACTIONTINDATASOURCETOALLMAPNODES/INEACHMAPNODEM/2CM1SIZE1FREQUENTITEMSATTHENODEM/INREDUCE,COMPUTEC1ANDL1WITHALLCM1/3C1FREQUENT1ITEMS/MINSUPPORTNUM/TOTALITEMS/4L1SIZE1FREQUENTITEMSMINSUPPORT5K16WHILELK6DO/INEACHMAPNODEM/LMKLKMAPPEDTOEACHNODEM/SORTTOREMOVEDUPLICATEDITEMS/7CMK1LK/LMK/INREDUCE,USEAPRIORIPROPERTY/COMPUTECK1WITHALLSORTEDCMK1/8IFK3THEN9PRUNECK110END11FOREACHTCK1DO/INEACHMAPNODEM/12INCREMENTTHECOUNTOFALLCANDIDATESINLMK1THATARECONTAINEDINT13END/INREDUCE,FINDLK1WITHLMK1ANDMINSUPPORT/14LK1SIZEK1FREQUENTITEMSMINSUPPORT15K16END17RETURNKLK2MAPREDUCEBASEDAPRIORI5FIG/PAPEREDUCN2DATAANALYSISANDPREPROCESSINGINTHISPAPER,WEUSEGNSIGNALINGASTHESAMPLEDATAGNINTERFACEISANINTERFACEBETWEENSGSNANDGGSNITCANPROVIDEWEB/WAPSERVICESACCESSBUSINESSDATA,USERLEVELACCESSLIST,INCLUDEDSTARTTIME/ENDTIME/MSISDN/USERIP/APN/LAC/CI/PROTOCOL/UPBYTES/DOWNBYTESETCITISTRANSACTIONAL,LINEONBEHALFOFAUSERSACCESSRECORD,SOITISCONDUCIVETOTHEAPPLICATIONOFAPRIORIALGORITHMINORDERTOENSURETHEQUALITYANDVALIDITYOFDATAMININGRESULTS,THEREISANEEDFORDATAPREPROCESSING1PROCESSINGOFVACANTANDERRORVALUESTHERAWDATACONTAINSALARGENUMBEROFVACANCIESVALUES,THEYNEEDTOBEADDRESSEDBEFORETHEDATAMININGBECAUSEOFTHERECORDSINDATASETAREARRANGEDINCHRONOLOGICALORDER,ADJACENTRECORDSARENOTRELEVANT,ANDTHEDATAWHICHSHARETHESAMECELLORTHESAMEUSERAREDISPERSEDINTHEENTIREDATASET,SOWEUNIFIEDIGNOREDTHEMISSINGVALUES2REDUNDANTPROCESSINGSIGNALINGDATAINCLUDES62FIELDS,SOMEOFTHEFIELDSCONTAINALARGEAMOUNTOFREDUNDANTINFORMATION,FOREXAMPLE,TELNUMBER/CLASS/SERVICEETCTHESEFIELDSAREUSELESSFORDATAMININGANDSTATISTICANALYSISONOURDATASET,THEREFORETHEYWILLBEREDUCED3STANDARDIZATIONSOMEOFTHESAMPLEDATATYPEISTEXTBASED,TEXTBASEDDATAONLYFORQUALITATIVEANALYSIS,NUMERICALCALCULATIONCANNOTBEMADEFORTHISTYPEOFDATA,SOWECONVERTEDTHEMINTOTEXTDATAENUMERATIONINADDITION,SOMETIMERELATEDFIELDSCONTAINSDATA/HOUR/MINUTE/SECOND,THESEVALUESARESPREADTOOTHINTOANALYSIS,ANDINCONDIDERATIONOFANALYSISDEMAND,WERESERVETWOFIELDSTOREPRESENTTIME,HOURANDMINUTEFORTHEMINUTE,CONVERTHEMTOFIVEMINUTESGRANULARITYASTHESTARTINGMINUTE4TRANSFORMATIONINORDERTOANALYZEMUTATIONSOFDATA,WENEEDTOGENERATEANEWDIFFERENCEINDICATORINACCORDANCEWITHTHEQUANTIZEDVALUEDIFFERENCEBETWEENNEIGHBORINGTIMEGRANULARITYASANALIZINGTHEDATAMUTATIONUNDERONECELL,FIRST,TOALLUSERSFOREACHCELLCALCULATEITSQUANTIFYFIELDSASASUMMATIONFIELDANDCALCULATEITSRATIOFIELDSASAVERAGINGFIELDFOREXAMPLE,ONECELLWHOSELACIS22852ANDCIID4414,SUMUPBYTESASANEWFIELDUPBYTESSUM,AVERAGINGUPFLOWASANEWFIELDUPFLOWAVGTHEN,MAKETHEDIFFERENCEONTHESESTATISTICALFIELDSBETWEENRECORDINTHISGRANULARITYANDTHEPREVIOUSGRANULARITYIFTHECORRESPONDINGFIELDINPREVIOUSGRANULARITYISNONE,USEZEROINSTEADFOREXAMPLE,ASSUMETHATTHEUPBYTESINTHIS5MINUTESGRANULARITYIS10,ANDITIS2FIVEMINUTESBEFORE,THENUPBYTESCDIFFERENCEFIELDIS8INCURRENTGRANULARITY6/PAPEREDUCN5DISCRETIZATIONASAPRIORIALGORITHMISABOOLEANASSOCIATIONRULE,SOWENEEDTODISCRETEQUANTITATIVEFIELDSINSAMPLEDATATHETRADITIONALMETHODUSUALLYDISCRETECONINUOUSVALUESINTOEQUIVALENTINTERVALSBUTWECANNOTDETERMINETHELEVELSOFVALUESINOURSAMPLEDATAJUSTONTHEBASISOFEQUIVALENTPARTITION,BECAUSETHEIRPROBABILITYDISTRIBUTIONISUNKNOWNTHUSWEADOPTTHEMETHODOFKMEANS6CLUSTERINGTOOBTAINTHEDIVISIONOFTHEINTERVALS,ANDTHENDISCRETEDATAINTOTHEM,LABELSOFINTERVALSIN15AFTERPREPROCESSING,WEOBTAINTHEFOLLOWINGMAINFIELDSINTABLE13EXPERIMENTANDRESULT31EXPERIMENTALPLATFORMINTRODUCTIONTOACHIEVEOURPROPOSEDMETHOD,WESETUPAHADOOPBASEDDATAMININGPLATFORMANDUSERHADOOPASTHEINTERFACEBETWEENHADOOPANDRRHADOOPISANOPENSOURCEPRODUCTTOCOMBINEHADOOPANDRITALLOWSUSTOUSETHERLANGUAGEENVIRONMENTONAHADOOPCLUSTER,BYMAKINGUSEOFTHENATURALADVANTAGESOFTHERLANGUAGEFORDATAANALYSIS,WHICHGREATLYSIMPLIFIESTHECOMPLEXITYOFWRITINGDATAMININGALGORITHMPROGRAMRHADOOPINCLUDESTHREEPACKAGES,RHDFS/RMR2/RHBASE,CORRESPONDINGTOHDFS/MAPREDUCE/HBASEINHADOOPARCHITECTURERESPECTIVELY1DATAFIELDSANDFORMATAFTERPREPROCESSINGINDEXFIELDFORMATSAMPLE1SHOUR24HOUR132SMINUTE5MINUTE553LACDECIMALINTEGER228524CIDECIMALINTEGER44145AREASTRING”RESIDENTIAL”6CITYSTRING”FUZHOU”7UPBYTESCDECIMALINTEGER21034208DOWNBYTESCDECIMALINTEGER86669119DELAYTIMECDECIMALINTEGER311710UPFLOWCDECIMALFRACTION5462511DOWNFLOWCDECIMALINTEGER1065412UPERRRATECDECIMALFRACTION00913DOWNERRRATECDECIMALFRACTION001OVERALLARCHITECTUREOFEXPERIMENTALPLATFORMINFIG3ONTHEBASISOFTHEPLATFORM7TAB/PAPEREDUCN3ARCHITECTUREOFEXPERIMENTALPLATFORMABOVE,WEREALIZEDTHEDATAPREPROCESSING,DATAMININGANDVISUALIZATIONCOMPONENTSDATACLEANINGANDREDUCTIONPHRASEAREEXECUTEDINREADINGSTAGEDATATRANSFORMATIONISACHIEVEDBYHIVESANALYTICFUNCTIONLAGLIKEUSEDINORACLEDATABASEMAPREDUCEBASEDASSOCIATIONRULEALGORITHMAPRIORIANDMAPREDUCEBASEDCLUSTERINGALGORITHMKMEANSAREIMPLEMENTEDBOTHINRLANGUAGETHEINPUTDATAANDTHEOUTPUTRESULTSARESTOREDINTOHDFS32DATAMININGOPERATIONANDRESULTANALYSISINTHISEXPERIMENT,THEGNSIGNALINGDATAISUSEDASANANALYSISOBJECTDATAISSTOREDINHDFSINCSVFORMAT,EACHRECORDREPRESENTSAROWINFILESTHEORIGINNALDATASIZEIS130G,CONTAINS1675829442RECORDS,STOREDIN5DATANODESWITH3REDUNDANTBACKUPFIELDS16FROMTABLE1WEREASSGINEDTOBETHERHSRIGHTHANDSIDEOFRESULTRULES,AND713ASLHSLEFTHANDSIDEWESETMINSUPPORTAS0001ANDMINCONFIDENCEAS09THETOP20RULESSORTEDBYLIFTASSHOWNINFIG4EACHCIRCLEREPRESENTSARULE,ARROWSPOINTTOITARELHSANDRHSISREPRESENTEDBYTHEARROWSSTARTINGFROMTHECIRCLETHESIZEOFTHECIRCLEONBEHALFOFTHESUPPORT,COLORDEPTHONBEHALFOFLIFTASCANBESEENFROMTHERESULTS,THREERULESWHICHHASTHEHIGHLIFTSHAREDOWNBYTESC1ASTHERHSONECELLLAC24581,CI13332,AREACOMMERCIALCENTREHASSUFFEREDFROMADOWNLOADDROPDURINGTHEPERIODOF14WITHCONFIDENCE1,LIFT78DOWNLOADFLOWDRAMATICALLYINCREASEDINCELLLAC24579,CI48062,AREAURBANROADTHEPERIODOF12BYQUERYINGTHELOGINEXISTINGNETWORKS,WEVERIFIEDTHESERESULTS8FIG/PAPEREDUCNS_HOUR12S_HOUR13S_HOUR14LAC23005LAC24579LAC24581LAC24834CI13332CI17717CI48062CI48303CI55412CI55662CI56023CI56631CI56692CI64231AREACOMMERCIALCENTREAREAHIGHSPEEDAREAOFFICEBUILDINGAREAPUBLICINSTITUTIONAREARAILWAYSTATIONAREAURBANROADDOWNBYTESC1DOWNBYTESC2DOWNBYTESC5DOWNFLOWC54TOP20RULESSORTEDBYLIFT4CONCLUSIONTHISPAPERRESEARCHEDTHEBIGDATAPROCESSINGTECHNIQUES,PROPOSEDMETHODOFOPER