Rousch Basics of Heavy Duty NVH Course (7 Nov 2006).ppt

september15,2008,datamining:conceptsandtechniques,1,chapter6.classificationandprediction,whatisclassification?whatisprediction?classificationbydecisiontreeinductionbayesianclassificationrule-basedclassificationclassificationbybackpropagation,supportvectormachines(svm)associativeclassificationlazylearners(orlearningfromyourneighbors)otherclassificationmethodspredictionaccuracyanderrormeasuresensemblemethodsmodelselectionsummary,september15,2008,datamining:conceptsandtechniques,2,classificationpredictscategoricalclasslabels(discreteornominal)classifiesdata(constructsamodel)basedonthetrainingsetandthevalues(classlabels)inaclassifyingattributeandusesitinclassifyingnewdata,classification,september15,2008,datamining:conceptsandtechniques,3,classificationatwo-stepprocess,modelconstruction:describingasetofpredeterminedclasseseachtuple/sampleisassumedtobelongtoapredefinedclass,asdeterminedbytheclasslabelattributethesetoftuplesusedformodelconstructionistrainingsetthemodelisrepresentedasclassificationrules,decisiontrees,ormathematicalformulaemodelusage:forclassifyingfutureorunknownobjectsestimateaccuracyofthemodeltheknownlabeloftestsampleiscomparedwiththeclassifiedresultfromthemodelaccuracyrateisthepercentageoftestsetsamplesthatarecorrectlyclassifiedbythemodeltestsetisindependentoftrainingset,otherwiseover-fittingwilloccuriftheaccuracyisacceptable,usethemodeltoclassifydatatupleswhoseclasslabelsarenotknown,september15,2008,datamining:conceptsandtechniques,4,process(1):modelconstruction,trainingdata,classificationalgorithms,ifrank=professororyears6thentenured=yes,classifier(model),september15,2008,datamining:conceptsandtechniques,5,process(2):usingthemodelinprediction,classifier,testingdata,unseendata,(jeff,professor,4),tenured?,september15,2008,datamining:conceptsandtechniques,6,supervisedvs.unsupervisedlearning,supervisedlearning(classification)supervision:thetrainingdata(observations,measurements,etc.)areaccompaniedbylabelsindicatingtheclassoftheobservationsnewdataisclassifiedbasedonthetrainingsetunsupervisedlearning(clustering)theclasslabelsoftrainingdataisunknowngivenasetofmeasurements,observations,etc.withtheaimofestablishingtheexistenceofclassesorclustersinthedata,september15,2008,datamining:conceptsandtechniques,7,decisiontreeinduction:trainingdataset,thisfollowsanexampleofquinlansid3(playingtennis),september15,2008,datamining:conceptsandtechniques,8,output:adecisiontreefor“buys_computer”,september15,2008,datamining:conceptsandtechniques,9,algorithmfordecisiontreeinduction,basicalgorithmtreeisconstructedinatop-downrecursivedivide-and-conquermanneratstart,allthetrainingexamplesareattherootattributesarecategorical(ifcontinuous-valued,theyarediscretizedinadvance)examplesarepartitionedrecursivelybasedonselectedattributestestattributesareselectedonthebasisofaheuristicorstatisticalmeasure(e.g.,informationgain)conditionsforstoppingpartitioningallsamplesforagivennodebelongtothesameclasstherearenoremainingattributesforfurtherpartitioningmajorityvotingisemployedforclassifyingtheleaftherearenosamplesleft,september15,2008,datamining:conceptsandtechniques,10,attributeselectionmeasure:informationgain(id3/c4.5),selecttheattributewiththehighestinformationgainletpibetheprobabilitythatanarbitrarytupleindbelongstoclassci,estimatedby|ci,d|/|d|expectedinformation(entropy)neededtoclassifyatupleind:informationneeded(afterusingatosplitdintovpartitions)toclassifyd:informationgainedbybranchingonattributea,september15,2008,datamining:conceptsandtechniques,11,attributeselection:informationgain,classp:buys_computer=“yes”classn:buys_computer=“no”,means“age=30”has5outof14samples,with2yesesand3nos.hencesimilarly,september15,2008,datamining:conceptsandtechniques,12,chapter6.classificationandprediction,whatisclassification?whatisprediction?issuesregardingclassificationandpredictionclassificationbydecisiontreeinductionbayesianclassificationrule-basedclassificationclassificationbybackpropagation,supportvectormachines(svm)associativeclassificationlazylearners(orlearningfromyourneighbors)otherclassificationmethodspredictionaccuracyanderrormeasuresensemblemethodsmodelselectionsummary,september15,2008,datamining:conceptsandtechniques,13,bayesianclassification:why?,astatisticalclassifier:performsprobabilisticprediction,i.e.,predictsclassmembershipprobabilitiesfoundation:basedonbayestheorem.performance:asimplebayesianclassifier,navebayesianclassifier,hascomparableperformancewithdecisiontreeandselectedneuralnetworkclassifiersincremental:eachtrainingexamplecanincrementallyincrease/decreasetheprobabilitythatahypothesisiscorrectpriorknowledgecanbecombinedwithobserveddatastandard:evenwhenbayesianmethodsarecomputationallyintractable,theycanprovideastandardofoptimaldecisionmakingagainstwhichothermethodscanbemeasured,september15,2008,datamining:conceptsandtechniques,14,bayesiantheorem:basics,letxbeadatasample(trainingdata)lethbeahypothesisthatxbelongstoclasscclassificationistodeterminep(h|x),theprobabilitythatthehypothesisholdsgiventheobserveddatasamplexp(h)(priorprobability),theinitialprobabilitye.g.,xwillbuycomputer,regardlessofage,income,p(x):probabilitythatsampledataisobservedp(x|h)(posterioriprobability),theprobabilityofobservingthesamplex,giventhatthehypothesisholdse.g.,giventhatxwillbuycomputer,theprobabilitythatxis31.40,andhasmediumincome,september15,2008,datamining:conceptsandtechniques,15,bayesiantheorem,giventrainingdatax,posterioriprobabilityofahypothesish,p(h|x),followsthebayestheoreminformally,thiscanbewrittenasposteriori=likelihoodxprior/evidencepredictsxbelongstociifftheprobabilityp(ci|x)isthehighestamongallthep(ck|x)forallthekclassespracticaldifficulty:requiresinitialknowledgeofmanyprobabilities,significantcomputationalcost,september15,2008,datamining:conceptsandtechniques,16,towardsnavebayesianclassifier,letdbeatrainingsetoftuplesandtheirassociatedclasslabels,andeachtupleisrepresentedbyannattributevectorx=(x1,x2,xn)supposetherearemclassesc1,c2,cm.classificationistoderivethemaximumposteriori,i.e.,themaximalp(ci|x)thiscanbederivedfrombayestheoremsincep(x)isconstantforallclasses,onlyp(x|ci)p(ci)needstobemaximized,september15,2008,datamining:conceptsandtechniques,17,derivationofnavebayesclassifier,asimplifiedassumption:attributesareconditionallyindependent(i.e.,nodependencerelationbetweenattributes),september15,2008,datamining:conceptsandtechniques,18,navebayesianclassifier:trainingdataset,class:c1:buys_computer=yesc2:buys_computer=nodatasamplex=(age=30,income=medium,student=yescredit_rating=fair),september15,2008,datamining:conceptsandtechniques,19,navebayesianclassifier:anexample,p(ci):p(buys_computer=“yes”)=9/14=0.643p(buys_computer=“no”)=5/14=0.357computep(x|ci)foreachclassp(age=“=30”|buys_computer=“yes”)=2/9=0.222p(age=“1),#ofunitsineachhiddenlayer,and#ofunitsintheoutputlayernormalizingtheinputvaluesforeachattributemeasuredinthetrainingtuplesto0.01.0oneinputunitperdomainvalue,eachinitializedto0output,ifforclassificationandmorethantwoclasses,oneoutputunitperclassisusedonceanetworkhasbeentrainedanditsaccuracyisunacceptable,repeatthetrainingprocesswithadifferentnetworktopologyoradifferentsetofinitialweights,september15,2008,datamining:conceptsandtechniques,41,backpropagation,iterativelyprocessasetoftrainingtuples&comparethenetworkspredictionwiththeactualknowntargetvalueforeachtrainingtuple,theweightsaremodifiedtominimizethemeansquarederrorbetweenthenetworkspredictionandtheactualtargetvaluemodificationsaremadeinthe“backwards”direction:fromtheoutputlayer,througheachhiddenlayerdowntothefirsthiddenlayer,hence“backpropagation”stepsinitializeweights(tosmallrandom#s)andbiasesinthenetworkpropagatetheinputsforward(byapplyingactivationfunction)backpropagatetheerror(byupdatingweightsandbiases)terminatingcondition(whenerrorisverysmall,etc.),september15,2008,datamining:conceptsandtechniques,42,backpropagationandinterpretability,efficiencyofbackpropagation:eachepoch(oneinterationthroughthetrainingset)takeso(|d|*w),with|d|tuplesandwweights,but#ofepochscanbeexponentialton,thenumberofinputs,intheworstcaseruleextractionfromnetworks:networkpruningsimplifythenetworkstructurebyremovingweightedlinksthathavetheleasteffectonthetrainednetworkthenperformlink,unit,oractivationvalueclusteringthesetofinputandactivationvaluesarestudiedtoderiverulesdescribingtherelationshipbetweentheinputandhiddenunitlayerssensitivityanalysis:assesstheimpactthatagiveninputvariablehasonanetworkoutput.theknowledgegainedfromthisanalysiscanberepresentedinrules,september15,2008,datamining:conceptsandtechniques,43,chapter6.classificationandprediction,whatisclassification?whatisprediction?issuesregardingclassificationandpredictionclassificationbydecisiontreeinductionbayesianclassificationrule-basedclassificationclassificationbybackpropagation,supportvectormachines(svm)associativeclassificationlazylearners(orlearningfromyourneighbors)otherclassificationmethodspredictionaccuracyanderrormeasuresensemblemethodsmodelselectionsummary,september15,2008,datamining:conceptsandtechniques,44,svmsupportvectormachines,anewclassificationmethodforbothlinearandnonlineardataitusesanonlinearmappingtotransformtheoriginaltrainingdataintoahigherdimensionwiththenewdimension,itsearchesforthelinearoptimalseparatinghyperplane(i.e.,“decisionboundary”)withanappropriatenonlinearmappingtoasufficientlyhighdimension,datafromtwoclassescanalwaysbeseparatedbyahyperplanesvmfindsthishyperplaneusingsupportvectors(“essential”trainingtuples)andmargins(definedbythesupportvectors),september15,2008,datamining:conceptsandtechniques,45,svmhistoryandapplications,vapnikandcolleagues(1992)groundworkfromvapnik&chervonenkisstatisticallearningtheoryin1960sfeatures:trainingcanbeslowbutaccuracyishighowingtotheirabilitytomodelcomplexnonlineardecisionboundaries(marginmaximization)usedbothforclassificationandpredictionapplications:handwrittendigitrecognition,objectrecognition,speakeridentification,benchmarkingtime-seriespredictiontests,september15,2008,datamining:conceptsandtechniques,46,svmgeneralphilosophy,september15,2008,datamining:conceptsandtechniques,47,svmmarginsandsupportvectors,september15,2008,datamining:conceptsandtechniques,48,svmwhendataislinearlyseparable,m,letdatadbe(x1,y1),(x|d|,y|d|),wherexiisthesetoftrainingtuplesassociatedwiththeclasslabelsyithereareinfinitelines(hyperplanes)separatingthetwoclassesbutwewanttofindthebestone(theonethatminimizesclassificationerroronunseendata)svmsearchesforthehyperplanewiththelargestmargin,i.e.,maximummarginalhyperplane(mmh),september15,2008,datamining:conceptsandtechniques,49,svmlinearlyseparable,aseparatinghyperplanecanbewrittenaswx+b=0wherew=w1,w2,wnisaweightvectorandbascalar(bias)for2-ditcanbewrittenasw0+w1x1+w2x2=0thehyperplanedefiningthesidesofthemargin:h1:w0+w1x1+w2x21foryi=+1,andh2:w0+w1x1+w2x21foryi=1anytrainingtuplesthatfallonhyperplanesh1orh2(i.e.,thesidesdefiningthemargin)aresupportvectorsthisbecomesaconstrained(convex)quadraticoptimizationproblem:quadraticobjectivefunctionandlinearconstraintsquadraticprogramming(qp)lagrangianmultipliers,september15,2008,datamining:conceptsandtechniques,50,whyissvmeffectiveonhighdimensionaldata?,thecomplexityoftrainedclassifierischaracterizedbythe#ofsupportvectorsratherthanthedimensionalityofthedatathesupportvectorsaretheessentialorcriticaltrainingexamplestheylieclosesttothedecisionboundary(mmh)ifallothertrainingexamplesareremovedandthetrainingisrepeated,thesameseparatinghyperplanewouldbefoundthenumberofsupportvectorsfoundcanbeusedtocomputean(upper)boundontheexpectederrorrateofthesvmclassifier,whichisindependentofthedatadimensionalitythus,ansvmwithasmallnumberofsupportvectorscanhavegoodgeneralization,evenwhenthedimensionalityofthedataishigh,september15,2008,datamining:conceptsandtechniques,51,svmlinearlyinseparable,transformtheoriginalinputdataintoahigherdimensionalspacesearchforalinearseparatinghyperplaneinthenewspace,september15,2008,datamining:conceptsandtechniques,52,svmkernelfunctions,insteadofcomputingthedotproductonthetransformeddatatuples,itismathematicallyequivalenttoinsteadapplyingakernelfunctionk(xi,xj)totheoriginaldata,i.e.,k(xi,xj)=(xi)(xj)typicalkernelfunctionssvmcanalsobeusedforclassifyingmultiple(2)classesandforregressionanalysis(withadditionaluserparameters),september15,2008,datamining:conceptsandtechniques,53,scalingsvmbyhierarchicalmicro-clustering,svmisnotscalabletothenumberofdataobjectsintermsoftrainingtimeandmemoryusage“classifyinglargedatasetsusingsvmswithhierarchicalclustersproblem”byhwanjoyu,jiongyang,jiaweihan,kdd03cb-svm(clustering-basedsvm)givenlimitedamountofsystemresources(e.g.,memory),maximizethesvmperformanceintermsofaccuracyandthetrainingspeedusemicro-clusteringtoeffectivelyreducethenumberofpointstobeconsideredatderivingsupportvectors,de-clustermicro-clustersnear“candidatevector”toensurehighclassificationaccuracy,september15,2008,datamining:conceptsandtechniques,54,cb-svm:clustering-basedsvm,trainingdatasetsmaynotevenfitinmemoryreadthedatasetonce(minimizingdiskaccess)constructastatisticalsummaryofthedata(i.e.,hierarchicalclusters)givenalimitedamountofmemorythestatisticalsummarymaximizesthebenefitoflearningsvmthesummaryplaysaroleinindexingsvmsessenceofmicro-clustering(hierarchicalindexingstructure)usemicro-clusterhierarchicalindexingstructureprovidefinersamplesclosertotheboundaryandcoarsersamplesfartherfromtheboundaryselectivede-clusteringtoensurehighaccuracy,september15,2008,datamining:conceptsandtechniques,55,cf-tree:hierarchicalmicro-cluster,september15,2008,datamining:conceptsandtechniques,56,cb-svmalgorithm:outline,constructtwocf-treesfrompositiveandnegativedatasetsindependentlyneedonescanofthedatasettrainansvmfromthecentroidsoftherootentriesde-clustertheentriesneartheboundaryintothenextlevelthechildrenentriesde-clusteredfromtheparententriesareaccumulatedintothetrainingsetwiththenon-declusteredparententriestrainansvmagainfromthecentroidsoftheentriesinthetrainingsetrepeatuntilnothingisaccumulated,september15,2008,datamining:conceptsandtechniques,57,selectivedeclustering,cftreeisasuitablebasestructureforselectivedeclusteringde-clusteronlytheclustereisuchthatdiri2values,kbitscanbeusedbasedonthenotionofsurvivalofthefittest,anewpopulationisformedtoconsistofthefittestrulesandtheiroffspringsthefitnessofaruleisrepresentedbyitsclassificationaccuracyonasetoftrainingexamplesoffspringsaregeneratedbycrossoverandmutationtheprocesscontinuesuntilapopulationpevolveswheneachruleinpsatisfiesaprespecifiedthresholdslowbuteasilyparallelizable,september15,2008,datamining:conceptsandtechniques,76,roughsetapproach,roughsetsareusedtoapproximatelyor“roughly”defineequivalentclassesaroughsetforagivenclasscisapproximatedbytwosets:alowerapproximation(certaintobeinc)andanupperapproximation(cannotbedescribedasnotbelongingtoc)findingtheminimalsubsets(reducts)ofattributesforfeaturereductionisnp-hardbutadiscernibilitymatrix(whichstoresthedifferencesbetweenattributevaluesforeachpairofdatatuples)isusedtoreducethecomputationintensity,september15,2008,datamining:conceptsandtechniques,77,fuzzysetapproaches,fuzzylogicusestruthvaluesbetween0.0and1.0torepresentthedegreeofmembership(suchasusingfuzzymembershipgraph)attributevaluesareconvertedtofuzzyvaluese.g.,incomeismappedintothediscretecategorieslow,medium,highwithfuzzyvaluescalculatedforagivennewsample,morethanonefuzzyvaluemayapplyeachapplicablerulecontributesavoteformembershipinthecategoriestypically,thetruthvaluesforeachpredictedcategoryaresummed,andthesesumsarecombined,september15,2008,datamining:conceptsandtechniques,78,chapter6.classificationandprediction,whatiscla