An Introduction to Multivariate Modeling Techniques Zoo.ppt

1 modelbuildingtraining maxkuhnkjelljohnsonglobalnonclinicalstatistics 2 overview typicaldatascenariosexampleswe llbeusinggeneralapproachestomodelbuildingdatapre processingregression typemodelsclassification typemodelsotherconsiderations 3 typicaldata responsemaybecontinuousorcategoricalpredictorsmaybecontinuous count and orbinarydenseorsparseobservedand orcalculated 4 predictivemodels whatisa predictivemodel amodelwhoseprimarypurposeisforprediction asopposedtoinference wewouldliketoknowwhythemodelworks aswellastherelationshipbetweenpredictorsandtheoutcome butthesearesecondaryexamples blood glucosemonitoring spamdetection computationalchemistry etc 5 whataretheynotgoodfor theyarenotasubstituteforsubjectspecificknowledgescience hard yikes models easy let sdotheseinstead tomakeagoodmodelthatpredictswellonfuturesamples youneedtoknowalotaboutyourpredictorsandhowtheyrelatetoeachotherthemechanismthatgeneratedthedata sampling technologyetc 6 whataretheynotgoodfor anexample anoncologistcollectssomedatafromasmallclinicaltrialandwantsamodelthatwouldusegeneexpressiondatatopredicttherapeuticresponse beneficialornot in4typesofcancertherewereabout54kpredictorsanddatawascollectedon 20subjectsifthereisalotofknowledgeofhowthetherapyworks pathwaysetc someeffortmustbeputintousingthatinformationtohelpbuildthemodel 7 thebigpicture intheend predictivemodeling isnotasubstituteforintuition butacompliment ianayres insupercrunchers 8 references statisticalmodeling thetwocultures byleobreiman statisticalscience vol16 3 2001 199 231 theelementsofstatisticallearningbyhastie tibshiraniandfriedmanregressionmodelingstrategiesbyharrellsupercrunchersbyayres 9 regressionmethods multiplelinearregressionpartialleastsquaresneuralnetworksmultivariateadaptiveregressionsplinessupportvectormachinesregressiontreesensemblesoftrees bagging boosting andrandomforests 10 classificationmethods discriminantanalysisframeworklinear quadratic regularized flexible andpartialleastsquaresdiscriminantanalysismodernclassificationmethodsclassificationtreesensemblesoftreesboostingandrandomforestsneuralnetworkssupportvectormachinesk nearestneighborsnaivebayes 11 interestingmodelswedon thavetimefor l1penaltymethodsthelasso theelasticnet nearestshrunkencentroidsotherboostedmodelslinearmodels generalizedadditivemodels etcothermodels conditionalinferencetrees c4 5 c5 cubist othertreemodelslearnedvectorquantizationself organizingmapsactivelearningtechniques 12 exampledatasets 13 bostonhousingdata thisisaclassicbenchmarkdatasetforregression itincludeshousingdatafor506censustractsofbostonfromthe1970census crim percapitacrimerateindus proportionofnon retailbusinessacrespertownchas charlesriverdummyvariable 1iftractboundsriver 0otherwise nox nitricoxidesconcentrationrm averagenumberofroomsperdwellingage proportionofowner occupiedunitsbuiltpriorto1940 dis weighteddistancestofivebostonemploymentcentersrad indexofaccessibilitytoradialhighwaystax full valueproperty taxrateptratio pupil teacherratiobytownb proportionofminoritiesmedv medianvaluehomes outcome 14 toyclassificationexample asimulateddatasetwillbeusedtodemonstrateclassificationmodelstwopredictorswithacorrelationcoefficientof0 5weresimulatedtwoclassesweresimulated active and inactive aprobabilitymodelwasusedtoassignaprobabilityofbeingactivetoeachsamplethe25 50 and75 probabilitylinesareshownontheright 15 toyclassificationexample theclasseswererandomlyassignedbasedontheprobabilitythetrainingdatahad250compounds plotonright thetestsetalsocontained250compoundswithtwopredictors theclassboundariescanbeshownforeachmodelthiscanbeasignificantaidinunderstandinghowthemodelswork butweacknowledgehowunrealisticthissituationis 16 modelbuildingtraining generalstrategies 17 objective toconstructamodelofpredictorsthatcanbeusedtopredictaresponse 18 modelbuildingsteps commonstepsduringmodelbuildingare estimatingmodelparameters i e trainingmodels determiningthevaluesoftuningparametersthatcannotbedirectlycalculatedfromthedatacalculatingtheperformanceofthefinalmodelthatwillgeneralizetonewdatathemodelerhasafiniteamountofdata whichtheymust spend toaccomplishthesestepshowdowe spend thedatatofindanoptimalmodel 19 spending data wetypically spend dataontrainingandtestdatasetstrainingset thesedataareusedtoestimatemodelparametersandtopickthevaluesofthecomplexityparameter s forthemodel testset akavalidationset thesedatacanbeusedtogetanindependentassessmentofmodelefficacy theyshouldnotbeusedduringmodeltraining themoredatawespend thebetterestimateswe llget providedthedataisaccurate givenafixedamountofdata toomuchspentintrainingwon tallowustogetagoodassessmentofpredictiveperformance wemayfindamodelthatfitsthetrainingdataverywell butisnotgeneralizable overfitting toomuchspentintestingwon tallowustogetagoodassessmentofmodelparameters 20 methodsforcreatingatestset howshouldwesplitthedataintoatrainingandtestset often therewillbeascientificrationalforthesplitandinothercases thesplitscanbemadeempirically severalempiricalsplittingoptions completelyrandomstratifiedrandommaximumdissimilarityinpredictorspace 21 creatingatestset completelyrandomsplits acompletelyrandom cr splitrandomlypartitionsthedataintoatrainingandtestsetforlargedatasets acrsplithasverylowbiastowardsanycharacteristic predictororresponse forclassificationproblems acrsplitisappropriatefordatathatisbalancedintheresponsehowever acrsplitisnotappropriateforunbalanceddataacrsplitmayselecttoofewobservations andperhapsnone ofthelessfrequentclassintooneofthesplits 22 creatingatestset stratifiedrandomsplits astratifiedrandomsplitmakesarandomsplitwithinstratificationgroupsinclassification theclassesareusedasstratainregression groupsbasedonthequantilesoftheresponseareusedasstratastratificationattemptstopreservethedistributionoftheoutcomebetweenthetrainingandtestsetsasrsplitismoreappropriateforunbalanceddata 23 over fitting over fittingoccurswhenamodelhasextremelygoodpredictionforthetrainingdatabutpredictspoorlywhenthedataareslightlyperturbednewdata i e testdata areusedcomplexregressionandclassificationmodelsassumethattherearepatternsinthedata withoutsomecontrolmanymodelscanfindveryintricaterelationshipsbetweenthepredictorandtheresponsethesepatternsmaynotbevalidfortheentirepopulation 24 over fittingexample theplotsbelowshowclassificationboundariesfortwomodelsbuiltonthesamedataoneofthemisover fit predictorb predictorb predictora predictora 25 over fittinginregression historically weevaluatethequalityofaregressionmodelbyit smeansquarederror supposethatarepredictionfunctionisparameterizedbysomevector 26 over fittinginregression msecanbedecomposedintothreeterms irreduciblenoisesquaredbiasoftheestimatorfromit sexpectedvaluethevarianceoftheestimatorthebiasandvarianceareinverselyrelatedasoneincreases theotherdecreasesdifferentratesofchange 27 over fittinginregression whenthemodelunder fits thebiasisgenerallyhighandthevarianceislowover fittingistypicallycharacterizedbyhighvariance lowbiasestimatorsinmanycases smallincreasesinbiasresultinlargedecreasesinvariance 28 over fittinginregression generally controllingthemseyieldsagoodtrade offbetweenover andunder fittingasimilarstatementcanbemadeaboutclassificationmodels althoughthemetricsaredifferent i e notmse howcanweaccuratelyestimatethemsefromthetrainingdata thena vemsefromthetrainingdatacanbeaverypoorestimateresamplingcanhelpestimatethesemetrics 29 howdoweestimateover fitting somemodelshavespecific knobs tocontrolover fittingneighborhoodsizeinnearestneighbormodelsisanexamplethenumberifsplitsinatreemodeloften poorchoicesfortheseparameterscanresultinover fittingresamplingthetrainingcompoundsallowsustoknowwhenwearemakingpoorchoicesforthevaluesoftheseparameters 30 howdoweestimateover fitting resamplingonlyaffectsthetrainingdatathetestsetisnotusedinthisprocedureresamplingmethodstryto embedvariation inthedatatoapproximatethemodel sperformanceonfuturecompoundscommonresamplingmethods k foldcrossvalidationleavegroupoutcrossvalidationbootstrapping 31 k foldcrossvalidation here werandomlysplitthedataintokblocksofroughlyequalsizeweleaveoutthefirstblockofdataandfitamodel thismodelisusedtopredicttheheld outblockwecontinuethisprocessuntilwe vepredictedallkhold outblocksthefinalperformanceisbasedonthehold outpredictions 32 k foldcrossvalidation theschematicbelowshowstheprocessfork 3groups kisusuallytakentobe5or10leaveoneoutcross validationhaseachsampleasablock 33 leavegroupoutcrossvalidation arandomproportionofdata say80 areusedtotrainamodeltheremainderisusedtopredictperformancethisprocessisrepeatedmanytimesandtheaverageperformanceisused 34 bootstrapping bootstrappingtakesarandomsamplewithreplacementtherandomsampleisthesamesizeastheoriginaldatasetcompoundsmaybeselectedmorethanonceeachcompoundhasa63 2 changeofshowingupatleastoncesomesampleswon tbeselectedthesesampleswillbeusedtopredictperformancetheprocessisrepeatedmultipletimes say30 35 thebootstrap withbootstrapping thenumberofheld outsamplesisrandomsomemodels suchasrandomforest usebootstrappingwithinthemodelingprocesstoreduceover fitting 36 trainingmodelswithtuningparameters asingletraining testsplitisoftennotenoughformodelswithtuningparameterswemustuseresamplingtechniquestogetgoodestimatesofmodelperformanceovermultiplevaluesoftheseparameterswepickthecomplexityparameter s withthebestperformanceandre fitthemodelusingallofthedata 37 simulateddataexample let sfitanearestneighborsmodeltothesimulatedclassificationdata theoptimalnumberofneighborsmustbechosenifweuseleavegroupoutcross validationandsetaside20 wewillfitmodelstoarandom200samplesandpredict50samples30iterationswereusedwe lltrainover11oddvaluesforthenumberofneighborswealsohavea250pointtestset 38 toydataexample theplotontherightshowstheclassificationaccuracyforeachvalueofthetuningparameterthegreypointsarethe30resampledestimatestheblacklineshowstheaverageaccuracythebluelineisthe250sampletestsetitlookslike7ormoreneighborsisoptimalwithanestimatedaccuracyof86 39 toydataexample whatifwedidn tresampleandusedthewholedataset theplotontherightshowstheaccuracyacrossthetuningparametersthiswouldpickamodelthatover fitsandhasoptimisticperformance 40 modelbuildingtraining datapre processing 41 whypre process inordertogeteffectiveandstableresults manymodelsrequirecertainassumptionsaboutthedatathisismodeldependentwewilllisteachmodel spre processingrequirementsattheendingeneral pre processingrarelyhurtsmodelperformance butcouldmakemodelinterpretationmoredifficult 42 commonpre processingsteps formostmodels weapplythreepre processingprocedures removalofpredictorswithvarianceclosetozeroeliminationofhighlycorrelatedpredictorscenteringandscalingofeachpredictor 43 zerovariancepredictors mostmodelsrequirethateachpredictorhaveatleasttwouniquevalueswhy apredictorwithonlyoneuniquevaluehasavarianceofzeroandcontainsnoinformationabouttheresponse itisgenerallyagoodideatoremovethem 44 nearzerovariance predictors additionally ifthedistributionsofthepredictorsareverysparse thiscanhaveadrasticeffectonthestabilityofthemodelsolutionzerovariancedescriptorscouldbeinducedduringresamplingbutwhatdoesa nearzerovariance predictorlooklike 45 nearzerovariance predictor therearetwoconditionsforan nzv predictoralownumberofpossiblevalues andahighimbalanceinthefrequencyofthevaluesforexample alownumberofpossiblevaluescouldoccurbyusingfingerprintsaspredictorsonlytwopossiblevaluescanoccur 0or1 butwhatifthereare999zerovaluesinthedataandasinglevalueof1 thisisahighlyunbalancedcaseandcouldbetrouble 46 nzvexample incomputationalchemistrywecreatedpredictorsbasedonstructuralcharacteristicsofcompounds asanexample thedescriptor nr11 isthenumberof11 memberringsthetabletotherightisthedistributionofnr11fromatrainingsetthedistinctvaluepercentageis5 535 0 0093thefrequencyratiois501 23 21 8 47 detectingnzvs twocriteriafordetectingnzvsarethediscretevaluepercentagedefinedasthenumberofuniquevaluesdividedbythenumberofobservationsrule of thumb discretevaluepercentage19couldindicateaproblemifbothcriteriaareviolated theneliminatethepredictor 48 highlycorrelatedpredictors somemodelscanbenegativelyaffectedbyhighlycorrelatedpredictorscertaincalculations e g matrixinversion canbecomeseverelyunstablehowcanwedetectthesepredictors varianceinflationfactor vif inlinearregressionor alternativelycomputethecorrelationmatrixofthepredictorspredictorswith absolute pair wisecorrelationsaboveathresholdcanbeflaggedforremovalrule of thumbthreshold 0 85 49 highlycorrelatedpredictorsandresampling recallthatresamplingslightlyperturbsthetrainingdatasettoincreasevariationifamodelisadverselyaffectedbyhighcorrelationsbetweenpredictors theresamplingperformanceestimatescanbepoorincomparisontothetestsetinthiscase resamplingdoesabetterjobatpredictinghowthemodelworksonfuturesamples 50 centeringandscaling standardizingthepredictorscangreatlyimprovethestabilityofmodelcalculations moreimportantly thereareseveralmodels e g partialleastsquares thatimplicitlyassumethatallofthepredictorsareonthesamescaleapartfromthelossoftheoriginalunits thereisnorealdownsideofcenteringandscaling 51 modelbuildingtraining regression typemodels 52 setting responseiscontinuous 53 objective toconstructamodelofpredictorsthatcanbeusedtopredictaresponse 54 regressionmethods multiplelinearregressionpartialleastsquaresneuralnetworksmultivariateadaptiveregressionsplinessupportvectormachinesregressiontreesensemblesoftrees bagging boosting andrandomforestseachofthesemethodsseektofindarelationshipbetweenthepredictorsandresponsethatminimizeserrorbetweentheobservedandpredictedresponse 55 additivemodels inthebeginningtherewerelinearmodels andhastieandtibshirani 1990 said lettherebegeneralizedadditivemodels andnelderandwedderburn 1972 said lettherebegeneralizedlinearmodels andlinkfunctionsappeared andscatterplotsmoothersandbackfittingalgorithmsappeared 56 familiesofadditivemodels glm gam recursivepartitioning trees boosting randomforests bagging multivariateadaptiveregressionsplines neuralnets supportvectormachines pls flexibility additivitydependsonmodelparameters 57 assessingmodelperformance 58 assessingmodelperformance howwelldoesaregressionmodelperform answeringthisquestiondependsonhowwewanttousethemodel possiblegoalsare tounderstandtherelationshipbetweenthepredictorandtheresponse tousethemodeltopredictfutureobservations response ineithercase wecanuseseveralofdifferentmeasurestoevaluatemodelperformance wewillfocusontwo coefficientofdetermination r2 rootmeansquareerror rmse however thesetofdatathatweusetoevaluateperformancewillchangedependingonourpurpose 59 whichsetofdatatousetoevaluateperformance ifweareonlyinterestedinunderstandingtheunderlyingrelationshipbetweenthepredictorandtheresponse thenwecancomputer2andrmseonthedataforwhichthemodelwasbuilt i ethetrainingdata however thesevalueswillbeoverlyoptimisticofthemodel sabilitytopredictfutureobservations ifweareinterestedinunderstandingthemodel sabilitytopredictfutureobservations thenweneedtocomputer2andrmseondataforwhichthemodelwasnotbuilt i e atestsetorcross validationset foraheld outsetofdata r2iscommonlyreferredtoasq2andrmseiscommonlyreferredtoasrootmeansquaredpredictionerror rmspe 60 rootmeansquarederror rmse androotmeansquaredpredictionerror rmspe rmsemeasurestheaveragedeviationofanobservationtothebest fitplanermspemeasurestheaveragedeviationofanobservationtoitspredictedvalueforthetestorcross validationset n thenumberofobservationsinthetestorcross validationset 61 computingq2 process partitionthedataintoatrainingandtestingset orblockstobeusedfortrainingandtestingbuildthemodelonthetrainingdataandpredictthetestingdataq2 r2oftherelationshipbetweentheobservedandpredictedvaluesforthetestingdata 62 multiplelinearregression aquickreview 63 multiplelinearregression objective findtheplanethroughthedatathatminimizesthesum of squareserror 64 thebestplane tofindthebestplane wesolve whereynx1 xnx p 1 and p 1 x1thebest is 65 aside abitmoreabout xtx xtx isacriticalmatrixformanystatisticalmodelingtechniquesafewfunfacts xtx isproportionaltothecovariancematrix sscontainsthevariancesandcovariancesofallpredictorstechniquesthatdependon xtx alsorequirethatitisinvertible 66 assumptions diagnosticplots 67 whendoesregressionfail whenaplanedoesnotcapturethestructureinthedatawhenthevariance covariancematrixisoverdeterminedrecall theplanethatminimizessseis tofindthebestplane wemustcomputetheinverseofthevariance covariancematrixthevariance covariancematrixisnotalwaysinvertible twocommonconditionsthatcauseittobeuninvertibleare twoormoreofthepredictorsarecorrelated multicollinearity therearemorepredictorsthanobservations 68 a trivial exampleofmulticollinearity supposethatwehaveoneobservation 3 5 andwewishtofindthe best lineforthedata inthisexample thenumberofobservations 1 islessthanthenumberofparameters 2 slopeandintercept whenthenumberofparametersisgreaterthanthenumberofobservations wecanfindaninfinitenumberof best solutions inthepresenceofmulticollinearity thebestsolutionwillbeunstable 69 bostonhousingdata let susealinearregressionmodeltopredictthemedianhousepriceinboston process splitthedataintoatrainingset n 337 andtestingset n 169 forthetrainingset usethebootstraptodeterminethermspeandq2forthetestdatadeterminermspeandq2iftheunderlyingmodelisstable thevaluesofrmspeandq2shouldbesimilarbetweenthebootstrapandtestingdata 70 results theresultsarefairlysimilar atleastwithinthevariationofresamplingonereasonyoumayseedifferences multicollinearitymulticollinearityinthepredictorscanproducesomewhatunstablesolutionsforeachresamplewhenthedataareslightlychanged themodelcandrasticallychangethetestsetisasingle staticsetofdataforverificationthebootstrapestimateofperformancemaybebetterwithcollinearity 71 partialleastsquaresregression 72 solutionsforoverdeterminedcovariancematrices variablereductiontrytoaccomplishthisthroughthepre processingstepspartialleastsquares pls othermethodsapplyageneralizedinverseridgeregression adjuststhevariance covariancematrixsothatwecanfindauniqueinverse principalcomponentregression pcr notrecommended butit sagoodwaytounderstandpls 73 understandingpartialleastsquares principalcomponentsanalysis pcaseekstofindlinearcombinationsofth