Lecture5缺失值处理策略.ppt

1 Lecture5Incompletedata ZiadTaibBiostatistics AZMay3 2011 Outlineoftheproblem MissingvaluesinlongitudinaltrialsisabigissueFirstaimshouldbetoreduceproportionEthicsdictatethatitcan tbeavoidedThereisnomagicmethodtofixitMagnitudeofproblemvariesacrossareas8 weekdepressiontrial 25 50 maydropoutbyfinalvisit12 weekasthmatrial maybeonly5 10 2 Date Name department 3 Outlineofthelecture PartI Missingdata PartII Multipleimputation Example Theanalgesictrial 4 5 Date Name department 6 PartI Missingdata Inrealdatasets like e g surveysandclinicaltrials itisquitecommontohaveobservationswithmissingvaluesforoneormoreinputfeatures Thefirstissueindealingwiththeproblemisdeterminingwhetherthemissingdatamechanismhasdistortedtheobserveddata LittleandRubin 1987 andRubin 1987 distinguishbetweenbasicallythreemissingdatamechanisms Dataaresaidtobemissingatrandom MAR ifthemechanismresultinginitsomissionisindependentofits unobserved value Ifitsomissionisalsoindependentoftheobservedvalues thenthemissingnessprocessissaidtobemissingcompletelyatrandom MCAR Inanyothercasetheprocessismissingnotatrandom MNAR i e themissingnessprocessdependsontheunobservedvalues http www emea europa eu pdfs human ewp 177699EN pdf 1 Introductiontomissingdata 7 Whatismissingdata Themissingnesshidesarealvaluethatisusefulforanalysispurposes Surveyquestions WhatisyourtotalannualincomeforFY2008 Whoareyouvotingforinthe2009electionfortheEuropeanparlament 8 Whatismissingdata Clinicaltrials Start Finish censoredatthispointintime time 9 Missingness Itmatterswhydataaremissing Supposeyouaremodellingweight Y asafunctionofsex X Somerespondentswouldn tdisclosetheirweight soyouaremissingsomevaluesforY Therearethreepossiblemechanismsforthenondisclosure Theremaybenoparticularreasonwhysomerespondentstoldyoutheirweightsandothersdidn t Thatis theprobabilitythatYismissingmayhasnorelationshiptoXorY InthiscaseourdataismissingcompletelyatrandomOnesexmaybelesslikelytodiscloseitsweight Thatis theprobabilitythatYismissingdependsonlyonthevalueofX SuchdataaremissingatrandomHeavy orlight peoplemaybelesslikelytodisclosetheirweight Thatis theprobabilitythatYismissingdependsontheunobservedvalueofYitself Suchdataarenotmissingatrandom 10 Missingdatapatterns mechanisms Pattern Whichvaluesaremissing Mechanism Ismissingnessrelatedtotheresponse Yi Ri Datamatrix withCOMPLETEDATA Rij 1 Yijmissing0 Yijobserved Rij Missingdataindicatormatrix ObservedpartofY MissingpartofY 11 Missingdatapatterns mechanisms Pattern concernsthedistributionofR Mechanism concernsthedistributionofRgivenY Rubin Biometrika1976 distinguishesbetween MissingCompletelyatRandom MCAR P R Y P R forallY 12 MissingAtRandom MAR Whatarethemostgeneralconditionsunderwhichavalidanalysiscanbedoneusingonlytheobserveddata andnoinformationaboutthemissingnessvaluemechanism Theanswertothisiswhen giventheobserveddata themissingnessmechanismdoesnotdependontheunobserveddata Mathematically ThisistermedMissingAtRandom andisequivalenttosayingthatthebehaviouroftwounitswhoshareobservedvalueshavethesamestatisticalbehaviourontheotherobservations whetherobservedornot 13 Asunits1and2havethesamevalueswherebothareobserved giventheseobservedvalues underMAR variables3 5and6fromunit2havethesamedistribution NBnotthesamevalue asvariables3 5and6fromunit1 NotethatunderMARtheprobabilityofavaluebeingmissingwillgenerallydependonobservedvalues soitdoesnotcorrespondtotheintuitivenotionof random Theimportantideaisthatthemissingvaluemechanismcanbeexpressedsolelyintermsofobservationsthatareobserved Unfortunately thiscanrarelybedefinitivelydeterminedfromthedataathand Example 14 IfdataareMCARorMAR youcanignorethemissingdatamechanismandusemultipleimputationandmaximumlikelihood IfdataareNMAR youcan tignorethemissingdatamechanism twoapproachestoNMARdataareselectionmodelsandpatternmixture 15 SupposeYisweightinpounds ifsomeonehasaheavyweight theymaybelessinclinedtoreportit SothevalueofYaffectswhetherYismissing thedataareNMAR Twopossibleapproachesforsuchdataareselectionmodelsandpatternmixture Selectionmodels Inaselectionmodel yousimultaneouslymodelYandtheprobabilitythatYismissing Unfortunately anumberofpracticaldifficultiesareoftenencounteredinestimatingselectionmodels Patternmixture Rubin1987 WhendataisNMAR analternativetoselectionmodelsismultipleimputationwithpatternmixture Inthisapproach youperformmultipleimputationsunderavarietyofassumptionsaboutthemissingdatamechanism Inordinarymultipleimputation youassumethatthosepeoplewhoreporttheirweightsaresimilartothosewhodon t Inapattern mixturemodel youmayassumethatpeoplewhodon treporttheirweightsareanaverageof20poundsheavier Thisisofcourseanarbitraryassumption theideaofpatternmixtureistotryoutavarietyofplausibleassumptionsandseehowmuchtheyaffectyourresults Patternmixtureisamorenatural flexible andinterpretableapproach 16 Simpleanalysisstrategies 1 CompleteCase CC analysis Advantages CompleteCases discard Easy Doesnotinventdata Disadvantages Inefficient Discardingdataisbad CCareoftenbiasedsamples Whensomevariablesarenotobservedforsomeoftheunits onecanomittheseunitsfromtheanalysis Theseso called completecases arethenanalyzedastheyare 17 Analysisstrategies 2 Analyzeasincomplete summarymeasures GEE Advantages CompleteCases Advantages Doesnotinventdata Disadvantages Restrictedinwhatyoucaninfer Maximumlikelihoodmethodsmaybecomputationallyintensiveornotfeasibleforcertaintypesofmodels 18 Analysisstrategies 3 Analysisaftersingleimputation Advantages CompleteCases imputation Rectangularfile Goodformultipleusers Disadvantages Na veimputationsnotgood Inventsdata inferenceisdistortedbytreatingimputationsasthetruth 19 Simplemethodsofanalysisofincompletedata cc locf 20 Variousstrategies 21 Notation DROPOUT 22 Ignorability InalikelihoodsettingthetermignorableisoftenusedtorefertoMARmechanism Itisthemechanismwhichisignorable notthemissingdata 23 Ignorability 24 Directlikelihoodmaximisation 25 Example1 Growthdata 26 27 Growthdata 28 29 Example Thedepressiontrial Patientsareevaluatedbothpretreatmentandposttreatmentwiththe17 itemHamiltonRatingScaleforDepression Ham D 17 30 Thedepressiontrial 31 32 5 PartII Multipleimputation 33 Datasetwithmissingvalues Result Completedset 34 35 Generalprinciples 36 Informaljustification 37 Thealgorithm 38 Poolinginformation 39 Hypothesistesting 40 41 MIinpractice 42 MIinpractice Asimulation basedapproachtomissingdata 1 GenerateM 1plausibleversionsof CompleteCases imputationforMthdataset 2 AnalyzeeachoftheMdatasetsbystandardcomplete datamethods 3 CombinetheresultsacrosstheMdatasets M 3 5isusuallyOK 43 MIinpractice Step1 GenerateM 1plausibleversionsofviasoftware i e obtainMdifferentdatasets Anassumptionwemake thedataareMCARorMAR i e themissingdatamechanismisignorable Shoulduseasmuchinformationisavailableinordertoachievethebestimputation Ifthepercentageofmissingdataishigh weneedtoincreaseM 44 Howmanydatasetstocreate TheefficiencyofanestimatorbasedonMimputationsis where isthefractionofmissinginformation Efficiencyofmultipleimputation M0 10 30 50 70 939791868177598949188851099979593922010099989796 45 MIinpractice Step2 AnalyzeeachoftheMdatasetsbystandardcomplete datamethods Letbbetheparameterofinterest istheestimateofbfromthecomplete dataanalysisofthemthdataset m 1 M isthevarianceoffromtheanalysisofthemthdataset 46 MIinpractice Step3 CombinetheresultsacrosstheMdatasets isthecombinedinferenceforb Varianceforis between within 47 Software 1 JoeSchafer ssoftwarefromhiswebsite 0 http www stat psu edu 7Ejls misoftwa htmlSchaferhaswrittenpubliclyavailablesoftwareprimarilyforS plus Thereisastand aloneWindowspackagefordatathatismultivariatenormal Thiswebsitecontainsmuchusefulinformationregardingmultipleimputation 48 Software 2 SASsoftware experimental ItispartofSAS STATversion8 02SASinstitutepaperonmultipleimputation givesanexampleandSAScode 49 Software 3 SOLASversion3 0 1K http www statsol ie index php pageID 5Windowsbasedsoftwarethatperformsdifferenttypesofimputation Hot deckimputation PredictiveOLS discriminantregression Nonparametricbasedonpropensityscores LastvaluecarriedforwardWillalsocombineparameterresultsacrosstheManalyses 50 MIAnalysisoftheOrthodonticGrowthData 51 Propertiesofmethods MCAR drop outindependentofresponseCCisvalid thoughitignoresinformationLOCFisvalidiftherearenotrendswithtimeMAR drop outdependsonlyonobservationsCC LOCF GEEinvalidMI MNLM weightedGEEvalidMNAR drop outdependsalsoonunobservedCC LOCF GEE MI MNLMinvalidSM PMMvalidif uncheckable assumptionstrue 52 References Allison P 2002 Missingdata ThousandOaks CA Sage greenback Horton NJ Lipsitz SR 2001 Multipleimputationinpractice Comparisonofsoftwarepackagesforregressionmodelswithmissingvariables TheAmericanStatistician55 3 244 254 Little R J A 1992 RegressionwithmissingX s Areview JournaloftheAmericanStatisticalAssociation87 420 1227 1237 RoderickJ A LittleandDonaldB Rubin 2002 StatisticalAnalysiswithMissingData 2ndeditionApril2002 ApplicationsofModernMissingDataMethods byRoderick