MOOSE Statement Meta-analysis of observational studies in epidemiology：A Proposal for Reporting

CONSENSUS STATEMENT Meta analysis of Observational Studies in Epidemiology A Proposal for Reporting Donna F Stroup PhD MSc Jesse A Berlin ScD Sally C Morton PhD Ingram Olkin PhD G David Williamson PhD Drummond Rennie MD David Moher MSc Betsy J Becker PhD Theresa Ann Sipe PhD Stephen B Thacker MD MSc for the Meta analysis Of Observational Studies in Epidemiology MOOSE Group B ECAUSEOFPRESSUREFORTIMELY andinformeddecisionsinpub lichealthandmedicineandthe explosionofinformationinthe scientificliterature researchresultsmust be synthesized to answer urgent ques tions 1 4Principles of evidence based methods to assess the effectiveness of healthcareinterventionsandsetpolicy arecitedincreasingly 5Meta analysis a systematic approach to identifying ap praising synthesizing and ifappropri ate combining the results of relevant studies to arrive at conclusions about a bodyofresearch hasbeenappliedwith increasingfrequencytorandomizedcon trolledtrials RCTs whichareconsid ered to provide the strongest evidence regarding an intervention 6 7 However inmanysituationsrandom ized controlled designs are not fea sible and only data from observational studiesareavailable 8Here wedefinean observationalstudyasanetiologicoref fectivenessstudyusingdatafromanex istingdatabase across sectionalstudy acaseseries acase controldesign ade sign with historical controls or a co hortdesign 9Observationaldesignsmay lacktheexperimentalelementofaran dom allocation to an intervention and rely on studies of association between changesordifferences in 1characteris tic eg anexposureorintervention and changesordifferencesinanoutcomeof Author Affiliations Centers for Disease Control and Prevention Atlanta Ga Drs Stroup Williamson and Thacker University of Pennsylvania School of Medi cine Philadelphia DrBerlin RANDCorporation Santa Monica DrMorton UniversityofCalifornia SanFran cisco Dr Rennie Stanford University Stanford Dr Olkin Calif JAMA Chicago Ill Dr Rennie Thomas C Chalmers Centre for Systematic Reviews Chil dren s Hospital of Eastern Ontario Research Insti tute Ottawa Mr Moher Michigan State Univer sity East Lansing Dr Becker and Georgia State University Atlanta Dr Sipe AcompletelistofmembersoftheMOOSEGroupap pears at the end of this article CorrespondingAuthorandReprints DonnaF Stroup PhD MSc Centers for Disease Control and Preven tion 1600 Clifton Rd NE Mail Stop C08 Atlanta GA 30333 e mail dfs2 cdc gov Objective Becauseofthepressurefortimely informeddecisionsinpublichealthand clinical practice and the explosion of information in the scientific literature research results must be synthesized Meta analyses are increasingly used to address this prob lem and they often evaluate observational studies A workshop was held in Atlanta Ga in April 1997 to examine the reporting of meta analyses of observational studies and to make recommendations to aid authors reviewers editors and readers Participants Twenty sevenparticipantswereselectedbyasteeringcommittee based onexpertiseinclinicalpractice trials statistics epidemiology socialsciences andbiomedi calediting Deliberationsoftheworkshopwereopentootherinterestedscientists Fund ing for this activity was provided by the Centers for Disease Control and Prevention Evidence We conducted a systematic review of the published literature on the con duct and reporting of meta analyses in observational studies using MEDLINE Educa tionalResearchInformationCenter ERIC PsycLIT andtheCurrentIndextoStatistics We also examined reference lists of the 32 studies retrieved and contacted experts in thefield Participantswereassignedtosmall groupdiscussionsonthesubjectsofbias searchingandabstracting heterogeneity studycategorization andstatisticalmethods Consensus Process From the material presented at the workshop the authors developedachecklistsummarizingrecommendationsforreportingmeta analysesofob servational studies The checklist and supporting evidence were circulated to all confer ence attendees and additional experts All suggestions for revisions were addressed Conclusions The proposed checklist contains specifications for reporting of meta analyses of observational studies in epidemiology including background search strat egy methods results discussion and conclusion Use of the checklist should improve the usefulness of meta analyses for authors reviewers editors readers and decision makers An evaluation plan is suggested and research areas are explored JAMA 2000 283 2008 2008JAMA April 19 2000 Vol 283 No 15 2000 American Medical Association All rights reserved at Second Military Medical Univ on March 31 2009 Downloaded from interest These designs have long been used in the evaluation of educational programs10and exposures that might causediseaseorinjury 11Studiesofrisk factors generally cannot be random izedbecausetheyrelatetoinherenthu mancharacteristicsorpractices andex posing subjects to harmful risk factors isunethical 12Attimes clinicaldatamay besummarizedinordertodesignaran domized comparison 13Observational data may also be needed to assess the effectiveness of an intervention in a community as opposed to the special setting of a controlled trial 14Thus a clear understanding of the advantages and limitations of statistical syntheses of observational data is needed 15 Althoughmeta analysisrestrictedto RCTs is usually preferred to meta analysesofobservationalstudies 16 18the number of published meta analyses concerning observational studies in healthhasincreasedsubstantiallydur ing the past 4 decades 678 in 1955 1992 525in1992 1995 andmorethan 400 in 1996 alone 19 While guidelines for meta analyses have been proposed many are written fromthemeta analyst s author s rather than from the reviewer s editor s or reader s perspective20and restrict at tentiontoreportingofmeta analysesof RCTs 21 22Meta analyses of observa tional studies present particular chal lenges because of inherent biases and differencesinstudydesigns23 yet they may provide a tool for helping to un derstand and quantify sources of vari ability in results across studies 24 We describe here the results of a workshop held in Atlanta Ga in April 1997 toexamineconcernsregardingthe reportingofMeta analysisOfObserva tional Studies in Epidemiology MOOSE Thisarticlesummarizesde liberations of 27 participants the MOOSE group of evidence leading to recommendationsregardingthereport ingofmeta analyses Meta analysisofin dividual level data from different stud ies sometimescalled pooledanalysis or meta analysis of individual patient data 25 26hasuniquechallengesthatwe will not address here We propose a checklist of items for reporting that builds on similar activities for RCTs22 and is intended for use by authors re viewers editors and readers of meta analyses of observational studies METHODS We conducted a systematic review of the published literature on the con duct and reporting of meta analyses in observational studies Databases searched included MEDLINE Educa tional Resources Information Center PsycLIT http www wesleyan edu libr and the Current Index to Statis tics In addition we examined refer ence lists and contacted experts in the field Weusedthe32articlesretrieved togeneratetheconferenceagendaand set topics of bias searching and ab stracting heterogeneity studycatego rization andstatisticalmethods Wein vitedexpertsinmeta analysisfromthe fields of clinical practice trials statis tics epidemiology socialsciences and biomedical editing Theworkshopincludedanoverview of the quality of reporting of meta analyses in education and the social sciences Plenarytalksweregivenonthe topicssetbytheconferenceagenda For each of 2 sessions workshop partici pants were assigned to 1 of 5 small dis cussion groups organized around the topic areas For each group 1 of the authors served as facilitator and a recorder summarized points of discus sion for issues to be presented to all participants Timewasprovidedforthe 2 recorders and 2 facilitators for each topic to meet and prepare plenary pre sentations given to the entire group We proposed a checklist for meta analyses of observational studies based on the deliberation of the independent groups Finally wecirculatedthecheck listforcommenttoallconferenceattend ees and representatives of several con stituencieswhowouldusethechecklist RESULTS The checklist resulting from work group deliberations is organized around recommendations for report ing background search strategy methods results discussion andcon clusions TABLE Background Reporting of the background should include the definition of the problem under study statement of hypothesis description of the study outcome s considered type of exposure or inter ventionused typeofstudydesignused and complete description of the study population Whencombiningobserva tional studies heterogeneity of popu lations eg US vs international stud ies design eg case controlvscohort studies and outcome eg different studies yielding different relative risks that cannot be accounted for by sam pling variation is expected 8 Search Reportingofthesearchstrategyshould include qualifications of the search ers specification of databases used searchstrategyandindexterms useof any special features eg explosion search software used use of hand searchingandcontactwithauthors use ofmaterialsinlanguagesotherthanEn glish useofunpublishedmaterial and exclusion criteria used Published re searchshowsthatuseofelectronicda tabases may find only half of all rel evant studies and contacting authors may be useful 27although this result may not be true for all topic areas 28 For example a meta analysis of de pressioninelderlymedicalinpatients29 used 2 databases for the search In addition bibliographies of retrieved papersweresearched However theau thors did not report their search strat egy in enough detail to allow replica tion An example of a thorough reject log canbefoundinthereportofameta analysisofelectricalandmagneticfield exposure and leukemia 30Examples of a table characterizing studies included can be found in Franceschi et al31and Saag et al 32Complete specification of searchstrategyisnotuniform areview of103publishedmeta analysesinedu cation showed that search procedures weredescribedinadequatelyinthema jority of the articles 10 REPORTING META ANALYSES OF OBSERVATIONAL STUDIES 2000 American Medical Association All rights reserved JAMA April 19 2000 Vol 283 No 152009 at Second Military Medical Univ on March 31 2009 Downloaded from Methods Items in this checklist section are con cerned with the appropriateness of any quantitative summary of the data de greetowhichcodingofdatafromthear ticleswasspecifiedandobjective assess mentofconfounding studyquality and heterogeneity use of statistical meth ods anddisplayofresults Empiricalevi dence shows that reporting of proce dures for classification and coding and quality assessment is often incomplete fewer than half of the meta analyses re ported details of classifying and coding theprimarystudydata andonly22 as sessed quality of the primary studies 10 We recognize that the use of quality scoring in meta analyses of observa tionalstudiesiscontroversial asitisfor RCTs 16 33becausescoresconstructedin an ad hoc fashion may lack demon strated validity and results may not be associatedwithquality 34Nevertheless some particular aspects of study qual ity have been shown to be associated witheffect eg adequateconcealmentof allocationinrandomizedtrials 35Thus key components of design rather than aggregatescoresthemselves maybeim portant Forexample inastudyofblind ing masking of readers participating in meta analyses masking essentially made no difference in the summary odds ratios across the 5 meta analy ses 36We recommend the reporting of quality scoring if it has been done and also recommend subgroup or sensitiv ity analysis rather than using quality scores as weights in the analysis 37 38 While some control over heteroge neity of design may be accomplished through the use of exclusion rules we recommendusingbroadinclusioncri teria for studies and then performing analysesrelatingdesignfeaturestoout come 8In cases when heterogeneity of outcomes is particularly problematic asinglesummarymeasuremaywellbe inappropriate 39Analyses that stratify by study feature or regression analysis with design features as predictors can be useful in assessing whether study outcomes indeed vary systematically with these features 40 Investigatingheterogeneitywasakey feature of a meta analysis of observa tional studies of asbestos exposure and riskofgastrointestinalcancer 41Theau thors of the meta analysis hypoth esized that studies allowing for a latent period between the initiation of expo sure and any increases in risk should show onaverage appropriatelyhigher standardizedmortalityratiosthanstud iesthatignoredlatency Inotherwords the apparent effect of exposure would be attenuated by including the latent period in the calculation of time at risk the denominator since exposure relateddeaths the numerator would by definition not occur during that la tent period FIGURE In fact the data suggested that stud ies allowing for latent periods found on average somewhat higher standard ized mortality ratios than studies ignoring latency This example shows that sources of bias and heterogeneity can be hypothesized prior to analysis and subsequently confirmed by the analysis Results Recommendations for reporting of re sults include graphical summaries of study estimates and any combined es timate atablelistingdescriptiveinfor mation for each study results of sen sitivity testing and any subgroup analysis and an indication of statisti cal uncertainty of findings Table A Proposed Reporting Checklist for Authors Editors and Reviewers of Meta analyses of Observational Studies Reporting of background should include Problem definition Hypothesis statement Description of study outcome s Type of exposure or intervention used Type of study designs used Study population Reporting of search strategy should include Qualifications of searchers eg librarians and investigators Search strategy including time period included in the synthesis and keywords Effort to include all available studies including contact with authors Databases and registries searched Search software used name and version including special features used eg explosion Use of hand searching eg reference lists of obtained articles List of citations located and those excluded including justification Method of addressing articles published in languages other than English Method of handling abstracts and unpublished studies Description of any contact with authors Reporting of methods should include Description of relevance or appropriateness of studies assembled for assessing the hypothesis to be tested Rationale for the selection and coding of data eg sound clinical principles or convenience Documentation of how data were classified and coded eg multiple raters blinding and interrater reliability Assessment of confounding eg comparability of cases and controls in studies where appropriate Assessment of study quality including blinding of quality assessors stratification or regression on possible predictors of study results Assessment of heterogeneity Description of statistical methods eg complete description of fixed or random effects models justification of whether the chosen models account for predictors of study results dose response models or cumulative meta analysis in sufficient detail to be replicated Provision of appropriate tables and graphics Reporting of results should include Graphic summarizing individual study estimates and overall estimate Table giving descriptive information for each study included Results of sensitivity testing eg subgroup analysis Indication of statistical uncertainty of findings Reporting of discussion should include Quantitative assessment of bias eg publication bias Justification for exclusion eg exclusion of non English language citations Assessment of quality of included studies Reporting of conclusions should include Consideration of alternative explanations for observed results Generalization of the conclusions ie appropriate for the data presented and within the domain of the literature review Guidelines for future research Disclosure of funding source REPORTING META ANALYSES OF OBSERVATIONAL STUDIES 2010JAMA April 19 2000 Vol 283 No 15 2000 American Medical Association All rights reserved at Second Military Medical Univ on March 31 2009 Downloaded from Discussion Thediscussionshouldincludeissuesre latedtobias includingpublicationbias confounding and quality Bias can oc curintheoriginalstudies resultingfrom flawsinthestudydesignthattendtodis tortthemagnitudeordirectionofasso ciations in the data or from the way in which studies are selected for inclu sion 42Publication bias the selective publicationofstudiesbasedonthemag nitude usuallylarger anddirectionof their findings represents a particular threattothevalidityofmeta analysisof observational studies 43 45Thorough specificationsofqualityassessmentcan contributetounderstandingsomeofthe variations in the observational studies themselves Methodsshouldbeusedto aidinthedetectionofpublicationbias eg fail safeproceduresorfunnelplots 46 Schlesselman47commentsonsuchbi ases in assessing the possible associa tion between endometrial cancer and oral contraceptives This meta anal ysis combined both cohort and case control studies and used a sensitivity analysis to illustrate the influence of specific studies such as those pub lished in English Conclusion Due to these biases in observational studies the conclusion of the report should contain consideration of alter native explanations for observed re sultsandappropriategeneralizationsof the conclusion A carefully conducted meta analysiscanrevealareaswarrant ingfurtherresearch Finally sincefund ingsourcehasbeenshowntobeanim portant source of heterogeneity 48the sponsoringorganizationshouldbedis closedandanyeffectonanalysisshould be examined COMMENT Taking stock of what is kn