Issues in Experimental Design

1、Issues in Experimental DesignReliability and ErrorMore things to think about in experimental design The relationship of reliability and power Treatment effect not the same for everyoneSome benefit more than others Sounds like no big deal (or even obvious), but all of these designs discussed assume e

2、qual effect of treatment for individualsReliability What is reliability? Often thought of as consistency, but this is more of a by-product of reliability Not to mention that you could have perfectly consistent scores lacking variability (i.e. constants) for which one could not obtain measures of rel

3、iability Reliability may refer to a measures ability to capture an individuals true score, to distinguish accurately one person from another on some measure It is the correlation of scores on some measure with their true scores regarding that constructClassical True Score Theory Each subjects score

4、is true score + error of measurement Obsvar = Truevar + Errorvar Reliability = Truevar/ Obsvar = 1 Errorvar/ Obsvar Reliability and powerReliability = Truevar/ Obsvar = 1 Errorvar/ Obsvar If observed variance goes up, power will decreaseHowever if observed variance goes up, we dont know automaticall

5、y what happens to reliabilityObsvar = Truevar + ErrorvarIf it is error variance that is causing the increase in observed variance, reliability will decrease1Reliability goes down, Power goes downIf it is true variance that is causing the increase in observed variance, reliability will increaseReliab

6、ility goes up, Power goes downThe point is that psychometric properties of the variables play an important, and not altogether obvious role in how we will interpret results, and not having a reliable measure is a recipe for disasterError in ANOVA Typical breakdown in a between groups designSStot = S

7、Sb/t + SSe Variation due to treatment and random variation (error) The F statistic is a ratio of these variances F = MSb/MSeError in ANOVA Classical True Score Theory Each subjects score = true score + error of measurement MSe can thus be further partitioned Variation due to true differences on scor

8、es between subjects and error of measurement (unreliability) MSe = MSer + MSes MSer regards measurement error MSes systematic differences between individuals MSes comes has two sources Individual differences Treatment differences Subject by treatment interactionError in ANOVA The reliability of the

9、measure will determine the extent to which the two sources of variability (MSer or MSes) contribute to the overall MSe If Reliability = 1.00, MSer = 0Error term is a reflection only of systematic individual differences If Reliability = 0.00, MSes = 0Error term is a reflection of measurement error on

10、ly MSer = (1-Rel)MSe MSes = (Rel)MSeError in ANOVA We can actually test to see if systematic variation is significantly larger than variation due to error of measurement(Rel)()(Rel)(1 Rel)()(1 Rel)(1,1)eseereMSMSFMSMSdfnnError in ANOVA With a reliable measure, the bulk of MSe will be attributable to

11、 systematic individual differences However with strong main effects/interactions, we might see sig F for this test even though the contribution to model is not very much Calculate an effect size (eta-squared) SSes/SStotal Lyons and Howard suggest (based on Cohens rules of thumb) that .33 would sugge

12、st that further investigation may not be necessary How much of the variability seen in our data is due to systematic variation outside of the main effects? Subjects responding differently to the treatmentGist Discerning the true nature of treatment effects, e.g. for clinical outcomes, is not easy, a

13、nd not accomplished just because one has done an experiment and seen a statistically significant effect Small though significant effects with not so reliable measures would not be reason to go with any particular treatment as most of the variance is due poor measures and subjects that do not respond

14、 similarly to that treatment One reason to perhaps suspect individual differences due to the treatment would be heterogeneity of variance For example, lots of variability in treatment group, not so much in control Even with larger effects and reliable measures, a noticeable amount of the unaccounted

15、 for variance may be due to subjects responding differently to the treatment Methods for dealing with the problem are outlined in Bryk and Raudenbush (hierarchical linear modeling), but one strategy may be to single out suspected covariates and control for them (ANCOVA or Blocking)Repeated Measure a

16、nd Hierarchical Linear Modeling Another issue with ANOVA design again concerns the subject by treatment interaction, this time with regard to repeated measurements RM design can be seen as a special case of HLM where the RM (e.g. time) is nested within subjects The outcome is predicted by the repeat

17、ed measure as before, but one can allow the intercept and slope(s) to vary over subjects, and that variance taken into account for the model In this manner the HLM approach is specifically examining the treatment by subject interaction, getting a sense of the correlation between starting point and s

18、ubsequent changeRepeated Measures and Hierarchical Linear ModelingBriefly, HLM is a regression approach in which intercepts and/or coefficients are allowed to vary depending on other variablesAs an example, the basic linear model for RM is the sameHowever, as an example, the intercept may be allowed

19、 to vary as a function of another variable (in this case Subject)Which gives a new regression equation (note how this compares to RM in the GLM notes)eTbby110uZb1100ueZTby11110Example with One-way From before, stress week before, the week of, or the week after their midterm exam Using lmer in R1, al

20、lowing a random intercept for a linear model where time predicts stress level but the intercept is allowed to vary by subject reveals the same ANOVA lmemod0=lmer(ScoreTime+ (1|Subject),rmdata) anova(lmemod0)SourcedfSSMSFpSubject9654.372.7time2204.8102.46.747.0065error18273.215.178Analysis of Variance Table Df Sum Sq Mean Sq F valueTime 2 204.8 102.4 6