实验设计与方差分析 Chap6_handoutwith_summary

Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Jingyuan Liu SOE and WISE Xiamen University DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Connection between Regression and ANOVA You have learned the regression model before y 0 1X1 2X2 pXp The predictors x variables are usually continuous In this course we have covered diff erent ANOVA models The predictors factors have discrete levels In this chapter we connect the two using regression approach to analyze the ANOVA models combining the ANOVA and regression when you have both discrete factors and continuous predictors analysis of covariance ANCOVA Ref Chapter 14 3 in the book 5th edition and Chapter16 8 and Chapter 22 in Applied Linear Statistical Models Kutner et al DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Regression Approach to One way ANOVA Reconsider the one way fi xed eff ect model yij i ij where Pa i 1 i 0 In Chapter 3 we used ANOVA approach to estimate the paramters and to obtain the ANOVA table SourcedfSSMSE MS F Treatmenta 1SSTrtMSTrt 2 n Pa i 1 2i a 1 MSTrt MSE ErrorN aSSEMSE 2 TotalN 1SST But notice this is still a linear model with a 1 parameters 1 a although one of i s is redundant Why to be estimated So intuitively we should be able to use the regression approach to estimate the parameters But how DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Regression Approach to One way ANOVA Model Representation by Regression To apply the regression approach to the ANOVA model we need rewrite the ANOVA model yij i ij i 1 a j 1 n in the regression model matrix form Y X DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Regression Approach to One way ANOVA Model Representation by Regression E g Consider a one way ANOVA model with a 3 and n 2 The ANOVA model is yij i ij i 1 2 3 j 1 2 We have 3 parameters to estimate 1and 2 3 1 2 So the coeffi cient vector 1 2 0 Write down all the eff ects model equations Match each term of these equations with the regression Or element wise yij 1Iij1 2Iij2 ij DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Regression Approach to One way ANOVA Model Representation by Regression What about the general case Notice that except for the intercept term the other columns in the X matrix consist of only 0 1 and 1 Thus the corresponding x variables are called the indicator dummy variable This is the essential of the regression representation of the ANOVA model defi ne indicator dummy variables with a reference level without loss of generality treat the last level as reference level DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Regression Approach to One way ANOVA Model Representation by Regression In general for the one way ANOVA model yij i ij i 1 a j 1 n we can rewrite the model by regression yij 1Iij1 2Iij2 a 1Iij a 1 ij where Iijkdenotes the value of indicator variable Ik k 1 a 1 for the jth observation from the ith factor level Iijk 1 if i k 1 if i a 0 otherwise Or equivalently on the population level y 1I1 2I2 a 1Ia 1 Ik 1 if the observation is from factor level k 1 if the observation is from factor level a 0 otherwise DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Regression Approach to One way ANOVA Model Representation by Regression Remarks If the factor has a levels we need a 1 dummy variables Why There are other dummy coding methods but this parameterization gives the most straightforward interpretation of the coeffi cients and we can easily recover the ANOVA model from the regression How For an observation if Ik 1 or 1 then the value of the other I s are fi xed but if Ik 0 we cannot determine the value of other I s When constructing X matrix do not forget the intercept column 1 Why can t we use the regular regression with only one predictor the factor rather than using the multiple dummy variables Because the factor is often nominal variable which has no magnitude Usually the regression approach is only applied to the fi xed eff ect models where the estimates of i s are meaningful DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Regression Approach to One way ANOVA Estimation and Inference Since we already obtain the regression representation of the ANOVA model we can adopt regression approach to estimate the parameters and to draw any inference Recall that the least square estimate of in the multiple linear regression model is b 1 a 1 0 X0X 1X0Y By matching each term we can show that 1 a Pa i 1 yi y for balanced case and i yi Note In this course we only focus on the balanced case where the regression approach is always equivalent to the ANOVA approach However under unbalanced situation the equivalence only holds for the ANOVA models with the unweighted mean 1 a Pn i 1 i we didn t cover this in class DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Regression Approach to One way ANOVA Example Consider an unbalanced case here A food company wished to test four package designs for a new breakfast cereal 19 comparable stores were randomly selected and each store was randomly assigned one of the package design Sales in number of cases were observed after a period DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Regression Approach to One way ANOVA Example I ANOVA approach Of course you can conduct multiple comparison to further dig out which pairs of means are diff erent DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Regression Approach to One way ANOVA Example II Regression approach DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Regression Approach to One way ANOVA Example II Regression approach cont d The least squared estimation yields the fi tted regression equation y 18 675 4 075I1 5 275I2 0 825I3 What are the estimated eff ects and treatment means The regression ANOVA table looks essentially the same as the ANOVA table based on the ANOVA model but we use SSRto denote the sum of squares due to regression model refer to your regression course We can show that SSR SSTrt DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Analysis of Covariance ANCOVA As you may notice the regression approach seems to be more troublesome than the ANOVA approach for this type of data So why bother to discuss it That s because in addition to the factor eff ects we sometimes also want to include one or more quantitative predictors variables here usually called covariates to reduce the variance of error in the model i e to make the analysis more precise and powerful This requires a combination of ANOVA and regression called the analysis of covariance ANCOVA DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA ANCOVA Intuition Example How does ANCOVA reduce the variance of error and make the analysis more precise E g Consider a study for the eff ects of three diff erent fi ve minutes fi lms on promoting travel in Xiamen 15 subjects are randomly selected each was shown one of the three fi lms and afterwards he she was questioned about the desire score to travel in Xiamen One way ANOVA with three levels three diff erent fi lms DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA ANCOVA Intuition Example We can compare the three estimated treatment means from the separate treatment mean plots DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA ANCOVA Intuition Example Is there any potential issue with this analysis The error term varies much indicating a large error variance In other words there should be something important that we didn t take into consideration Intuitively does the diff erence in scores by the three diff erent fi lms really result from the fi lms Or maybe due to diff erent pre impressions to Xiamen by the 15 subjects How to deal with the issue We may conduct a survey questionnaire before showing the fi lms and utilize the initial attitude scores of the subjects DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA ANCOVA Intuition Example We plot the after fi lm desire scores against the initial attitude score for each of the 15 subjects DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA ANCOVA Intuition Example Comments Intuitively the after fi lm mean desire scores yi should be highly related to the initial attitude scores although sometimes need not be linearly Each of the three regression lines is fi tted by the fi ve data points of the corresponding treatment They still represent the characteristics of treatments but now depend on a quantitative variable the prestudy attitude The scatter spread around the treatment regression lines is much less than the previous plots because we extract useful information the prestudy attitude from the error Therefore the error variability is reduced and the inferences such as tests are more precise and powerful DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Single factor ANCOVA Model Statement The single factor ANCOVA is constructed as yij i Xij X ij Notations and constraints i 1 2 a j 1 2 ni y ij the jth observation in the ith treatment unweighted overall mean i the ith fi xed eff ect Pa i 1 i 0 regression coeffi cient for the relation between y and X Xij the value of the covariate associated with yij ij random error i i d N 0 2 Assumption of y s derived from above yijindependent N ij 2 where ij i Xij X P i 0 DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Single factor ANCOVA Model Statement Comments The reason to subtract X is to ensure that is still the overall mean Why We consider Xij and hence X as fi xed constants rather than random variables With ANOVA models all observations for the ith treatment have the same mean response E yij ifor all j However this is not so with the ANCOVA since E yij also relies on the covariate Xij E yij ij i Xij X This is actually a regression line DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Single factor ANCOVA Graph Understanding Go back to the example The following graph illustrates how the treatment regression lines appear DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Single factor ANCOVA Graph Understanding Cont d DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Single factor ANCOVA Graph Understanding Remark The assumption in ANCOVA is the same slope for all treatments When the treatments interact with the covariate X nonparallel slopes the ANCOVA is not appropriate Instead we study each treatment regression line separately DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Single factor ANCOVA Regression Formulation An easy way to estimate the parameters of ANCOVA and make inference is through the regression formulation yij 1Iij1 a 1Iij a 1 xij ij where xij Xij X and the a 1 indicators are defi ned as DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Single factor ANCOVA Regression Formulation The X matrix for this model and the coeffi cient vector are Thus we can use b 1 a 1 0 X0X 1X0Y to estimate the unknown parameters DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Single factor ANCOVA LOF Test for Treatment Eff ect The key inferences in ANCOVA are the same as ANOVA test of the treatment eff ects H0 1 2 a 0 v s H1 not all i s equal 0 Under regression setting we adopt the lack of fi t LOF test Fit both the full model and reduced model and compare the SSE s from the two ANOVA tables Full model yij 1Iij1 a 1Iij a 1 xij ij Reduced model yij xij ij DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Single factor ANCOVA LOF Test for Treatment Eff ect Full model yij 1Iij1 a 1Iij a 1 xij ij SourcedfSSMS RegressiondfR F aSSR F MSR F ErrordfE F N a 1SSE F MSE F TotalN 1SST Reduced model yij xij ij SourcedfSSMS RegressiondfR R 1SSR R MSR R ErrordfE R N 2SSE R MSE R TotalN 1SST DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Single factor ANCOVA LOF Test for Treatment Eff ect The LOF statistic is constructed as F MSE LOF MSE F SSE R SSE F dfE R dfE F SSE F N a 1 SSE R SSE F a 1 SSE F N a 1 where F F a 1 N a 1 under H0 Thus reject H0if F F a 1 N a 1 DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Single factor ANCOVA LOF Test for Treatment Eff ect Comments The main idea of LOF test is to evaluate the diff erence between the model fi t from the full model and that from the reduced model F small MSE LOF small SSE R SSE F small the two model fi ts are similar the reduced model is enough no signifi cant treatment eff ects cannot reject H0 The reduced model is just a simple linear regression For any regression model with p parameters the degree of freedom for the error term should be N p since p degrees of freedom are used up for estimating parameters In the full model we have in total a 1 parameters 1 a 1 thus dfE F N a 1 DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Single factor ANCOVA Example Example A company studied the eff ects of three diff erent types of promotions on sales of its crackers Treatment 1 Sampling of product by customers in store Treatment 2 Additional shelf space in regular location Treatment 3 Additional display shelves at ends of aisle 15 comparable stores were randomly assigned to the 3 promotion types 5 stores for each type Response y is the number of sales during the promotional period and the sale of the product in the preceding period denoted by X were also collected DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Single factor ANCOVA Example Graphical understanding Linear regression and parallel slopes appear to be reasonable DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Single factor ANCOVA Example Development of model yij 1Iij1 2Iij2 xij ij where xij Xij X and the two indicators are defi ned as Data structure DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Single factor ANCOVA Example Parameter estimates The least squared estimation b 1 2 0 X0X 1X0Y can be obtained by any statistical software Then we obtain the fi tted model yij 33 8 6 017Iij1 0 942Iij2 0 899xij xij Xij 25 Note We can get three treatment regression lines from above Trt 1 Iij1 1 Iij2 0 y1j 33 8 6 017 0 899 Xij 25 Trt 2 Iij1 0 Iij2 1 y2j 33 8 0 942 0 899 Xij 25 Trt 3 Iij1 Iij2 1 y3j 33 8 6 017 0 942 0 899 Xij 25 DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Single factor ANCOVA Example Fitted treatment regression lines DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Single factor ANCOVA Example LOF Test for treatment eff ects H0 1 2 3 0 vs H1 16 0 or 26 0 Full model yij 1Iij1 2Iij2 xij ij Reduced model yij xij ij F SSE R SSE F dfE R dfE F MSE F 455 722 38 571 13 11 3 506 59 5 F F0 05 13 11 11 3 98 thus H0 is rejected and the three cracker promotions diff er in sales eff ectiveness DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Single factor ANCOVA Additional Topics Test for H0 0 This is simply the individual t test of the regression coeffi cients in the multiple linear regression model Reject H0 if s t 2 N a 1 where s is the square root of the estimated variance of Test for parallel slopes Strictly speaking this should be done before conducting the ANCOVA It is actually testing the interaction between the treatment eff ects and the covariate Why We can still use the LOF test to compare two model fi ttings but here the full model is Why yij 1Iij1 a 1Iij a 1 xij 1Iij1xij a 1Iij a 1xij ij the reduced model is yij 1Iij1 a 1Iij a 1 xij ij DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Single factor ANCOVA Generalization Two factor ANCOVA To generalize the single factor ANCOVA to two factor case yijk i j ij Xijk X ijk we now need a 1 indicators for factor A and b 1 for factor B The interaction terms between A and B is constructed as the products of indicators E g if a b 2 we need one indicator for factor A and one for B yijk 1Iijk1 1Iijk2 11Iijk1Iijk2 Xijk X ijk The main analyses are identical with single factor ANCOVA DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA Summary of Chapter 6 Use regression approach to analyze ANOVA model yij i ij yij 1Iij1 a 1Iij a 1 ij Estimation of unknown parameters Regression ANOVA table ANCOVA with both factor and quantitative covariate yij i Xij X ij yij 1Iij1 a 1Iij a 1 Xij X ij Estimation of unknown parameters Fitted treatment regression lines Lack of fi t test for the treatment eff ects Test for the slope 1 parallel 2 zero DOE and ANOVA Chapter6Chapter 6 Regression Approach to ANOVA and Analysis of Covariance ANCOVA To Sum Everything Up Chapter 3 Chapter 6 Recall that in Chapter 1 we introduced the major steps for a complete scientifi c research project 1 Recognition and statement of the problem Non statistical 2 Selec