北大暑期课程《回归分析》(Linear-Regression-Analysis)讲义3.doc

Class 3, Page 7Class 3: Multiple regression I. Linear Regression Model in MatricesFor a sample of fixed size is the dependent variable; are independent variables. We can write the model in the following way:(1),where and expand from the matrix form into the element formAssumption A0 (model specification assumption):We call R(Y) the regression function. That is, the regression function of y is a linear function of the x variables. Also, we assume nonsingularity of XX. That is, we have meaningful Xs.II. Least Squares Estimator in MatricesPre-multiply (1) by X(2)Assumption A1 (orthogonality assumption): we assume that is uncorrelated with each and every vector in X. That is,(3)Sample analog of expectation operator is.Thus, we have(4)That is, there are a total of p restriction conditions, necessary for solving p linear equations to identify p parameters. In matrix format, this is equivalent to:(5)Substitute (5) into (2), we have(6)The LS estimator is then:(7)which is the same as the least squares estimator.Note: A1 assumption is needed for avoiding biases.III. Properties of the LS EstimatorFor the mode ,that is, is unbiased. V(b) is a symmetric matrix, called variance and covariance matrix of b.(after assuming non-serial correlation and homoscedasticity) important result, using A2blackboard1. Assumption A2 (iid assumption): independent and identically distributed errors. Two implications:1. Independent disturbances, Obtaining neat v(b).2. Homoscedasticity, Obtaining neat v(b)., scalar matrix. IV. Fitted Values and Residualsis called H matrix, or hat matrix. H is an idempotent matrix:For residuals:(I-H) is also an idempotent matrix.V. Estimation of the Residual VarianceA. Sample Analog (8)e is unknown but can be estimated by e, where e is residual. Some of you may have noticed that I have intentionally distinguished from e. e is called disturbance, and e is called residual. Residual is defined by the difference between observed and predicted values. The sample analog of (8) isIn matrix:The sample analog is thenB. Degrees of FreedomAs a general rule, the correct degrees of freedom equals the number of total observations minus the number of parameters used in estimation. In multiple regression, there are p parameters to be estimated. Therefore, the remaining degrees of freedom for estimating disturbance variance is n-p. C. MSE as the EstimatorMSE is the unbiased estimator. It is unbiased because it corrects for the loss of degrees of freedom in estimating the parameters. D. Statistical InferencesNow that we have point estimates (b) and the variance-covariance matrix of b. But we cannot do formal statistical tests yet. The question, then, is how to make statistical inferences, such as testing hypotheses and constructing confidence intervals. Well, the only remaining thing we need is the ability to use some tests, say t, Z, or F tests. Statistical theory tells us that we can conduct such tests if e is not only iid, but iid in a normal distribution. That is, we assume Assumption A3 (normality assumption): is distributed as With this assumption, we can look up tables for small samples. However, A3 is not necessary for large samples. For large samples, central limit theory assures that we can still make the same statistical inferences based on t, z, or F tests if the sample is large enough. A Summary of Assumptions for the LS Estimator 1. A0: Specification assumption Including nonsingularity of. Meaningful Xs.With A0, we can compute 2. A1:orthoganality assumption, for k = 0, . p-1, x0 = 1. Meaning: is needed for the identification of. All other column vectors in X are orthogonal with respect to e. A1 is needed for avoiding biases. With A1, b is unbiased and consistent estimator of. Unbiasedness means that Consistency:. For large samples, consistency is the most important criterion for evaluating estimators. 3. A2. iid independent and identically distributed errors.Two implications:1. Independent disturbances, Obtaining neat v(b).2. Homoscedasticity, Obtaining neat v(b)., scalar matrix. With A2, b is an efficient estimator. Efficiency: an efficient estimator has the smallest sampling variance among all unbiased estimators. That is where denotes any unbiased estimator. Roughly, for efficient estimators, imprecision i.e., SD(b) decreases by the inverse of the square root of n. That is, if you wish to increase precision by 10 times, (i.e., reduce S.E. by a factor of ten), you would need to increase the sample size by 100 times. A1 + A2 make OLS a BLUE estimator, where BLUE means the best, linear, unbiased estimator. That is, no other unbiased linear estimator has a smaller sampling variance than b. This result is called Gauss-Markov theorem.4. A3. Normality, is distributed as Inferences: looking up tables for small samples. A1 + A2 + A3 make OLS a maximum likelihood (ML) estimator. Like all other ML estimators, OLS in this case is BUE (best unbiased estimator). That is, no other unbiased estimator can have a smaller sampling variance than OLS. Note that ML is always the most efficient estimator among all unbiased estimators. The cost of ML is really the requirement of we know the true parametric distribution of the residual. If you can afford the assumption, ML is always the best. Very often, we dont make the assumption because we dont know the parametric family of the disturbance. In general, the following tradeoff is true: More information = more efficiency. Less assumption = less efficiency. It is not correct to call certain models OLS models and other ML models. Theoretically, a same model can be est