北大暑期课程《回归分析》(Linear-Regression-Analysis)讲义2

Class 2 Basics of matrix 2 Linear regression I Basics of matrix 2 1 Inverse of a Matrix The inverse of a square matrix exists if the matrix is nonsingular The inverse nn A A 1 is defined as A 1A AA 1 I Alternatively the condition can be expressed in three other forms 1 A has rank n 2 the n rows are linearly independent and 3 the n columns are linearly independent Inverse is a difficult operation Usually we can use computer softwares to find the inverse Here we only want to know a simple example For a 2x2 matrix A 1 D a D c D b D d dc ba 1 where D is the determinant of A D A ad bc 2 Determinant of a Matrix The determinant of a matrix is a scale A nonsingular matrix has a non zero determinant 3 Operation Rules of Matrices A B means for all i j ijij ba A B B A A B C A B C AB C A BC C A B CA CB c A B cA cB where c is a scalar IA AI A A O A AO OA O A A A B A B AB B A ABC C B A AB 1 B 1 A 1 provided A and B are each nonsingular proof AB B 1 A 1 I ABC 1 C 1B 1A 1 A 1 1 A A 1 A 1 4 Variance covariance matrix For a vector of variables b with elements b0 b1 bk its variance covariance matrix Class 2 Page 2 b V b b Cov b V b b Cov b V b V kk0 110 0 II Linear Regression with a Single Regressor Simple Regression For simple linear regression we learned iii xy 10 We assume that this model is true only in the population What we can observe however is a sample For a sample of fixed size i 1 n we can write the model in the following way 1 xy 10 where n y y y 1 n x x x 1 n 1 Let us further assume that x x x n n 1 1 1 1 2 and 1 0 Equation 1 becomes 2 122n Xy expand from the matrix form into the element form Class 2 Page 3 1 y 2 y n y Pre multiply 2 by X 3 XXXyX We set orthogonality condition meaning oeX first element 0 i e second element 0 iie x Given the orthogonality condition we can easily solve b as 5 yXXXb 1 Why do we assume the orthogonality condition Because orthogonality gives the least squares solution best linear predictor Blackboard Partial with respect to set to zero 2 10 ii xbby 10 bb 0 0 iii xee In practice we don t know whether X satisfies the orthogonality condition We usually make the assumption 0 x x 0 Cov Note that the first assumption means orthogonality between 1 and The second assumption means that x is not correlated with 2 1 1 1 1 1 1 1 1 1 1 1 1 ii i n n nn xx xn x x xx x x x x Similarly Class 2 Page 4 ii i n n yx y y y xx y 1 1 1 1 Det 222 2 2 2 XxnXnxnxxn iiii 22 22 2 1 XxnnXxnx XXnxXxnx iii iiii Let us solve for b yb 1 22 1iiiiii xnyxnXxnyxb 22 XxnYXnyxn iii 22 XxYyXxXxnYyXxn iiiiii 22 2 0 XxnyxxXxnyxb iiiiiii 2 2 Xxnyxxyx iiiiii 2 22 2 XxnyxxyXnyXnyx iiiiiiii 22 XxnyxnyXnXXxy iiiiii 22 XxnYXnyxnXny iiii 2 XxYXnyxXY iii 2 XxYyXxXY iii 1 bXY Thus b is indeed your old friend Class 2 Page 5 b 2 1 XxYyXx XbY iii III Inference of Regression Coefficients simple regression A Define expectation of a vector take expectation of each of the elements bE B Define variance of a vector is a symmetric matrix called variance and covariance matrix of b bV 110 100 bVbbCov bbCovbV bV C Property if A is a matrix with only constant elements bEAbAE AbVAbAV D The LS Estimator For the model Xy yXXXb 1 E Properties of the LS Estimator 1 yXXXEbE 1 XXXXE 11 XXXEXXXXE 11 XEXXEXXXX that is b is unbiased yX X V X bV 1 1 XXXXV 11 XXXXXXXV Class 2 Page 6 1 XXXV 11 XXXVXXX after assuming non serial correlation and homoscedasticity IV 2 21 XX blackboard 2 2 0 0 V We then need normality assumption for statistical inferences Recall the formula V b1 22 Xxi III Fitted Values and Residuals yHyXXXXbXy 1 Interpretation of projection 3 d graph 1X XXXH nn is called H matrix or hat matrix H is an idempotent matrix HHH For residuals yHIHyyyye I H is also a idempotent matrix IV Estimation of the Residual Variance A Sample Analog 6 222 iiii EEEV is unknown but can be estimated by e where e is residual Some of you may have noticed that I have intentionally distinguished from e is called disturbance and e is called residual Residual is defined by the difference between observed and predicted values The sample analog of 6 is 2 112110 2 2 2 1 1 1 pipiii ii i xbxbxbby n yy n e n Class 2 Page 7 In matrix The sample analog is then eeei 2 e e n B Degrees of Freedom Let us review briefly the concept of degrees of freedom As a general rule the correct degrees of freedom equals the number of total observations minus the number of parameters used in estimation Since we obtain residuals after we use estimated coefficients the residuals are subject to linear constrained recall orthogonality constraints For example If n 2 p 2 we have the saturated model e1 0 e2 0 If n 3 p 2 there is only 1 degree of freedom e1 e1 In multiple regression there are p parameters to be estimated Therefore the re