北大暑期课程《回归分析》(Linear Regression Analysis)义2.docVIP

Class 2, Page 7Class 2: Basics of matrix (2) Linear regression. I. Basics of matrix (2) 1. Inverse of a MatrixThe inverse of a square matrix exists if the matrix is nonsingular. The inverse A-1 is defined as:A-1A = AA-1 = I Alternatively, the condition can be expressed in three other forms: (1) A has rank n, (2) the n rows are linearly independent, and (3) the n columns are linearly independent. Inverse is a difficult operation. Usually we can use computer softwares to find the inverse. Here we only want to know a simple example. For a 2x2 matrix:A-1 = where D is the determinant of A. D(A) = ad-bc. 2. Determinant of a MatrixThe determinant of a matrix is a scale. A nonsingular matrix has a non-zero determinant. 3. Operation Rules of MatricesA = B means for all i, jA + B = B + A(A + B) + C = A + (B + C) (AB)C = A(BC)C(A + B) = CA + CBc(A + B) = cA + cB, where c is a scalarIA = AI = AA + O = AAO = OA = O (A) = A(A + B) = A + B(AB) = B A (ABC) = C B A (AB)-1=B-1 A-1, provided A and B are each nonsingular(proof: AB B-1 A-1=I)(ABC)-1=C-1B-1A-1(A-1)-1=A(A)-1=(A-1)4. Variance-covariance matrixFor a vector of variables b with elements (b0, b1, bk), its variance-covariance matrix II. Linear Regression with a Single Regressor (Simple Regression) For simple linear regression, we learned, We assume that this model is true only in the population. What we can observe, however, is a sample. For a sample of fixed size i = 1,.n, we can write the model in the following way:(1) where Let us further assume that and Equation (1) becomes (2)expand from the matrix form into the element form.Pre-multiply (2) by X(3)We set (orthogonality condition), meaning (first element);(second element). Given the orthogonality condition, we can easily solve b as (5),Why do we assume the orthogonality condition? Because orthogonality gives the least squares solution best linear predictor. BlackboardPartial with respect to set to zero. In practice, we dont know whether X satisfies the orthogonality condition. We usually make the assumption:Note that the first assumption means orthogonality between 1 and . The second assumption means that x is not correlated with . Similarly, Det Let us solve for b Thus, b is indeed your old friend: b =III. Inference of Regression Coefficients (simple regression)A. Define expectation of a vector: take expectation of each of the elements.B. Define variance of a vector: is a symmetric matrix, called variance and covariance matrix of b.C. Property: if A is a matrix with only constant elements,D. The LS EstimatorFor the model E. Properties of the LS Estimatorthat is, b is unbiased. (after assuming , non-serial correlation and homoscedasticity) blackboard We then need normality assumption for statistical inferences. Recall the formula:V(b1) = III. Fitted Values and ResidualsInterpretation of projection 3-d graphis called H matrix, or hat matrix. H is an idempotent matrix:For residuals: (I-H) is also a idempotent matrix.IV. Estimation of the Residual VarianceA. Sample Analog (6) is unknown but can be estimated by e, where e is residual. Some of you may have noticed that I have intentionally distinguished from e. is called disturbance, and e is called residual. Residual is defined by the difference between observed and predicted values. The sample analog of (6) isIn matrix:The sample analog is thenee/nB. Degrees of FreedomLet us review briefly the concept of degrees of freedom. As a general rule, the correct degrees of freedom equals the number of total observations minus the number of parameters used in estimation. Since we obtain residuals after we use estimated coefficients, the residuals are subject to linear constrained (recall orthogonality constraints). For example:If n=2, p=2, we have the saturated model, e1=0, e2=0. If n=3, p=2, there is only 1 degree of freedom. e1= - e1In multiple regression, there are p parameters to be estimated. Therefore, the remaining degrees o