北大暑期课程回归分析LinearRegressionAnalysis讲义2

1、Class 2: Basics of matrix (2) Linear regression. I. Basics of matrix (2) 1. Inverse of a MatrixThe inverse of a square matrix exists if the matrix is nonsingular. The inverse A-1 is defined as:A-1A = AA-1 = I Alternatively, the condition can be expressed in three other forms: (1) A has rank n, (2) t

2、he n rows are linearly independent, and (3) the n columns are linearly independent. Inverse is a difficult operation. Usually we can use computer softwares to find the inverse. Here we only want to know a simple example. For a 2x2 matrix:A-1 = where D is the determinant of A. D(A) = ad-bc. 2. Determ

3、inant of a MatrixThe determinant of a matrix is a scale. A nonsingular matrix has a non-zero determinant. 3. Operation Rules of MatricesA = B means for all i, jA + B = B + A(A + B) + C = A + (B + C) (AB)C = A(BC)C(A + B) = CA + CBc(A + B) = cA + cB, where c is a scalarIA = AI = AA + O = AAO = OA = O

4、 (A')' = A(A + B)' = A' + B'(AB)' = B' A' (ABC)' = C' B' A' (AB)-1=B-1 A-1, provided A and B are each nonsingular(proof: AB B-1 A-1=I)(ABC)-1=C-1B-1A-1(A-1)-1=A(A)-1=(A-1)4. Variance-covariance matrixFor a vector of variables b with elements (b0, b1, b

5、k), its variance-covariance matrix II. Linear Regression with a Single Regressor (Simple Regression) For simple linear regression, we learned, We assume that this model is true only in the population. What we can observe, however, is a sample. For a sample of fixed size i = 1,.n, we can write the mo

6、del in the following way:(1) where Let us further assume that and Equation (1) becomes (2)expand from the matrix form into the element form.Pre-multiply (2) by X'(3)We set (orthogonality condition), meaning (first element);(second element). Given the orthogonality condition, we can easily solve

7、b as (5),Why do we assume the orthogonality condition? Because orthogonality gives the least squares solution best linear predictor. BlackboardPartial with respect to set to zero. In practice, we don't know whether X satisfies the orthogonality condition. We usually make the assumption:Note that

8、 the first assumption means orthogonality between 1 and . The second assumption means that x is not correlated with . Similarly, Det Let us solve for b Thus, b is indeed your old friend: b =III. Inference of Regression Coefficients (simple regression)A. Define expectation of a vector: take expectati

9、on of each of the elements.B. Define variance of a vector: is a symmetric matrix, called variance and covariance matrix of b.C. Property: if A is a matrix with only constant elements,D. The LS EstimatorFor the model E. Properties of the LS Estimatorthat is, b is unbiased. (after assuming , non-seria

10、l correlation and homoscedasticity) blackboard We then need normality assumption for statistical inferences. Recall the formula:V(b1) = III. Fitted Values and ResidualsInterpretation of projection 3-d graphis called H matrix, or hat matrix. H is an idempotent matrix:For residuals: (I-H) is also a id

11、empotent matrix.IV. Estimation of the Residual VarianceA. Sample Analog (6) is unknown but can be estimated by e, where e is residual. Some of you may have noticed that I have intentionally distinguished from e. is called disturbance, and e is called residual. Residual is defined by the difference b

12、etween observed and predicted values. The sample analog of (6) isIn matrix:The sample analog is thene'e/nB. Degrees of FreedomLet us review briefly the concept of degrees of freedom. As a general rule, the correct degrees of freedom equals the number of total observations minus the number of par

13、ameters used in estimation. Since we obtain residuals after we use estimated coefficients, the residuals are subject to linear constrained (recall orthogonality constraints). For example:If n=2, p=2, we have the saturated model, e1=0, e2=0. If n=3, p=2, there is only 1 degree of freedom. e1= - e1In multiple regression, there are p parameters to be estimated. Therefore, the remaining degree