Introduction to Linear Regression Analysis. Douglas C. Montgomery

Example 2.1. The upper portion of the table contains the fitted regression model. Notice that before rounding the regression coefficients agree with those we calculated manually. Table 2.3 also contains other information about the regression model. We return to this output and explain these quantities in subsequent sections.

2.2.2 Properties of the Least-Squares Estimators and the Fitted Regression Model

      The least-squares estimators and have several important properties. First, note from Eqs. (2.6) and (2.7) that and are linear combinations of the observations yi. For example,

      where for i = 1, 2, …, n.

      The least-squares estimators and are unbiased estimators of the model parameters β₀ and β₁. To show this for , consider

since E(εi) = 0 by assumption. Now we can show directly that and , so

      That is, if we assume that the model is correct [E(yi) = β₀ + β₁xi], then is an unbiased estimator of β₁. Similarly we may show that of is an unbiased estimator of β₀, or

      The variance of is found as

      (2.13)

      because the observations yi are uncorrelated, and so the variance of the sum is just the sum of the variances. The variance of each term in the sum is , and we have assumed that Var(yi) = σ²; consequently,

      (2.14)

      The variance of is

      Now the variance of is just , and the covariance between and can be shown to be zero (see Problem 2.25). Thus,

      (2.15)

      Another important result concerning the quality of the least-squares estimators and is the Gauss-Markov theorem, which states that for the regression model (2.1) with the assumptions E(ε) = 0, Var(ε) = σ², and uncorrelated errors, the least-squares estimators are unbiased and have minimum variance when compared with all other unbiased estimators that are linear combinations of the yi. We often say that the least-squares estimators are best linear unbiased estimators, where “best” implies minimum variance. Appendix C.4 proves the Gauss-Markov theorem for the more general multiple linear regression situation, of which simple linear regression is a special case.

There are several other useful properties of the least-squares fit:

      1 The sum of the residuals in any regression model that contains an intercept β0 is always zero, that is,This property follows directly from the first normal equation in Eqs. (2.5) and is demonstrated in Table 2.2 for the residuals from Example 2.1. Rounding errors may affect the sum.

      2 The sum of the observed values yi equals the sum of the fitted values , orTable 2.2 demonstrates this result for Example 2.1.

      3 The least-squares regression line always passes through the centroid [the point ] of the data.

      4 The sum of the residuals weighted by the corresponding value of the regressor variable always equals zero, that is,

      5 The sum of the residuals weighted by the corresponding fitted value always equals zero, that is,

      2.2.3 Estimation of σ²

      In addition to estimating β₀ and β₁, an estimate of σ² is required to test hypotheses and construct interval estimates pertinent to the regression model. Ideally we would like this estimate not to depend on the adequacy of the fitted model. This is only possible when there are several observations on y for at least one value of x (see Section 4.5) or when prior information concerning σ² is available. When this approach cannot be used, the estimate of σ² is obtained from the residual or error sum of squares,

(2.16)

A convenient computing formula for SS_Res may be found by substituting into Eq. (2.16) and simplifying, yielding