(Textbook Question 10 on page 55)
Let \(\hat{\beta_0}\) and \(\hat{\beta_1}\) be the OLS intercept and slope estimators, respectively, and let \(\overline{u}\) be the sample average of the errors (not the residuals!).
(i) Show that \(\hat{\beta_1}\) can be written as \(\hat{\beta_1} =\beta_1 +\sum_{i=1}^{n} w_i u_i\), where \(w_i=\frac{d_i}{SST_x}\) and \(d_i =x_i -\overline{x}\).
The population model regression is \(y_i =\beta_0 +\beta_1 x_i +u_i\) and the sample regression model is \(y_i=\hat{\beta_0} +\hat{\beta_1} x_i +\hat{u_i}\) .
To find the estimate of \(\hat{\beta_1}\) we solve below:
\(y_i =\beta_0 +\beta_1 x_i +u_i\) \(y_i=\hat{\beta_0} +\hat{\beta_1} x_i +\hat{u_i}\)
\(\hat{\beta_1}=\frac{Cov(y_i ,x_i)}{Var(x_i)}=\frac{\Sigma_{i=1} ^n (x_i -\overline{x}) y_i}{\Sigma_{i=1} ^n (x_i -\overline{x})^2}= \frac{\Sigma_{i=1} ^n (x_i -\overline{x})(\beta_0 +\beta_1 x_i +u_i)}{SST_x}\)
\(\hat{\beta_1}=\frac{\Sigma_{i=1} ^n (x_i -\overline{x}) \beta_0 +\beta_1 \Sigma_{i=1} ^n (x_i -\overline{x}) x_i +\Sigma_{i=1} ^n (x_i -\overline{x}) u_i}{SST_x}\)
Also, \(\Sigma_{i=1} ^n (x_i -\overline{x})=0\) \(\Sigma_{i=1} ^n (x_i -\overline{x})x_i=\Sigma_{i=1} ^n (x_i -\overline{x}) (x_i -\overline{x})=\Sigma_{i=1} ^n (x_i -\overline{x})^2 =SST_x\)
Which gives \(\hat{\beta_1}=\frac{\Sigma_{i=1} ^n (x_i -\overline{x}) \beta_0 +\beta_1 \Sigma_{i=1} ^n (x_i -\overline{x}) x_i +\Sigma_{i=1} ^n (x_i -\overline{x}) u_i}{SST_x}=\frac{0+\hat{\beta_1} SST_x +\Sigma_{i=1} ^n (x_i -\overline{x})u_i}{SST_x}\)
\(\hat{\beta_1}=\beta_1 +\frac{\Sigma_{i=1} ^n (x_i -\overline{x}) u_i}{SST_x}=\beta_1 +\Sigma_{i=1} ^n w_i u_i\)
Since \(w_i=\frac{d_i}{SST_x}\) and \(d_i=x_i -\overline{x}\)
(ii) Use part (i), along with \(\Sigma_{i=1} ^n w_i=0\), to show that \(\hat{\beta_1}\) and \(\overline{u}\) are uncorrelated. [Hint: You are being asked to show that \(E[(\hat{\beta_1} -\beta_1) \cdot \overline{x}] =0\).]
Since \(\Sigma_{i=1} ^n w_i=0\) and \(\hat{\beta_1}=\beta_1 +\Sigma_{i=1} ^n w_i u_i\), the correlation between \(\hat{\beta_1}\) and \(\overline{u}\) can be shown by: \(Corr(\hat{\beta_1}, \overline{u})=\frac{Cov(\hat{\beta_1} ,\overline{u})}{\sigma_{\beta^\cdot _1} \sigma_{\overline{u}}}=\frac{E[[(\hat{\beta_1} -\beta_1)(\overline{u} -\overline{\overline{u}})]}{\sigma_{\beta^\cdot _1} \sigma_{\overline{u}}}\)
And since \(\overline{\overline{u}}=\overline{u}\), this means that \(E[\hat{\beta_1} -\beta_1)(\overline{u}-\overline{\overline{u}})]=0\) and the \(Corr(\hat{\beta_1}, \overline{u})=0\)
(iii) Show that \(\hat{\beta_0}\) can be written as \(\hat{\beta_0}=\beta_0 +\overline{u} -(\hat{\beta_1}-\beta_1) \overline{x}\).
\(y_i =\beta_0 +\beta_1 x_i +u_i\) and \(y_i=\hat{\beta_0} +\hat{\beta_1} x_i +\hat{u_i}\)
Taking the average gives \(\overline{y}=\beta_0 +\beta_1 \overline{x} +\overline{u}=\hat{\beta_0} +\hat{\beta_1} \overline{x}\)
\(\hat{\beta_0}=\beta_0 +\beta_1 \overline{x} +\overline{u} -\hat{\beta_1} \overline{x}=\beta_0 +\overline{u}-(\hat{\beta_1} -\beta_1) \overline{x}\)
(iv) Use parts (ii) and (iii) to show that \(Var(\hat{\beta_0})=\frac{\sigma^2}{n} +\frac{(\sigma^2)(\overline{x})^2}{SST_x}\)
Since \(Corr(\hat{\beta_1}, \overline{u})=E(\hat{\beta_1} \overline{u})=0\) and \(\hat{\beta_0}=\beta_0 +\overline{u}-(\hat{\beta_1} -\beta_1) \overline{x}\), the \(Var(\hat{\beta_1})=\frac{\sigma^2}{SST_x}\)
This means that \(\hat{\beta_0}-\beta_0=\overline{u} -(\hat{\beta_1}-\beta_1) \overline{x}\)
\(E(\hat{\beta_0}-\beta_0)^2=E[\overline{u} -(\hat{\beta_1}-\beta_1) \overline{x}]^2\)
\(=E[\overline{u}^2 -2\overline{u}(\hat{\beta_1}-\beta_1) \overline{x}+((\hat{\beta_1} -\beta_1)\overline{x})^2]\)
\(=E(\overline{u}^2) -2E[\overline{u}(\hat{\beta_1})\overline{x} +2\overline{x}E(\overline{u})+\overline{x}^2E[(\hat{\beta_1} -\beta_1)^2]\)
\(=\frac{\sigma^2}{n} -0 +0 +\overline{x}^2 Var(\hat{\beta_1})=\frac{\sigma^2}{n} +\frac{\overline{x}^2 \sigma^2}{SST_x}\)
Thus \(Var(\hat{\beta_0})=E(\hat{\beta_0} -\beta_0)^2=\frac{\sigma^2}{n} +\frac{\overline{x}^2 \sigma^2}{SST_x}\)
(v) Do the algebra to simplify the expression in part (iv) to equation (2.58) [Hint: \(\frac{SST_x}{n}=n^{-1} \Sigma_{i=1} ^n x_i ^2 -(\overline{x})^2\).]
Because \(\frac{SST_x}{n}=n^{-1} \Sigma_{i=1} ^n x_i ^2 -(\overline{x})^2\), \(Var(\hat{\beta_0})=\frac{\sigma^2}{n}+\frac{\sigma^2 (\overline{x})^2}{SST_x}=\frac{\sigma^2}{n}+\frac{\sigma^2 (\overline{x})^2}{\Sigma_{i=1} ^n x_i ^2 -n(\overline{x})^2}\)
\(SST_x=\Sigma_{i=1}^n (x_i -\overline{x})^2=\Sigma_{i=1}^n x_i ^2 -2\overline{x} \Sigma_{i=1}^n x_i +n\overline{x} ^2\)
\(=\Sigma_{i=1}^n x_i ^2-2\frac{\Sigma_{i=1}^n x_i}{n} \Sigma_{i=1}^n x_i+n(\frac{\Sigma_{i=1}^n x_i}{n})^2=\Sigma_{i=1}^nx_i ^2 -2\frac{(\Sigma_{i=1}^n x_i)^2}{n} +\frac{(\Sigma_{i=1}^n x_i)^2}{n}\)
\(=\Sigma_{i=1}^n x_i ^2 -\frac{(\Sigma_{i=1}^n x_i)^2}{n}=\Sigma_{i=1}^n x_i ^2 -n(\overline{x})^2\)
The gives \(\Sigma_{i=1}^n (x_i -\overline{x})^2=\Sigma_{i=1}^n (x_i ^2) -\frac{(\Sigma_{i=1}^n x_i)^2}{n} =\Sigma_{i=1}^n x_i ^2 -n(\overline{x})^2\)
\(Var(\hat{\beta_0})=\frac{\sigma^2}{n}+\frac{\sigma ^2 (\overline{x})^2}{SST_x}=\frac{\sigma ^2}{n} +\frac{\sigma ^2 (\overline{x})^2}{\Sigma_{i=1}^n x_i ^2 -n(\overline{x})^2}\)
\(=\frac{\sigma ^2}{n}(1 +\frac{ (\overline{x})^2}{\Sigma_{i=1}^n x_i ^2 -n(\overline{x})^2})\)
\(=\frac{\sigma^2}{n} (\frac{\Sigma_{i=1} ^n x_i ^2 -n(\overline{x})^2+n(\overline{x})^2}{\Sigma_{i=1} ^n x_i ^2 -n(\overline{x})^2)})=\frac{\sigma^2 n^{-1} \Sigma_{i=1} ^n x_i ^2}{\Sigma_{i=1} ^n s_i ^2 -n(\overline{x})^2}\)
\(\Sigma_{i=1} ^n (x_i -\overline{x})^2 =\Sigma_{i=1} ^n x_i ^2 -\frac{(\Sigma_{i=1} ^n x_i)^2}{n}=\Sigma_{i=1} ^n x_i ^2 -n(\overline{x})^2\)
\(Var(\hat{\beta_0})=\frac{\sigma ^2 n^{-1} \Sigma_{i=1} ^n x_i ^2}{\Sigma_{i=1} ^n (x_i -\overline{x})^2}\)