Correlation → Regression

The final step in Chapter 3 estimates relationships between variables. The same moment algebra, LLN, and CLT then reappear in regression.

1. Scatterplots before summary coefficients

A scatterplot places each observed pair \((X_{i},Y_{i})\) at a point. Inspect direction, curvature, spread, clusters, and unusual observations. A clear U-shaped relationship can have zero correlation. A high correlation indicates closeness to a straight line, not a steep slope.

The original age–earnings example uses 200 information-industry managers. Its positive but imperfect association is descriptive; age alone does not determine earnings. The example does not establish an effect of aging while holding all other factors fixed.

2. Sample covariance and correlation

\[\begin{aligned}s_{XY}&=\frac1{n-1}\sum_i(X_i-\bar X)(Y_i-\bar Y),\\ r_{XY}&=\frac{s_{XY}}{s_Xs_Y},\qquad |r_{XY}|\le1.\end{aligned}\]

Nonzero sample standard deviations are required. Adding constants leaves covariance unchanged; rescaling \(X\) and \(Y\) by b and c multiplies covariance by bc. Positive unit changes leave correlation unchanged; one negative rescaling reverses its sign.

SW reports \(s_{\text{Age}}=9.07, s_{\text{Earnings}}=14.37\), and \(s_{Age,Earnings}=33.16\). The correlation is \(33.16/(9.07\cdot 14.37)\approx .2544\). In cents, both the earnings SD and covariance multiply by 100 and the correlation stays the same. Correlation .25 does not mean that 25% of outcomes are predicted correctly.

3. Consistency is a sample-moment argument

\[\begin{aligned}s_{XY}&=\frac n{n-1}\left(\frac1n\sum_iX_iY_i-\bar X\bar Y\right)\\&\xrightarrow{p}\mathbb E[XY]-\mathbb E[X]\mathbb E[Y].\end{aligned}\]

For the textbook proof route, assume i.i.d. pairs and finite fourth moments of \(X\) and \(Y\). Cauchy–Schwarz gives \(\mathbb{E}[X^{2}Y^{2}]\le \sqrt{\mathbb{E}[X^{4}]\mathbb{E}[Y^{4}]}<\infty\), permitting the finite-variance LLN for XY. Combine it with LLNs for \(X\) and \(Y\). Consistent sample variances and continuous mapping then imply \(r_{XY}\xrightarrow{p} \rho _{XY}\), provided population variances are positive.

4. From a constant predictor to a line

Forward connection • regression

Module 7 chose a constant to minimize squared errors. Now choose a and b to minimize \(\mathbb{E}[(Y-a-\)bX\()^{2}\)]. With finite second moments and \(\operatorname{Var}(X)>0\), the first-order conditions give:

\[\begin{aligned}\mathbb E[Y-a-bX]&=0,\\ \mathbb E[X(Y-a-bX)]&=0,\\ b&=\frac{\operatorname{Cov}(X,Y)}{\operatorname{Var}(X)},\\ a&=\mathbb E[Y]-b\mathbb E[X].\end{aligned}\]

Replacing population moments by sample moments yields the simple OLS slope \(\hat{b} =s_{XY}/s_{X}^{2}\) and intercept â\(=\bar{Y} -\hat{b} \bar{X} . A\) positive sample variance is needed. Thus regression estimation is a direct continuation of Chapter 3'\(s\) sample covariance. For a binary \(X\), the fitted group values equal their sample means, and \(\hat{b} =\bar{Y} _{1}-\bar{Y} _{0}\).

The linear projection residual is orthogonal to 1 and \(X\) by construction. It need not have \(\mathbb{E}[u\mid X]=0\) unless the conditional mean is correctly represented by the line. Projection orthogonality and causal exogeneity are different statements.

5. Covariance ratios, omitted factors, and IV

Schellenberg Section 2 considers \(Y=a+\)bX\(+U\). Covariance algebra gives:

\[\frac{\operatorname{Cov}(X,Y)}{\operatorname{Var}(X)}=b+\frac{\operatorname{Cov}(X,U)}{\operatorname{Var}(X)}.\]

If \(\operatorname{Cov}(X,U)\ne 0\), the observed covariance ratio differs from the structural coefficient b. If an instrument \(Z\) satisfies \(\operatorname{Cov}(Z,U)=0\) and \(\operatorname{Cov}(Z,X)\ne 0\), then \(\operatorname{Cov}(Z,Y)/\operatorname{Cov}(Z,X)=b\). These are identification restrictions, not automatic consequences of observing a third variable.

6. The same probability engine behind OLS

For i.i.d. data satisfying \(Y=\beta _{0}+\beta _{1}X+u\) and \(\mathbb{E}[u\mid X]=0\), let \(Q=\operatorname{Var}(X)>0. A\) sufficient moment condition for the following argument is \(\mathbb{E}[X^{4}]<\infty\) and \(\mathbb{E}[u^{4}]<\infty\), which ensures a finite second moment of the score \((X-\mathbb{E}[X])u\):

\[\begin{aligned}\sqrt n(\hat b-\beta_1)&=\left\{\frac1n\sum_i(X_i-\bar X)^2\right\}^{-1}\\&\quad\cdot\left\{\frac1{\sqrt n}\sum_i(X_i-\bar X)u_i\right\},\\ \text{LLN: denominator}&\xrightarrow{p}Q,\\ \text{CLT: leading numerator}&\xrightarrow{d}\mathcal N(0,\Omega),\\ \Omega&=\mathbb E[(X-\mathbb E[X])^2u^2],\\ \text{Slutsky: }\sqrt n(\hat b-\beta_1)&\xrightarrow{d}\mathcal N\!\left(0,\frac\Omega{Q^2}\right).\end{aligned}\]

Replacing \(\bar{X}\) by \(\mathbb{E}[X]\) in the numerator changes it by \(\sqrt{n}(\bar{X} -\mathbb{E}[X])\)ū, which vanishes in probability under these conditions. The asymptotic variance allows the conditional error variance to vary with \(X\). Estimating it is the next step toward heteroskedasticity-robust standard errors.

The matrix version in Aden notes p. 23 uses \(Q=\mathbb{E}[\)xx′] and \(\Omega =\mathbb{E}[u^{2}\)xx′], giving \(Q^{-1}\Omega Q^{-1}\). The architecture is unchanged: consistent denominator, approximately normal average in the numerator, and studentization. This extension is a preview rather than an additional prerequisite for Chapters 2–3.

Check your understanding

Question. Does a slope \(\hat{b} =0\) establish that \(Y\) is unrelated to \(X\)?

Solution. No. It establishes zero estimated linear slope when the sample variance is positive. A nonlinear conditional mean can still vary with \(X\).

Sources: SW §3.7, pp. 137–141, Exercise 3.20; Schellenberg Section 2, pp. 2–5; Aden notes pp. 12–13 and 23. Regression derivations are forward extensions.