Comparisons & Causal Effects

The variance-of-a-difference formula leads directly to two-sample inference. Interpreting that difference as a causal effect requires a research design in addition to a small p-value.

1. Comparing two independent population means

Let the target be \(\delta =\mu _{A}-\mu _{B}\). Within each group assume i.i.d. sampling, and assume the two samples are independent. Then \(\hat{\delta } =\bar{Y} _{A}-\bar{Y} _{B}\) is unbiased, and:

\[\begin{aligned}\operatorname{Var}(\hat\delta)&=\frac{\sigma_A^2}{n_A}+\frac{\sigma_B^2}{n_B},\\ \operatorname{SE}(\hat\delta)&=\sqrt{\frac{s_A^2}{n_A}+\frac{s_B^2}{n_B}},\\ t&=\frac{\hat\delta-\delta_0}{\operatorname{SE}(\hat\delta)},\\ \operatorname{CI}&=\hat\delta\pm z_{1-\alpha/2}\operatorname{SE}(\hat\delta).\end{aligned}\]

The variances add because \(\operatorname{Var}(U-V)=\operatorname{Var}(U)+\operatorname{Var}(V)-2Cov(U,V)\) and the samples are independent. Different means do not require equal variances. With both group sizes sufficiently large and appropriate finite moments, the centered, studentized difference is approximately \(\mathcal{N}(0,1)\).

For two proportions, replace the group variance by \(\hat{p} _{g}(1-\hat{p} _{g})\) in the large-sample plug-in convention. A difference of proportions is measured in percentage points, not automatically percent.

2. Historical earnings example and changing populations

SW Table 3.1 reports, for 2012 college-educated full-time workers aged 25–34: men \(n=2004\), mean 25.30, SD 12.09; women \(n=1951\), mean 21.50, SD 9.99. The estimated gap is 3.80 dollars per hour\(. \operatorname{SE}\approx .35227\) and the 95% interval is approximately [3.110,4.490].

The 1992 gap in 2012 dollars is \(24.83-21.39=3.44\). Under the exercise'\(s\) assumption of independent samples across years, the change in the gap is .36 and its variance is the sum of the four group-mean variances. Using the reported sample sizes and SDs gives \(\operatorname{SE}\approx .49940\) and a 95% interval of approximately \([-.619,1.339]\). Evidence of a positive gap in each year does not establish evidence of an increase between years.

Table 2.4 covers a broader age population, so its means need not equal Table 3.1'\(s\) means. Nominal earnings across years must be converted to a common price level for purchasing-power comparisons: real earnings in base-year dollars = nominal earnings \(\times\) CPI\(_{\text{base}}/\)CPI\(_{\text{year}}\). The original table'\(s\) historical CPI adjustment is part of its measurement definition.

3. Paired observations are a different design

When the same person is measured before and after a course, the two measurements are dependent. Define \(D_{i}=Y_{i,after}-Y_{i,before}\) and analyze the independent pairs through their differences:

\[\begin{aligned}\hat\delta&=\bar D,\qquad\operatorname{SE}(\bar D)=\frac{s_D}{\sqrt n},\\ \operatorname{Var}(D_i)&=\operatorname{Var}(Y_{\text{after}})+\operatorname{Var}(Y_{\text{before}})\\&\quad-2\operatorname{Cov}(Y_{\text{after}},Y_{\text{before}}).\end{aligned}\]

A before–after gain can reflect a treatment, practice, aging, or other time changes. Pairing addresses a variance calculation; it does not by itself identify a causal treatment effect.

4. Randomized experiments and conditional means

SW §3.5 defines a treatment effect using an ideal randomized experiment. With binary treatment D, the difference \(\mathbb{E}[Y|D=1]-\mathbb{E}[Y|D=0]\) has a causal interpretation when treatment assignment makes the groups comparable in their potential outcomes.

Supporting notation • potential outcomes

Let Y(1) and Y(0) be the outcomes under treatment and control, and \(Y=D Y(1)+(1-D)Y(0)\). Assume well-defined treatments, no interference between units, positive probability of either assignment, and random assignment D independent of (Y(1),Y(0)). Then:

\[\begin{aligned}\mathbb E[Y\mid D=1]-\mathbb E[Y\mid D=0]&=\mathbb E[Y(1)]-\mathbb E[Y(0)]\\&=\mathbb E[Y(1)-Y(0)].\end{aligned}\]

Derivation. Among treated units the observed outcome is Y(1); independence removes the conditioning on D. Apply the same reasoning to control units and subtract. The sample difference estimates the average treatment effect for the experimental population. The standard-error formula must match the sampling and assignment design; the chapter uses its independent-group framework.

Random sampling helps generalize to a target population. Random assignment helps identify a treatment effect within the recruited population. An experiment can have good random assignment but limited external validity. Noncompliance, selective attrition, or spillovers can require additional analysis.

5. Why observational differences are not automatically causal

\[\begin{aligned}&\mathbb E[Y\mid D=1]-\mathbb E[Y\mid D=0]\\&\quad=\underbrace{\mathbb E[Y(1)-Y(0)\mid D=1]}_{\text{Average treatment effect on the treated}}\\&\qquad+\underbrace{\mathbb E[Y(0)\mid D=1]-\mathbb E[Y(0)\mid D=0]}_{\text{Selection bias}}.\end{aligned}\]

The first term is the average treatment effect on the treated; the bracketed term is selection bias. Observed education or gender earnings differences do not, by themselves, isolate discrimination, education effects, skills, occupation selection, or experience. A small sampling p-value does not remove the bracketed term.

Natural or quasi-experiments seek circumstances that plausibly assign exposure as if randomly, under explicit assumptions. They do not become valid merely because researchers call them experiments. The voter-turnout box in SW p. 136 illustrates randomized outreach; its identification lesson is the controlled comparison, not a comparison with an earlier national turnout rate.

Check your understanding

Question. Students volunteer for a prep course and score higher afterward. Which two issues must be separated?

Solution. Statistical uncertainty in the observed gain and identification of the course'\(s\) effect. Volunteer selection and practice effects are not removed by a precise paired mean.

Sources: SW §§3.4–3.5, pp. 128–133; voter-turnout box p. 136; Exercises 3.15–3.17; Appendix 3.1. Potential-outcome notation and the selection decomposition are explicit extensions.