Conditional Information & Prediction

Econometrics studies relationships between variables. The joint distribution is the starting point; conditioning turns information about one variable into a distribution for another.

1. Joint \(\to\) marginal \(\to\) conditional distributions

\[\begin{aligned}p_{X,Y}(x,y)&=P(X=x,Y=y),\\ p_Y(y)&=\sum_x p_{X,Y}(x,y),\\ p_{Y\mid X}(y\mid x)&=\frac{p_{X,Y}(x,y)}{p_X(x)},\quad p_X(x)>0.\end{aligned}\]

Marginalization adds over the variable no longer being tracked. Conditioning restricts attention to a subgroup and renormalizes its probabilities to sum to one. For variables with a joint density, replace sums by integrals and use \(f_{Y|X}(y|x)=f_{X,Y}(x,y)/f_{X}(x)\) where the denominator is positive. Do not divide by \(\Pr(X=x)=0\) in the continuous case.

SW Table 2.2: \(X=1\) means no rain; \(Y=1\) means short commute
	\(X=0\)	\(X=1\)	Marginal
\(Y=0\)	.15	.07	.22
\(Y=1\)	.15	.63	.78
Marginal	.30	.70	1

\(\Pr(Y=1\mid X=0)=.15/.30=.5\), while \(\Pr(Y=1\mid X=1)=.63/.70=.9\). Reversing the conditioning gives a different question: \(\Pr(X=0\mid Y=1)=.15/.78\approx .1923\).

Bayes' rule (supporting connection). \(\Pr(X=x|Y=y)=\Pr(Y=y|X=x)\Pr(X=x)/\Pr(Y=y)\). It follows by expressing the same joint probability in two orders. A large \(\Pr(Y\mid X)\) does not automatically imply a large \(\Pr(X\mid Y)\).

2. Conditional mean and conditional variance

\[\begin{aligned}m(x)&=\mathbb E[Y\mid X=x]\\&=\sum_y y\,p_{Y\mid X}(y\mid x),\\ \operatorname{Var}(Y\mid X=x)&=\mathbb E[(Y-m(x))^2\mid X=x].\end{aligned}\]

\(\mathbb{E}[Y\mid X=x]\) is a number for a fixed \(x. \mathbb{E}[Y\mid X]=m(X)\) is a random variable because the information \(X\) varies. Conditional variance measures dispersion around the relevant conditional mean, not the unconditional mean.

For the commuting table\(, \mathbb{E}[Y\mid X=0]=.5\) and \(\mathbb{E}[Y\mid X=1]=.9\); the corresponding conditional variances are .25 and .09. Both the conditional mean and the conditional spread change with weather.

3. Law of iterated expectations

\[\mathbb E[Y]=\mathbb E[\mathbb E[Y\mid X]]=\sum_x m(x)p_X(x).\]

Assumption. \(\mathbb{E}|Y|<\infty\). Discrete proof. Substitute the conditional mean and multiply the conditional probability by \(p_{X}(x)\). The result is \(\sum _{x}\sum _{y}y p_{X,Y}(x,y)=\sum _{y}y p_{Y}(y)=\mathbb{E}[Y]\).

In the commuting example\(, \mathbb{E}[Y]=.5(.3)+.9(.7)=.78\). With multiple conditioning variables\(, \mathbb{E}[Y]=\mathbb{E}[\mathbb{E}[Y\mid X,Z]]\). The nested form \(\mathbb{E}[\mathbb{E}[Y\mid X,Z]\mid X]=\mathbb{E}[Y\mid X]\) averages away the additional information \(Z\) while retaining \(X\).

If h(X) is known conditional on \(X, \mathbb{E}[h(X)Y|X]=h(X)\mathbb{E}[Y|X]\), provided the relevant products are integrable. Therefore \(\mathbb{E}[Y\mid X]=0\) implies \(\mathbb{E}[Y]=0\) and \(\mathbb{E}[h(X)Y]=0\). This implication is central to regression moment conditions.

4. Conditional expectation as the best predictor

This chain is developed in SW Exercise 2.27 and made explicit here. Assume \(\mathbb{E}[Y^{2}]<\infty\). Let \(m(X)=\mathbb{E}[Y\mid X]\) and \(u=Y-m(X)\). Then \(\mathbb{E}[u\mid X]=0\) and \(\mathbb{E}[u h(X)]=0\) for square-integrable h(X).

\[\begin{aligned}\mathbb E[(Y-g(X))^2]&=\mathbb E[u^2]+\mathbb E[(m(X)-g(X))^2]\\&\ge\mathbb E[u^2].\end{aligned}\]

Proof. Write \(Y-g(X)=u+m(X)-g(X)\), square, and take expectations. The cross term vanishes because \(\mathbb{E}[u\mid X]=0\). Thus \(m(X)\) minimizes mean squared prediction error over all square-integrable functions of \(X\); equality holds when \(g(X)=m(X)\) almost surely.

Setting the alternative predictor to the constant \(\mathbb{E}[Y]\) gives the law of total variance:

\[\operatorname{Var}(Y)=\mathbb E[\operatorname{Var}(Y\mid X)]+\operatorname{Var}(\mathbb E[Y\mid X]).\]

For the commute table: \(.78(.22)=.3(.25)+.7(.09)+.3(.5-.78)^{2}+.7(.9-.78)^{2}=.138+.0336=.1716\).

Check your understanding

Question. If \(\mathbb{E}[u\mid X]=0\), must \(\operatorname{Var}(u|X)\) be constant?

Solution. No. Take \(u=X\varepsilon\) where \(\varepsilon\) is independent of \(X\) with mean zero and variance one. Then \(\mathbb{E}[u\mid X]=0\) but \(\operatorname{Var}(u|X)=X^{2}\).

Sources: SW §2.3, pp. 72–78; Exercises 2.19–2.20 and 2.27, pp. 106–109. Total variance and Bayes' rule are supporting extensions. Continue to dependence and covariance algebra.