Sampling, LLN & CLT

The sample mean is random because the sample is random. The LLN describes concentration around the target; the CLT describes the shape of the remaining error after magnification.

1. Population, sample, and sampling distribution

Let \(Y_{1},\ldots ,Y_{n}\) be independent and identically distributed (i.i.d.) with mean \(\mu\) and variance \(\sigma ^{2}\). “Identically distributed” means each observation has the same marginal distribution. “Independent” means one observation provides no information about the others. These are distinct restrictions.

The population distribution describes an individual \(Y\). The empirical distribution describes observed values in one sample. The sampling distribution describes a statistic across repeated samples of size \(n\). Before observing the sample\(, \bar{Y}\) is random; after observing it, its realized value is a number.

Sampling clarification. The textbook works with an i.i.d. simple-random-sampling model. Sampling without replacement from a finite population introduces dependence. If the sampling fraction is substantial, finite-population corrections matter. Equal inclusion probabilities alone do not prove independence. Nonresponse, selective recruitment, and common shocks can also break the working model.

2. Exact moments of the sample mean

\[\begin{aligned}\bar Y&=\frac1n\sum_{i=1}^nY_i,\\ \mathbb E[\bar Y]&=\frac1n\sum_{i=1}^n\mathbb E[Y_i]=\mu,\\ \operatorname{Var}(\bar Y)&=\frac1{n^2}\sum_{i=1}^n\operatorname{Var}(Y_i)=\frac{\sigma^2}{n},\\ \operatorname{SD}(\bar Y)&=\frac\sigma{\sqrt n}.\end{aligned}\]

The expectation calculation uses linearity. The variance calculation additionally uses zero cross-covariances, supplied by independence. These equalities are exact for every \(n\) under the assumptions; they do not require normality or a large sample.

If the observations are normal\(, \bar{Y} \sim \mathcal{N}(\mu ,\sigma ^{2}/n)\) exactly because a sum of independent normals is normal. For Bernoulli observations\(, K=\sum Y_{i}\) has a binomial distribution and \(\bar{Y} =K/n\):

\[P\!\left(\bar Y=\frac kn\right)=\binom nk p^k(1-p)^{n-k},\quad k=0,\ldots,n.\]

The binomial expression makes the small-sample distribution explicit. Its support is discrete, even though the large-sample approximation is continuous.

3. Convergence in probability and the LLN

\[\begin{gathered}T_n\xrightarrow{p}\theta\quad\Longleftrightarrow\quad\forall\varepsilon>0,\;P(|T_n-\theta|>\varepsilon)\to0.\\ \underbrace{Y_i\text{ i.i.d.},\;\mathbb E[Y_i]=\mu,\;\operatorname{Var}(Y_i)<\infty}_{\text{Sufficient LLN conditions}}\\ \implies\bar Y\xrightarrow{p}\mu.\end{gathered}\]

Proof using Chebyshev'\(s\) inequality. For a variable \(W\) with finite variance\(, \Pr(|W-\mathbb{E}[W]|\ge \varepsilon )\le \operatorname{Var}(W)/\varepsilon ^{2}\). Apply it to \(\bar{Y}\) to obtain \(\Pr(|\bar{Y} -\mu |\ge \varepsilon )\le \sigma ^{2}/(n\varepsilon ^{2})\to 0\). The bound need not be tight; its vanishing proves consistency.

Consistency concerns arbitrarily small fixed neighborhoods of the target as \(n\) grows. It is not a claim that every additional observation improves a particular realized estimate. More general LLNs permit weaker conditions; finite variance is the sufficient version used in SW Key Concept 2.6.

4. Convergence in distribution and the CLT

\[\begin{gathered}Z_n\xrightarrow{d}Z\quad\Longleftrightarrow\quad F_{Z_n}(z)\to F_Z(z)\\ \text{at every continuity point of }F_Z.\\ Y_i\text{ i.i.d.},\quad 0<\sigma^2<\infty\\ \implies\frac{\sqrt n(\bar Y-\mu)}{\sigma}\xrightarrow{d}\mathcal N(0,1).\end{gathered}\]

Equivalently\(, \sqrt{n}(\bar{Y} -\mu ) \xrightarrow{d} \mathcal{N}(0,\sigma ^{2})\). The practical approximation is \(\bar{Y} \approx \mathcal{N}(\mu ,\sigma ^{2}/n)\). The n-dependent approximation is not a claim that the unscaled mean converges to a fixed nondegenerate normal distribution.

	LLN	CLT
Object	\(\bar{Y}\)	\(\sqrt{n}(\bar{Y} -\mu )/\sigma\)
Question	Does it concentrate at \(\mu\)?	What is the shape of normalized error?
Use	Consistency, plug-in estimates	Approximate probabilities and inference

5. How large is large?

No universal \(n=30\) or \(n=100\) guarantee follows from the CLT. Accuracy depends on skewness, tails, event probabilities, and the desired tail approximation. Rare Bernoulli successes can require much larger \(n\). When a third moment exists, the standardized skewness of an i.i.d. sample mean is the individual skewness divided by \(\sqrt{n}\). This helps explain why a skewed population can require more averaging.

Dependence requires another CLT and often another variance. Infinite variance lies outside the classical result. The Cauchy example in the Aden notes is a useful boundary case: i.i.d. Cauchy averages remain Cauchy and do not acquire the concentration described above.

6. Sampling laboratory

The local illustration below draws Bernoulli samples repeatedly. The top histogram shows sample means; the bottom shows the same means standardized using the true \(\sigma =\sqrt{p(1-p)}\). Change \(n\) while holding \(p\) fixed to distinguish concentration from shape. The simulation illustrates a theorem; it does not prove it.

Sample size \(n\) Success probability \(p\)

Check your understanding

Question. How much must \(n\) increase to halve \(\operatorname{SD}(\bar{Y} )\)? Does this halve \(\operatorname{SD}(Y)\)?

Solution. Quadruple \(n\). The population spread \(\sigma\) is unchanged.

Sources: SW §§2.5–2.6, pp. 89–99; Schellenberg Section 2, pp. 5–8; Aden notes §§3.1–3.4, pp. 10–14. Chebyshev proof and sampling caveat are supporting elaborations.