Inference measures how strongly a sample distinguishes a hypothesized population mean from alternatives. A hypothesis test, a p-value, and a confidence interval are related views of the same sampling uncertainty.
1. State the target and the hypotheses
For a population mean \(\mu , a\) two-sided test compares \(H_{0}\):\(\mu =\mu _{0}\) with \(H_{1}\):\(\mu \ne \mu _{0}\). Upper- and lower-sided alternatives are \(\mu >\mu _{0}\) and \(\mu <\mu _{0}\). Choose the direction before looking at the result. A test either rejects \(H_{0}\) or fails to reject it; failure to reject does not establish equality.
Under the true null, the large-sample reference distribution is \(\mathcal{N}(0,1)\), using the CLT and, when necessary, Slutsky. With normal observations and known \(\sigma\), the \(Z\) distribution is exact for any \(n\). The exact small-sample unknown\(-\sigma\) case is in Module 11.
2. What a p-value means
A p-value is a null-model probability of a statistic at least as adverse to \(H_{0}\) as the observed statistic, using the preselected direction of the test.
| Alternative | Approximate p-value | 5% rejection rule |
|---|---|---|
| \(\mu \ne \mu _{0}\) | \(2[1-\Phi (|t_{\text{obs}}|)]\) | \(|t_{\text{obs}}|>1.96\) |
| \(\mu >\mu _{0}\) | \(1-\Phi (t_{\text{obs}})\) | \(t_{\text{obs}}>1.645\) |
| \(\mu <\mu _{0}\) | \(\Phi (t_{\text{obs}})\) | \(t_{\text{obs}}<-1.645\) |
For a two-sided test, distance is measured in both tails. For an upper-sided test, a large negative statistic gives a large p-value. A p-value is not \(\Pr(H_{0}\) is true|data), the probability that chance “caused” the result, or the probability of the exact observed sample.
3. Significance, size, power, and errors
| Truth / decision | Reject \(H_{0}\) | Fail to reject \(H_{0}\) |
|---|---|---|
| \(H_{0}\) true | Type I error | Correct nonrejection |
| \(H_{0}\) false | Correct rejection | Type II error |
The significance level \(\alpha\) is the chosen Type I error target. The size is the actual null rejection probability (or its supremum over a composite null). Approximation errors can make actual size differ from nominal \(\alpha\). The power at an alternative is the probability of rejection there; it equals one minus its Type II error probability. Critical values define a rejection region.
Lowering \(\alpha\) while retaining the same test family raises the cutoff and reduces power. Larger \(n\) typically raises power against a fixed nonzero effect by reducing standard error. Statistical significance does not measure economic importance: a tiny effect can be precisely estimated, and an important effect can be imprecisely estimated.
This formula is exact for normal observations and approximate under a suitable CLT. It displays the roles of the effect, noise, sample size, and cutoff. With many true nulls, repeated testing can create false positives; for m independent tests at \(\alpha\), the probability of at least one false rejection is \(1-(1-\alpha )\)ᵐ.
4. Worked inference from start to finish
Use the textbook'\(s\) hypothetical wage sample: \(n=200, \bar{Y} =22.64, s=18.14\), and \(H_{0}\):\(\mu =20\) versus a two-sided alternative. Then \(\operatorname{SE}=18.14/\sqrt{200}\approx 1.2827, t\approx 2.0582\), and \(p\approx .0396\). Reject at 5%, but not at 1%. The estimated difference is 2.64 dollars per hour; report it with uncertainty, not just a rejection label.
5. Confidence intervals by test inversion
Collect all null values \(\mu _{0}\) that the two-sided test does not reject:
The 95% wage interval is approximately [20.126,25.154]. Since 20 lies outside it, the matching two-sided 5% test rejects. Test–interval duality requires the same estimator, standard error, reference distribution, and sidedness. Mixing conventions can break the apparent agreement.
Coverage: \(P_\mu\{\mu\in\operatorname{CI}(\text{data})\}\approx1-\alpha\) across repeated samples. The interval endpoints are random; \(\mu\) is fixed in this frequentist formulation. After observing the data, the particular interval either contains \(\mu\) or does not. A 95% interval for the mean is not an interval containing 95% of individual outcomes.
6. Precision and sample-size planning
The margin of error is \(z\cdot \operatorname{SE}\). At fixed confidence level and population spread, halving the margin requires four times as many independent observations. For a Bernoulli proportion\(, p(1-p)\le 1/4\) gives a conservative normal-approximation planning formula:
where e is the desired half-width, and \(n\) is rounded up. For 95% and \(e=.005\), using \(z=1.96\) gives \(n\ge 38,416\). This controls the planned approximate sampling margin under the design; it does not eliminate selection bias or guarantee realized coverage under every small-sample configuration.
A one-sided 95% lower confidence set is [\(\bar{Y} -1.645SE,\infty\)); an upper confidence set is (\(-\infty ,\bar{Y} +1.645SE\)]. These invert the corresponding one-sided tests.
Check your understanding
Question. A two-sided p-value for \(\mu _{0}\) is .07. Is \(\mu _{0}\) in the matching 90% and 95% intervals?
Solution. It is excluded from the 90% interval, since .07<.10; included in the 95% interval, since .07>.05.
Sources: SW §§3.2–3.3, pp. 117–128; Exercises 3.5–3.6 and 3.9. The explicit power and familywise-error formulas are supporting derivations.