Skip to content

Central Limit Theorem

The central limit theorem explains why normal distributions appear so often in probability.
CENTRAL LIMIT THEOREM
Let X1,X2,…X_1,X_2,\ldots be independent identically distributed random variables with the math expectation EX1=μ\mathbb{E}X_1=\mu and the finite variation Var⁡(X1)=σ2 \operatorname{Var}(X_1)=\sigma^2. If Sn=X1+⋯+XnS_n=X_1+\cdots+X_n, then
Sn−nμσn→n→∞dN(0,1).\begin{equation*}\frac{S_n-n\mu}{\sigma\sqrt{n}} \xrightarrow[n\to\infty]{d} N(0,1).\end{equation*}
Equivalently,
P(a≤Sn−nμσn≤b)→n→∞12π∫abe−x2/2 dx.\begin{equation*}\mathbb{P}\left(a\leq \frac{S_n-n\mu}{\sigma\sqrt{n}}\leq b\right)\xrightarrow{n\to\infty}\frac{1}{\sqrt{2\pi}}\int_a^b e^{-x^2/2}\,\mathrm{d}x.\end{equation*}

Motivation

The sum of 10 dices

A standard example is the sum of dice rolls. Let Y1,…,YnY_1,\ldots,Y_n be independent rolls of a fair six-sided die. Then
EY1=1+2+3+4+5+66=72andVar⁡(Y1)=3512.\begin{equation*}\mathbb{E}Y_1=\frac{1+2+3+4+5+6}{6}=\frac{7}{2} \quad \text{and} \quad \operatorname{Var}(Y_1)=\frac{35}{12}.\end{equation*}
Let T10=Y1+⋯+Y10T_{10}=Y_1+\cdots+Y_{10} be the total of ten independent dice rolls.By independence,
ET10=10EY1=35andVar⁡(T10)=10Var⁡(Y1)=1756..\begin{equation*}\mathbb{E}T_{10}=10\mathbb{E}Y_1 = 35 \quad \text{and} \quad \operatorname{Var}(T_{10})=10\operatorname{Var}(Y_1)=\frac{175}{6}.\end{equation*}.
In the animation, each brick records one sum of ten dice. Individual rolls are discrete and bounded, but the histogram of many sums is already close to the normal curve with mean 3535 and variance 10⋅35/12=175/610\cdot 35/12=175/6.

Galton board

A Galton board gives another concrete example of the same phenomenon. Each ball makes a sequence of independent left-or-right choices, so the final bucket is determined by a binomial count. With many balls, the bucket heights begin to form the same bell-shaped profile predicted by the central limit theorem.
A Galton board turns repeated independent binary choices into a binomial histogram.
With more rows and many more balls, the same binomial mechanism produces a smoother approximation to the normal density.
A larger Galton board makes the normal approximation visually sharper.

Proof of the Central Limit Theorem

We prove the theorem using characteristic functions. Replacing XiX_i by Xi−μX_i-\mu, it is enough to prove the result in the centered case μ=0\mu=0. Let
φ(t)=EeitX1\begin{equation*}\varphi(t)=\mathbb{E}e^{itX_1}\end{equation*}
be the characteristic function of X1X_1. Since EX1=0\mathbb{E}X_1=0 and Var⁡(X1)=σ2\operatorname{Var}(X_1)=\sigma^2, the characteristic function has the expansion
φ(t)=1−σ2t22+o(t2),t→0.\begin{equation*}\varphi(t)=1-\frac{\sigma^2t^2}{2}+o(t^2),\qquad t\to 0.\end{equation*}
For Sn=X1+⋯+XnS_n=X_1+\cdots+X_n, independence gives
Eexp⁡(itSnσn)=[φ(tσn)]n=(1−t22n+o(1n))n.\begin{equation*}\mathbb{E}\exp\left(it\frac{S_n}{\sigma\sqrt{n}}\right)=\left[\varphi\left(\frac{t}{\sigma\sqrt{n}}\right)\right]^n=\left(1-\frac{t^2}{2n}+o\left(\frac{1}{n}\right)\right)^n.\end{equation*}
Thus the desired limiting characteristic function should be e−t2/2e^{-t^2/2}, which is the characteristic function of the standard normal distribution. The only point needing care is that the expression above is complex-valued, so we use the following elementary extension of the familiar limit (1+c/n)n→ec(1+c/n)^n\to e^c.
By the continuity theorem for characteristic functions, this convergence of characteristic functions implies
Snσn→dN(0,1)\begin{equation*}\frac{S_n}{\sigma\sqrt{n}}\xrightarrow{d}N(0,1)\end{equation*}
in the centered case.

References

  • [Durrett2019]Durrett, Rick. Probability: Theory and Examples. Fifth edition. Cambridge University Press, 2019. PDF.