{"id":20499,"date":"2024-09-05T17:32:52","date_gmt":"2024-09-05T15:32:52","guid":{"rendered":"https:\/\/djalil.chafai.net\/blog\/?p=20499"},"modified":"2024-09-20T17:30:42","modified_gmt":"2024-09-20T15:30:42","slug":"back-to-basics-lindeberg-principle","status":"publish","type":"post","link":"https:\/\/djalil.chafai.net\/blog\/2024\/09\/05\/back-to-basics-lindeberg-principle\/","title":{"rendered":"Back to basics : Lindeberg principle"},"content":{"rendered":"
\"Photo<\/a>
Jarl Waldemar Lindeberg (1876 - 1932)<\/figcaption><\/figure>\n

Lindeberg, in his 1922 paper, gave a complete proof of the central limit theorem under more general conditions than Liapunov. He introduced the famous Lindeberg condition of which I shall have more to say below. I was happy to make his personal acquaintance at a mathematical congress in Helsingfors in the summer of 1922. He was Professor at the University of Helsingfors, and owned a beautiful farm in the eastern part of the country. When he was reproached for not being sufficiently active in his scientific work, he said, \u201cWell, I am really a farmer.\u201d And if somebody happened to say that his farm was not properly cultivated, his answer was \u201cOf course, my real job is to be a professor.\u201d I was very fond of him, and saw him often during the following years.
\n<\/em><\/p><\/blockquote>\n

Excerpt from Half a century with probability theory. Some personal recollections<\/em>, by Harald Cram\u00e9r (1893 - 1985), published in The Annals of Probability (1976)<\/p>\n

The usual proof of the Central Limit Theorem (CLT) served nowadays in courses is based on characteristic functions (Fourier transform). It is attributed essentially to Lyapunov, but was also (re)discovered by others, and extended notably by L\u00e9vy. Before him, Chebyshev was using the method of moments! This tiny post is devoted to a less usual yet useful approach by coupling, due to Lindeberg. There are plenty of other approaches, such as the method of (Charles) Stein based on integration by parts, and the method of Linnik based on Shannon information.<\/p>\n

Lindeberg replacement or exchange principle, based on coupling.<\/strong>. Let $X_1,\\ldots,X_n$ be independent real random variables with with $\\mathbb{E}(|X_k|^3)<\\infty$. Set $$m_k:=\\mathbb{E}(X_k),\\quad\\sigma^2_k:=\\mathbb{E}(|X_k-m_k|^2),\\quad\\tau_k^3:=\\mathbb{E}(|X_k-m_k|^3).$$Let $Y_1,\\ldots,Y_n$ be independent, and independent of $X_1,\\ldots,X_n$, such that $$Y_k\\sim\\mathcal{N}(m_k,\\sigma^2_k).$$Then for all $f\\in\\mathcal{C}^3(\\mathbb{R},\\mathbb{R})$ with $f,f',f'',f^{(3)}$ bounded,$$|\\mathbb{E}(f(X_1+\\cdots+X_n))-\\mathbb{E}(f(Y_1+\\cdots+Y_n))|\\leq(\\tau_1^3+\\cdots+\\tau_n^3)\\frac{{\\|f^{(3)}\\|}_\\infty}{2}.$$This Lindeberg coupling inequality implies immediately that if $$S_n:=\\frac{X_1-m_1+\\cdots+X_n-m_n}{\\sqrt{\\sigma_1^2+\\cdots+\\sigma_n^2}}\\quad\\text{and}\\quad G\\sim\\mathcal{N}(0,1)$$ then for all $f\\in\\mathcal{C}^3(\\mathbb{R},\\mathbb{R})$ with $f,f',f'',f^{(3)}$ bounded, $$|\\mathbb{E}(f(S_n))-\\mathbb{E}(f(G))|\\leq\\frac{\\tau_1^3+\\cdots+\\tau_n^3}{\\sqrt{\\sigma_1^2+\\cdots+\\sigma_n^2}^3}\\frac{{\\|f^{(3)}\\|}_\\infty}{2}.$$In the iid case, $m_k$, $\\sigma_k$, and $\\tau_k$ no longer depend on $k$, and $$\\frac{\\tau_1^3+\\cdots+\\tau_n^3}{\\sqrt{\\sigma_1^2+\\cdots+\\sigma_n^2}^3}=\\frac{\\tau^3}{\\sigma^3\\sqrt{n}}.$$We thus get a quantitative (non-asymptotic) version of the CLT, in the spirit of the Berry-Esseen inequality. In terms of asymptotic analysis, this also leads to the classical iid CLT under finite third moment by approximating indicators by smooth functions, namely, for all $x\\in\\mathbb{R}$, $\\varepsilon>0$, there exists $f,g\\in\\mathcal{C}^3(\\mathbb{R},\\mathbb{R})$ with $f,f',f'',f^{(3)},g,g',g'',g^{(3)}$ bounded such that $$\\mathbf{1}_{(-\\infty,x-\\varepsilon]}\\leq f_\\varepsilon\\leq\\mathbf{1}_{(-\\infty,x]}\\leq g_\\varepsilon\\leq\\mathbf{1}_{(-\\infty,x+\\varepsilon]}.$$<\/p>\n

Let us prove the Lindeberg coupling inequality above. Since the statement is invariant by translation on $f$, we can assume without loss of generality that $m_k=0$ for all $k$. The idea now is to replace, in $X_1+\\cdots+X_n$, $X_k$ by $Y_k$, step by step. Namely, introducing $$Z_k:=X_1+\\cdots+X_{k-1}+Y_{k+1}\\cdots+Y_n,$$we get the telescopic sum $$f(X_1+\\cdots+X_n)-f(Y_1+\\cdots+Y_n)=\\sum_{k=1}^n(f(Z_k+X_k)-f(Z_k+Y_k)).$$Now, the Taylor-Lagrange formula applied at $Z_k$ at order $2$ gives $$f(Z_k+X_k)=f(Z_k)+f'(Z_k)X_k+f''(Z_k)\\frac{X_k^2}{2!}+f^{(3)}(Z_k)\\frac{A_k^3}{3!}$$where $A_k\\in[Z_k,Z_k+X_k]$. Similarly, $$f(Z_k+Y_k)=f(Z_k)+f'(Z_k)Y_k+f''(Z_k)\\frac{Y_k^2}{2!}+f^{(3)}(Z_k)\\frac{B_k^3}{3!}$$ where $B_k\\in[Z_k,Z_k+Y_k]$. Taking the expectation, using the independence of $(X_k,Y_k)$ and $Z_k$, and using the fact that the first two moments of the $X_k$'s and the $Y_k$'s match, we get $$|f(Z_k+X_k)-f(Z_k+Y_k)|\\leq\\frac{{\\|f^{(3)}\\|}_\\infty}{3!}\\mathbb{E}(|X_k|^3+|Y_k|^3).$$ It remains to note that $Y_k=\\sigma(X_k)G_k$ where $G_k\\sim\\mathcal{N}(0,1)$, hence $$\\mathbb{E}(|Y_k|^3)=\\mathbb{E}(|X_k|^2)^{3\/2}\\mathbb{E}(|G_k|^3)\\leq2\\mathbb{E}(|X_k|^3).$$<\/p>\n

Lindeberg second moment truncation condition.<\/strong> It should not be confused with the Lindeberg replacement principle above. More precisely, for sums of independent square integrable random variables, beyond the finite third moment condition, the CLT holds as soon as the Lindeberg second moment truncation condition holds: for all $\\varepsilon>0$, $$\\lim_{n\\to\\infty}\\frac{1}{\\sigma_1^2+\\cdots+\\sigma_n^2}\\sum_{k=1}^n\\mathbb{E}((X_k-m_k)^2\\mathbf{1}_{\\{|X_k-m_k|\\geq\\varepsilon\\sigma_k\\}})=0.$$ This holds in particular if the Lyapunov moment condition<\/strong> is satisfied: for some $p>1$, $$\\lim_{n\\to\\infty}\\frac{1}{{(\\sigma_1^2+\\cdots+\\sigma_n^2)}^{p}}\\sum_{k=1}^n\\mathbb{E}(|X_k-m_k|^{2p})=0,$$as we can check using the Markov inequality $\\mathbf{1}_{\\{|X_k-m_k|\\geq\\varepsilon\\sigma_k\\}}\\leq\\frac{|X_k-m_k|}{\\varepsilon\\sigma_k}$. These versions of the CLT for sums of independent variables extend to martingales. Lindeberg's work on the CLT was reinvented independently by Alan Turing in his dissertation on the central limit theorem!<\/p>\n

Comments.<\/b> The replacement principle can be used beyond the realm of sums of independent variables, typically for nonlinear stochastic models involving independent ingredients, for instance for the high dimensional asymptotic analysis of the eigenvalues of random matrices.<\/p>\n

Je regardai d'abord les livres classiques de Joseph Bertrand, de Poincar\u00e9, et d'\u00c9mile Borel. Seul Poincar\u00e9 avait esquiss\u00e9 une tentative de justification de la loi de Gauss; mais ce n'\u00e9tait qu'une esquisse. ... II fallait me mettre \u00e0 la t\u00e2che, ce que je fis. Mais je devais avoir des d\u00e9ceptions ; je ne devais d'abord que red\u00e9couvrir des r\u00e9sultats d\u00e9j\u00e0 obtenus en Russie par Tchebichev et Liapounov. Mais, avant de parler des erreurs exp\u00e9rimentales, il fallait parler des principes du calcul des probabilit\u00e9s. Je m'aper\u00e7us alors qu'en un certain sens ce calcul n'existait pas ; il fallait le cr\u00e9er (note: more than ten years before Kolmogorov 1933).<\/em><\/p><\/blockquote>\n

Excerpt from Quelques aspects de la pens\u00e9e d'un math\u00e9maticien<\/em>, by Paul L\u00e9vy (1886 - 1971), published by Librairie Armand Blanchard (1970)<\/p>\n

\u00ab Cette loi ne s'obtient pas par des d\u00e9ductions rigoureuses ; plus d\u2019une d\u00e9monstration qu\u2019on a voulu en donner est grossi\u00e8re, entre autres celle qui s\u2019appuie sur l\u2019affirmation que la probabilit\u00e9 des \u00e9carts est proportionnelle aux \u00e9carts. Tout le monde y croit cependant, me disait un jour M. Lippmann, car les exp\u00e9rimentateurs s\u2019imaginent que c\u2019est un th\u00e9or\u00e8me de math\u00e9matiques, et les math\u00e9maticiens que c\u2019est un fait exp\u00e9rimental. \u00bb <\/em><\/p><\/blockquote>\n

Excerpt from Le calcul des Probabilit\u00e9s<\/em> (1896), by Henri Poincar\u00e9 (1854 - 1912), about the normal law and the CLT known at that time as \u00ab loi des erreurs \u00bb. Quoting the catalan physicist Oriol Bohigas (1925 - 2021), \u00ab de nos jours, nous savons que c'est \u00e0 la fois un fait exp\u00e9rimental et un th\u00e9or\u00e8me de math\u00e9matiques !\u00bb.<\/p>\n

Further reading.<\/strong><\/p>\n