{"id":8203,"date":"2015-03-16T23:55:07","date_gmt":"2015-03-16T22:55:07","guid":{"rendered":"http:\/\/djalil.chafai.net\/blog\/?p=8203"},"modified":"2022-04-03T13:19:54","modified_gmt":"2022-04-03T11:19:54","slug":"entropy-ubiquity","status":"publish","type":"post","link":"https:\/\/djalil.chafai.net\/blog\/2015\/03\/16\/entropy-ubiquity\/","title":{"rendered":"Entropy ubiquity"},"content":{"rendered":"
\"Ludwig<\/a>
Ludwig Boltzmann (1844 - 1906)<\/figcaption><\/figure>\n

Recently a friend of mine asked about finding a good reason to explain the presence of the Boltzmann-Shannon entropy here and there in mathematics. Well, a vague answer is to simply say that the logarithm is already in many places, waiting for a nice interpretation. A bit less vaguely, here are some concrete fundamental formulas involving the Boltzmann-Shannon entropy \\( {\\mathcal{S}} \\) also denoted \\( {-H} \\).<\/p>\n

Combinatorics.<\/b> If \\( {n=n_1+\\cdots+n_r} \\) and \\( {\\lim_{n\\rightarrow\\infty}\\frac{(n_1,\\ldots,n_r)}{n}=(p_1,\\ldots,p_r)} \\) then<\/p>\n

\\[ \\frac{1}{n}\\log\\binom{n}{n_1,\\ldots,n_r} =\\frac{1}{n}\\log\\frac{n!}{n_1!\\cdots n_r!} \\underset{n\\rightarrow\\infty}{\\longrightarrow} -\\sum_{k=1}^r p_k\\log(p_k)=:\\mathcal{S}(p_1,\\ldots,p_r). \\]<\/p>\n

Also \\( {\\binom{n}{n_1,\\ldots,n_r}\\approx e^{-nH(p_1,\\ldots,p_r)}} \\) when \\( {n\\gg1} \\) and \\( {\\frac{(n_1,\\ldots,n_r)}{n}\\approx (p_1,\\ldots,p_r)} \\). Wonderful!<\/p>\n

Volumetrics.<\/b> In terms of microstates and macrostate we also have<\/p>\n

\\[ \\inf_{\\varepsilon>0} \\varlimsup_{n\\rightarrow\\infty} \\frac{1}{n} \\log\\left| \\left\\{ f:\\{1,\\ldots,n\\}\\rightarrow\\{1,\\ldots,r\\}: \\max_{1\\leq k\\leq r}\\left|\\frac{f^{-1}(k)}{n}-p_k\\right|<\\varepsilon \\right\\}\\right| =\\mathcal{S}(p_1,\\ldots,p_r). \\]<\/p>\n

This formula can be related to the Sanov Large Deviations Principle<\/a>, some sort of refinement of the strong Law of Large Numbers<\/a>.<\/p>\n

Maximization.<\/b> If \\( {\\displaystyle\\int\\!V(x)\\,f(x)\\,dx=\\int\\!V(x)f_\\beta(x)\\,dx} \\) with \\( {f_\\beta(x)=\\frac{e^{-\\beta V(x)}}{Z_\\beta}} \\) then<\/p>\n

\\[ \\mathcal{S}(f_\\beta) - \\mathcal{S}(f) =\\int\\!\\frac{f}{f_\\beta}\\log\\frac{f}{f_\\beta}f_\\beta\\,dx \\geq\\left(\\int\\!\\frac{f}{f_\\beta}f_\\beta\\,dx\\right)\\log\\left(\\int\\!\\frac{f}{f_\\beta}f_\\beta\\,dx\\right)=0. \\]<\/p>\n

This formula plays an important role in statistical physics<\/a> and in Bayesian statistics<\/a>.<\/p>\n

Legendre transform is log-Laplace.<\/b> If \\( {\\mu} \\) and \\( {\\nu} \\) are probability measures then<\/p>\n

\\[ \\sup_{\\substack{g\\geq0\\\\\\int g\\mathrm{d}\\mu=1}} \\Bigr\\{\\int fg\\mathrm{d}\\mu-\\int f\\log f\\mathrm{d}\\mu\\Bigr\\} =\\log\\int\\mathrm{e}^f\\mathrm{d}\\mu \\quad\\text{where}\\quad f:=\\frac{\\mathrm{d}\\nu}{\\mathrm{d}\\mu}. \\]<\/p>\n

and the reverse identity by convex duality<\/p>\n

\\[ \\int f\\log f\\mathrm{d}\\mu =\\sup_{\\int g\\mathrm{d}\\mu=1} \\Bigr\\{\\int fg\\mathrm{d}\\mu-\\log\\int\\mathrm{e}^g\\mathrm{d}\\mu\\Bigr\\}. \\]<\/p>\n

Involved notably in large deviations theory<\/b> and functional inequalities<\/b>.<\/p>\n

Likelihood.<\/b> If \\( {X_1,X_2,\\ldots} \\) are i.i.d. r.v. on \\( {\\mathbb{R}^d} \\) with density \\( {f} \\) then<\/p>\n

\\[ L(f;X_1,\\ldots,X_n) =\\frac{1}{n}\\log(f(X_1,\\ldots,X_n)) \\overset{a.s.}{\\underset{n\\rightarrow\\infty}{\\longrightarrow}} \\int\\!f\\log(f)\\,dx=:-\\mathcal{S}(f). \\]<\/p>\n

This formula allows to reinterpret the maximum likelihood estimator<\/a> as a minimum contrast estimator<\/a> for the Kullback-Leibler divergence<\/a> or relative entropy. It is also at the heart of Shannon coding theorems<\/a> in information theory.<\/p>\n

\\( {L^p} \\) norms.<\/b> If \\( {f\\geq0} \\) then<\/p>\n

\\[ \\partial_{p=1}\\left\\Vert f\\right\\Vert_p^p =\\partial_{p=1}\\int\\!e^{p\\log(f)}\\,dx =\\int\\!f\\log(f)\\,dx =-\\mathcal{S}(f). \\]<\/p>\n

This formula is at the heart of a famous theorem of Leonard Gross<\/a> which relates the hypercontractivity<\/a> of ergodic Markov semigroups<\/a> with a logarithmic Sobolev inequality<\/a> for the invariant measure of the semigroup.<\/p>\n

Fisher information.<\/b> If \\( {\\partial_t f_t(x)=\\Delta f_t(x)} \\) then by integration by parts<\/p>\n

\\[ \\partial_t\\mathcal{S}(f_t) =-\\int\\!\\log(f_t)\\,\\Delta f_t\\,dx =\\int\\!\\frac{\\left|\\nabla f_t\\right|^2}{f_t}\\,dx =\\mathcal{F}(f_t). \\]<\/p>\n

This formula, attribued to de Bruijn<\/a>, is at the heart of the analysis and geometry of heat kernels<\/a>, diffusion processes<\/a>, and gradient flows<\/a> in partial differential equations.<\/p>\n

\"Claude<\/a>
Claude Shannon (1916 - 2001)<\/figcaption><\/figure>\n","protected":false},"excerpt":{"rendered":"

Recently a friend of mine asked about finding a good reason to explain the presence of the Boltzmann-Shannon entropy here and there in mathematics. Well,…<\/p>\n

Continue readingEntropy ubiquity<\/span><\/a><\/div>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":348},"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/djalil.chafai.net\/blog\/wp-json\/wp\/v2\/posts\/8203"}],"collection":[{"href":"https:\/\/djalil.chafai.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/djalil.chafai.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/djalil.chafai.net\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/djalil.chafai.net\/blog\/wp-json\/wp\/v2\/comments?post=8203"}],"version-history":[{"count":18,"href":"https:\/\/djalil.chafai.net\/blog\/wp-json\/wp\/v2\/posts\/8203\/revisions"}],"predecessor-version":[{"id":16040,"href":"https:\/\/djalil.chafai.net\/blog\/wp-json\/wp\/v2\/posts\/8203\/revisions\/16040"}],"wp:attachment":[{"href":"https:\/\/djalil.chafai.net\/blog\/wp-json\/wp\/v2\/media?parent=8203"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/djalil.chafai.net\/blog\/wp-json\/wp\/v2\/categories?post=8203"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/djalil.chafai.net\/blog\/wp-json\/wp\/v2\/tags?post=8203"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}