A few words about entropy

Why and how entropy emerges in basic mathematics? This tiny post aims to provide some answers. We have already tried something in this spirit in a previous post almost ten years ago.

Combinatorics. Asymptotic analysis of the multinomial coefficient $\binom{n}{n_1,\ldots,n_r}:=\frac{n!}{n_1!\cdots n_r!}$ :
$$\frac{1}{n}\log\binom{n}{n_1,\ldots,n_r} \xrightarrow[n=n_1+\cdots+n_r\to\infty]{\nu_i=\frac{n_i}{n}\to p_i} \mathrm{S}(p):=-\sum_{i=1}^rp_i\log(p_i).$$ Recall that if $A=\{a_1,\ldots,a_r\}$ is a finite set of cardinal $r$ and $n=n_1+\cdots+n_r$ then
$$\mathrm{Card}\Bigr\{(x_1,\ldots,x_n)\in A^n:\forall 1\leq i\leq r,\sum_{k=1}^n\mathbf{1}_{x_k=a_i}=n_i\Bigr\}=\binom{n}{n_1,\ldots,n_r}.$$ The multinomial coefficient can be interpreted as the number of microstates $(x_1,\ldots,x_n)$ compatible with the macrostate $(n_1,\ldots,n_r)$, while the quantity $\mathrm{S}(p)$ appears as a normalized asymptotic measure of additive degrees of freedom or disorder. This is already in the work of Ludwig Eduard Boltzmann (1844 -- 1906) in kinetic gas theory at the origins of statistical physics. The quantity $\mathrm{S}(p)$ is also the one used by Claude Elwood Shannon (1916 -- 2001) in information and communication theory as the average length of optimal lossless coding. It is characterized by the following three natural axioms or properties, denoting $\mathrm{S}^{(n)}$ to remember $n$ :

• for all $n\geq1$, $p\mapsto\mathrm{S}^{(n)}(p)$ is continuous
• for all $n\geq1$, $\mathrm{S}^{(n)}(\frac{1}{n},\ldots,\frac{1}{n})<\mathrm{S}^{(n+1)}(\frac{1}{n+1},\ldots,\frac{1}{n+1})$
• for all $n=n_1+\cdots+n_r\geq1$, $\mathrm{S}^{(n)}(\frac{1}{n},\ldots,\frac{1}{n})=\mathrm{S}^{(r)}(\frac{n_1}{n},\ldots,\frac{n_r}{n})+\sum_{i=1}^r\frac{n_i}{n}\mathrm{S}^{(n_i)}(\frac{1}{n_i},\ldots,\frac{1}{n_i}).$

Probability. If $X_1,\ldots,X_n$ are independent and identically distributed random variables of law $\mu$ on a finite set or alphabet $A=\{a_1,\ldots,a_r\}$, then for all $x_1,\ldots,x_n\in A$,
$$\begin{align*} \mathbb{P}((X_1,\ldots,X_n)=(x_1,\ldots,x_n)) &=\prod_{i=1}^r\mu_i^{\sum_{k=1}^n\mathbb{1}_{x_k=a_i}} =\prod_{i=1}^r\mu_i^{n\nu_i} =\mathrm{e}^{n\sum_{i=1}^n\nu_i\log\mu_i}\\ &=\mathrm{e}^{-n(\mathrm{S}(\nu)+\mathrm{H}(\nu\mid\mu))}, \end{align*}$$ a remarkable identity where $\mathrm{S}(\nu)$ is the Boltzmann-Shannon entropy considered before, and where $\mathrm{H}(\nu\mid\mu)$ is a new quantity known as the Kullback-Leibler divergence or relative entropy :
$$\mathrm{S}(\nu):=-\sum_{i=1}^r\nu_i\log\nu_i=-\int f(x)\log f(x)\mathrm{d}x$$ where $f$ is the density of $\nu$ with respect to the counting measure $\mathrm{d}x$, and
$$\mathrm{H}(\nu\mid\mu):=\sum_{i=1}^r\nu_i\log\frac{\nu_i}{\mu_i} =\sum_{i=1}^r\frac{\nu_i}{\mu_i}\log\frac{\nu_i}{\mu_i}\mu_i =\int\frac{\mathrm{d}\nu}{\mathrm{d}\mu}\log\frac{\mathrm{d}\nu}{\mathrm{d}\mu}\mathrm{d}\mu.$$ This comes from information theory after Solomon Kullback (1907 - 1994) and Richard Leibler (1914 -- 2003). Here $\mathrm{S}(\nu)$ measures the combinatorics on $x_1,\ldots,x_n$ at prescribed frequencies $\nu$, while $\mathrm{H}(\nu\mid\mu)$ measures the cost or energy of deviation from the actual distribution $\mu$. This is a Boltzmann--Gibbsfication of the probability $\mathbb{P}((X_1,\ldots,X_n)=(x_1,\ldots,x_n))$, see below, leading via the Laplace method to the large deviations principle of Ivan Nikolaevich Sanov (1919 -- 1968). The Jensen inequality for the strictly convex function $u\mapsto u\log(u)$ gives
$$\mathrm{H}(\nu\mid\mu)\geq0\quad\text{with equality iff}\quad\nu=\mu.$$

Statistics. If $Y_1,\ldots,Y_n$ are independent and identically distributed random variables of law $\mu^{(\theta)}$ in parametric family parametrized by $\theta$, on a finite set $A$, then, following Ronald Aylmer Fisher (1890 - 1962), the likelihood of data $(x_1,\ldots,x_n)\in A^n$ is
$$\ell_{x_1,\ldots,x_n}(\theta):=\mathbb{P}(Y_1=x_1,\ldots,Y_n=x_n) =\prod_{i=1}^n\mu^{(\theta)}_{x_i}.$$ It can also be seen as the likelihood of $\theta$ with respect to $x_1,\ldots,x_n$. This dual point of view leads to the following : if $X_1,\ldots,X_n$ is an observed sample of $\mu^{(\theta_*)}$ with $\theta_*$ unknown then the maximum likelihood estimator of $\theta_*$ is
$$\widehat{\theta}_n:=\arg\max_{\theta\in\Theta}\ell_{X_1,\ldots,X_n}(\theta) =\arg\max_{\theta\in\Theta}\Bigr(\frac{1}{n}\log\ell_{X_1,\ldots,X_n}(\theta)\Bigr).$$ The asymptotic analysis via the law of large numbers reveals entropy as asymptotic contrast
$$\begin{align*} \frac{1}{n}\log\ell_{X_1,\ldots,X_n}(\theta) &=\frac{1}{n}\sum_{i=1}^n\log\mu^{(\theta)}_{X_i}\\&\xrightarrow[n\to\infty]{\mathrm{a.s.}} \sum_{k=1}^r\mu^{(\theta_*)}_k\log\mu^{(\theta)}_k =\underbrace{-\mathrm{S}(\mu^{(\theta_*)})}_{\text{const}}-\mathrm{H}(\mu^{(\theta_*)}\mid\mu^{(\theta)}). \end{align*}$$

Analysis. The entropy appears naturally as a derivative of the $L^p$ norm of $f\geq0$ as follows:
$$\partial_p\|f\|_p^p =\partial_p\int f^p\mathrm{d}\mu =\partial_p\int \mathrm{e}^{-p\log(f)}\mathrm{d}\mu =\int f^p\log(f)\mathrm{d}\mu =\frac{1}{p}\int f^p\log(f^p)\mathrm{d}\mu.$$ This is at the heart of the Leonard Gross (1931 -- ) theorem relating the hypercontractivity of Markov semigroups with the logarithmic Sobolev inequality for the invariant measure. This can also be used to extract from the William Henry Young (1843 - 1942) convolution inequalities certain entropic uncertainty principles.

Boltzmann-Gibbs measures, variational characterizations, and Helmholtz free energy. We take $V:A\to\mathbb{R}$, interpreted as an energy. Maximizing $\mu\mapsto\mathrm{S}(\mu)$ over the constraint of average energy $\int V\mathrm{d}\mu=v$ gives the maximizer $$\mu_\beta :=\frac{1}{Z_\beta}\mathrm{e}^{-\beta V}\mathrm{d}x \quad\text{where}\quad Z_\beta:=\int\mathrm{e}^{-\beta V}\mathrm{d}x.$$ We use integrals instead of sums to lightnen notation. The notation $\mathrm{d}x$ stands for the counting measure on $A$. The parameter $\beta>0$, interpreted as inverse temperature, is dictated by $v$. Such a probability distribution $\mu_\beta$ is known as a Boltzmann-Gibbs distribution, after Ludwig Eduard Boltzmann (1844 - 1906) and Josiah Willard Gibbs (1839 - 1903). We have a variational characterization as a maximum entropy at fixed average energy :
$$\int V\mathrm{d}\mu=\int V\mathrm{d}\mu_\beta \quad\Rightarrow\quad \mathrm{S}(\mu_\beta)-\mathrm{S}(\mu) =\mathrm{H}(\mu\mid\mu_\beta).$$ There is a dual point of view in which instead of fixing the average energy, we fix the inverse temperature $\beta$ and we introduce the Hermann von Helmholtz (1821 - 1894) free energy
$$\mathrm{F}(\mu):=\int V\mathrm{d}\mu-\frac{1}{\beta}\mathrm{S}(\mu)$$ This can be seen as a Joseph-Louis Lagrange (1736 - 1813) point of view in which the constraint is added to the functional. We have
$$\mathrm{F}(\mu_\beta)=-\frac{1}{\beta}\log(Z_\beta) \quad\text{since}\quad \mathrm{S}(\mu_\beta)=\beta\int V\mathrm{d}\mu_\beta+\log Z_\beta.$$ We have then a new variational characterization as a minimum free energy at fixed temperature :
$$\mathrm{F}(\mu)-\mathrm{F}(\mu_\beta)=\frac{1}{\beta}\mathrm{H}(\mu\mid\mu_\beta).$$ This explains why $\mathrm{H}$ is often called free energy.

Legrendre transform. The relative entropy $\nu\mapsto\mathrm{H}(\nu\mid\mu)$ is the Legendre transform of the log-Laplace transform, in the sense that
$$\sup_g\Bigr\{\int g\mathrm{d}\nu-\log\int\mathrm{e}^g\mathrm{d}\mu\Bigr\}=\mathrm{H}(\nu\mid\mu).$$ Indeed, for all $h$ such that $\int\mathrm{e}^h\mathrm{d}\mu=1$, by the Jensen inequality, with $f:=\frac{\mathrm{d}\nu}{\mathrm{d}\mu}$,
$$\begin{align*} \int h\mathrm{d}\nu &=\int f\log(f)\mathrm{d}\mu+\int\log\frac{\mathrm{e}^h}{f}f\mathrm{d}\mu\\ &\leq\int f\log(f)\mathrm{d}\mu+\log\int\mathrm{e}^h\mathrm{d}\mu =\int f\log(f)\mathrm{d}\mu=\mathrm{H}(\nu\mid\mu), \end{align*}$$ and equality is achieved for $h=\log f$. It remains to reparametrize with $h=g-\log\int\mathrm{e}^g\mathrm{d}\mu$. Conversely, the Legendre transform of the relative entropy is the log-Laplace transform :
$$\sup_{\nu} \Bigr\{\int g\mathrm{d}\nu-\mathrm{H}(\nu\mid\mu)\Bigr\}=\log\int\mathrm{e}^g\mathrm{d}\mu.$$ This is an instance of the convex duality for the convex functional $\nu\mapsto\mathrm{H}(\nu\mid\mu)$.

Same story for $-\mathrm{S}$ which is convex as a function of the Lebesgue density of its argument.

Heat equation. The heat equation $\partial_tf_t=\Delta f_t$ is the gradient flow of entropy : $$\partial_t\int f_t\log(f_t)\mathrm{d}x=-\int\frac{\|\nabla f_t\|^2}{f_t}\mathrm{d}x$$ where we used integration by parts, the right hand side is the Fisher information. In other words, the entropy is a Lyapunov function for the heat equation seen as an infinite dimensional ODE.

Further reading.

• Boltzmann-Gibbs entropic variational principle
  On this blog (2022)
• Entropy ubiquity
  On this blog (2015)
• Bosons and fermions
  On this blog (2012)