Foundations of Entropy II
Let's calculate 'em
Lecture series at the
School on Information, Noise, and Physics of Life
Nis 19.-30. September 2022
by Jan Korbel
all slides can be found at: slides.com/jankorbel
Activity II
You have 3 minutes to write down on a piece of paper:
What is the most unexpected/surprising application
of entropy
you have seen? (unexpected field, unexpected result)
Using Boltzmann's formula for non-multinomial systems
As we saw in the previous lecture, the multinomial multiplicity
$$W(n_1,\dots,n_k) = \binom{n}{n_1,\dots,n_k}$$
leads to Boltzmann-Gibbs-Shannon entropy.
Are there systems with non-multinomial multiplicity?
What is their entropy?
Ex. I: MB, FD & BE statistics
1. Maxwell-Boltzmann statistics - \(N\) distinguishable particles, \(N_i\) particles in state \(\epsilon_i\)
Multiplicity can be calculated as
$$W(N_1,\dots,N_k) = \binom{N}{N_1} \binom{N-N_1}{N_2} \dots \binom{N-\sum_{i=1}^{k-1}N_1}{N_k} = N! \prod_{i=1}^k \frac{1}{N_i!}$$
If \(\epsilon_i\) has degeneracy \(g_i\), then
$$W(N_1,\dots,N_k) = N! \prod_{i=1}^k \frac{g_i^{N_i}}{N_i!}$$
Then, we get
$$S_{MB} = - N \sum_{i=1}^k p_i \log \frac{p_i}{g_i}$$
Ex. I: MB, FD & BE statistics
2. Bose-Einstein statistics - \(N\) indistinguishable particles, \(N_i\) particles in state \(\epsilon_i\) with degeneracy \(g_i\),
Multiplicity can be calculated as
$$W(N_1,\dots,N_k) = \prod_{i=1}^k \binom{N_i + g_i-1}{N_i}$$
Let us introduce \(\alpha_i = g_i/N\). Then, we get
$$S_{BE} = N \sum_{i=1}^k \left[(\alpha_i + p_i) \log (\alpha_i +p_i) - \alpha_i \log \alpha_i - p_i \log p_i\right]$$
(|*)
3. Fermi-Dirac statistics - \(N\) indistinguishable particles, \(N_i\) particles in state \(\epsilon_i\) with degeneracy \(g_i\), maximally 1 particle per sub-level (thus \(N_i \leq g_i\))
Multiplicity can be calculated as
$$W(N_1,\dots,N_k) = \prod_{i=1}^k \binom{g_i}{N_i}$$
Let us introduce \(\alpha_i = g_i/N\). Then, we get
$$S_{FD} = N \sum_{i=1}^k \left[-(\alpha_i - p_i) \log (\alpha_i -p_i) + \alpha_i \log \alpha_i - p_i \log p_i\right]$$
Ex. I: MB, FD & BE statistics
Ex. II: structure-forming systems
Let us start with a simple example of a coin tossing.
States are:
But! let's make a small change, we consider magnetic coins
The bound (or sticky) state is simply
State space grows super-exponentially (\(W(n) \sim n^n \sim e^{n \log n}\) )
picture taken from: Jensen et al 2018 J. Phys. A: Math. Theor. 51 375002
Multiplicity
2 x 1x
1 x 1x
3
3
Microstate
Mesostate
Mesostate
2 x 1x
1 x 1x
1 1 2 2 3 3
2 3 1 3 1 2
3 2 3 1 2 1
1 1 2 2 3 3
2 3 1 3 1 2
3 2 3 1 2 1
= (1,2,3) , (2,1,3)
= (1,3,2) , (3,1,2)
= (2,3,1) , (3,2,1)
= (1,2,3) , (1,3,2)
= (2,1,3) , (2,3,1)
= (3,1,2) , (3,2,1)
How to calculate multiplicity
General formula: \(W(n_i^{(j)}) = \frac{n!}{\prod_{ij} n_i^{(j)}! {\color{red} (j!)^{n_i^{(j)}}}}\)
we have \(n_i^{(j)}\) molecules of size \(j\) in a state \(s_i^{(j)}\)
Multiplicity
Boltzmann's 1884 paper
Entropy of structure-forming systems
$$ S = \log W \approx n \log n - \sum_{ij} \left(n_i^{(j)} \log n_i^{(j)} - n_i^{(j)} + {\color{red} n_i^{(j)} \log j!}\right)$$
Introduce "probabilities" \(\wp_i^{(j)} = n_i^{(j)}/n\)
$$\mathcal{S} = S/n = - \sum_{ij} \wp_i^{(j)} (\log \wp_i^{(j)} {\color{red}- 1}) {\color{red}- \sum_{ij} \wp_i^{(j)}\log \frac{j!}{n^{j-1}}}$$
Finite interaction range: concentration \(c = n/b\)
$$\mathcal{S} = S/n = - \sum_{ij} \wp_i^{(j)} (\log \wp_i^{(j)} {\color{red}- 1}) {\color{red}- \sum_{ij} \wp_i^{(j)}\log \frac{j!}{{\color{purple}c^{j-1}}}}$$
Nat. Comm. 12 (2021) 1127
Ex. III: sample-space reducing processes (SSR)
Corominas Murtra, Hanel, Thurner PNAS 112(17) (2015) 5348
Multiplicity
The number of states is \(n\). Let us denote the states as \(x_n \rightarrow \dots \rightarrow x_1\), where \(x_1\) is the ground state, where the process restarts. Let us sample \(R\) relaxation sequences \(x = (x_{k_1},\dots,x_1)\).
The sequences can be visualised as
How many of these sequences contain a state \(x_{j}\) exactly \(k_j\) times?
Each run must contain \(x_1\)
Multiplicity
Number or runs \(R \equiv k_1\), number of them containing \(x_j\) is \(k_j\)
Multiplicity of these sequences: \(\binom{k_1}{k_j}\)
By multiplying the multiplicity for each state we get
$$W(k_1,\dots,k_n) = \prod_{j=2}^n \binom{k_1}{k_j}$$
$$ \log W \approx \sum_{j=2}^n \left[k_1 \log k_1 - \cancel{k_1} - k_j \log k_j + \cancel{k_j} - (k_1-k_j) \log (k_1 - k_j) + \cancel{(k_1-k_j)}\right]$$
$$\approx \sum_{j=2}^n \left[ k_1 \log k_1 \textcolor{red}{-k_j \log k_1} - k_j log \frac{k_j}{\textcolor{red}{k_1}} - (k_1-k_j) \log(k_1-k_j) \right]$$
By introducing \(p_i = k_i/N\) where \(N\) is the total number of steps, we get
$$S_{SSR}(p) = - N \sum_{j=2}^n \left[p_i \log \left(\frac{p_i}{p_1}\right) + (p_1-p_i) \log \left(1-\frac{p_i}{p_1}\right)\right]$$
Ex. IV: Pólya urns
Hanel, Corominas Murtra, Thurner New J. Phys. (2017) 19 033008
Probability of a sequence
We have \(c\) colors, initially \(n_i(0) \equiv n_i\) balls of color \(c_i\). After a ball is drawn, we return \(\delta\) balls of the same color to the urn.
After \(N\) draws, the number of balls in the urn is
$$n_i(N) = n_i + \delta k_i$$
where \(k_i\) is the number of draws of color \(c_i\). The total number of balls is \(n(R) = \sum_{c} n_c(N) = N + \delta N\)
The probability of drawing a ball of color \(c_i\) in \(N\)-th run, is \(p_i(N) = n_i(N)/n(N)\). The probability of sequence
\(\ \mathcal{I} =\{i_1,\dots,i_N\} \) is
$$p(\mathcal{I}) = \prod_{j=1}^c \frac{n_{j}^{(\delta,k_j)}}{n^{(\delta,N)}}$$
where \(m^{(\delta,r)} = m(m+\delta)\dots(m+r\delta)\)
Probability of a histogram
A histogram \(\mathcal{K} = \{k_1,\dots,k_c\}\) is defined as \(k_c = \sum_{j = 1}^N \delta(i_j,c) \)
Thus the probability of observing a histogram is
$$ p(\mathcal{K}) = \binom{N}{k_1,\dots,k_c} p(\mathcal{I}) $$
\(n_{j}^{(\delta,k_{j})} \approx k_{j}! \delta^{k_{j}} (k_i+1)^{n_i/\delta} \)
\( p(\mathcal{K}) = \frac{N!}{\prod_{j=1}^c \cancel{k_j!}} \frac{\prod_{j=1}^c \cancel{k_j!} \delta^{k_j} (k_j+1)^{n_j/\delta}}{n^{(\delta,N)}} \)
... technical calculation ...
$$S_{Pólya}(p) = \log p(\mathcal{K}) \approx - \sum_{i=1}^c \log(p_i + 1/N)$$
Ex. IV: q-deformations
This example is rather theoretical, but provides us a useful hint of what happens if there are correlations in the sample space
Motivation: finite versions of \(\exp\) and \(\log\)
\(\exp(x) = \lim_{n \rightarrow \infty} \left(1+\frac{x}{n}\right)^n\)
So define
\(\exp_q(x) := \left(1 + (1-q) x\right)^{1/(1-q)}\)
\(\log_q(x) := \frac{x^{1-q}-1}{1-q}\)
Let us find an operation s.t.
\(\exp_q(x) \otimes_q \exp_q(y) \equiv \exp_q(x+y)\)
\(\Rightarrow a \otimes_q b = \left[a^{1-q} + b^{1-q}-1\right]^{1/(1-q)}\)
Suyari, Physica A 368 (2006) 63-82
Calculus of q-deformations
In analogy to \(n! = 1 \cdot 2 \cdot \dots \cdot n\) introduce \(n!_q := 1 \otimes_q 2 \otimes_q \dots \otimes_q n\)
It is than easy to show that \(\log_q n!_q = \frac{\sum_{k=1}^n k^{1-q} - n}{1-q}\) which can be used for generalized Stirling's approximation \(\log_q n!_q \approx \frac{n}{2-q} \log_q n\)
Let us now consider a q-deformed multinomial factor
$$\binom{n}{n_1,\dots,n_k}_q := n!_q \oslash_q (n_1!_q \otimes_q \dots \otimes_q n_k!_q) $$
$$ = \left[\sum_{l=1}^n l^{1-q} - \sum_{i_1}^{n_1} i_i^{1-q} - \dots - \sum_{i_k}^{n_k} i_k^{1-q}\right]^{1/(1-q)}$$
Tsallis entropy
Let us consider that the multiplicity is given by a q-multinomial factor \( W(n_1,\dots,n_k) = \binom{n}{n_1,\dots,n_k}_q\)
In this case, it is more convenient to define entropy as \(S = \log_q W\), which gives us:
$$\log_q W = \frac{n^{2-q}}{2-q} \frac{\sum_{i=1}^k p_i^{2-q}-1}{q-1} = \frac{n^{2-q}}{2-q} S_{2-q}(p)$$
This entropy is known as Tsallis entropy
Note that the prefactor is not \(n\) but \(n^{2-q}\)
(non-extensivity) - we will discuss this later
Entropy and energy
Until now, we have been just counting states; let us now discuss the relation with energy.
We consider that the states describe the energy of the system (either Hamiltonian or more generalized energy functional)
Therefore, entropy is defined as
$$ S(E) := \log W(E)$$
Ensembles
There are a few typical situations:
1. Isolated system = microcanonical ensemble
Let \(H(s)\) be the energy of a state \(s\). Multiplicity is then \(W(E) = \sum_{s} \delta(H(s) - E)\)
Phenomena like negative "temperature" \(T = \frac{\mathrm{d} S(E)}{\mathrm{d} E} < 0\)
2. closed system = canonical ensemble
Total system is composed of the system of interest (S) and the heat reservoir/bath (B). They are weakly coupled i.e., \(H_{tot}(s,b) = H_S(s) + H_B(b)\) (no interaction energy)
$$W(E_{tot}) = \sum_{s,b} \delta(H_{tot}(s,b) -E_{tot})$$
3. open system = grandcanonical ensemble
This can be further rewritten as
$$= \int \mathrm{d} E_S\, \sum_s \delta (H_{S}(s)-E_S) \sum_b(H_B(b)-(E_{tot}-E_S)) $$
$$= \int \mathrm{d} E_S \, W_S(E_S) W_B(E_{tot}-E_S) $$
Entropy in canonical ensemble
This is hard to calculate. Typically, the dominant contribution is from the maximal configuration of the integrand, which we obtain from
$$\frac{\partial W_S(E_S) W_B(E_{tot}-E_S)}{\partial E_S} \stackrel{!}{=} 0 \Rightarrow \frac{W'_S(E_S)}{W_S(E)} = \frac{W'_B(E_{tot}-E_S)}{W_B(E_{tot}-E_S)}$$
As a consequence \(\frac{\partial S_E(E_S)}{\partial E_S} \stackrel{!}{=} \frac{\partial S_B(E_{tot}-E_S)}{\partial E_S} := \frac{1}{k_B T}\)
and \(\underbrace{S_B(E_{tot} - E_S)}_{free \ entropy} = \underbrace{S_B(E_{tot})}_{bath \ entropy} - \frac{\partial S}{\partial E_S} E_S + \dots\)
This is the emergence of Maximum entropy principle
Canonical ensemble & Coarse-graining
S
B
weak coupling
S
coarse-graining
Deterministic picture
Statistical
picture