Lecture series at the
School on Information, Noise, and Physics of Life
Nis 19.-30. September 2022
by Jan Korbel
all slides can be found at: slides.com/jankorbel
You have 3 minutes to write down on a piece of paper:
What is the most important
result/implication/phenomenon
that is related to entropy?
Spin glass
Now we do the opposite approach compared to lecture II.
We postulate the properties we think entropy should have
and derive the corresponding entropic funcional
These axiomatic approaches have different nature, we will discuss their possible connection
you know them from the other lectures
Introduced independently by Shannon and Khinchin
Motivated by information theory
These four axioms uniquely determine Shannon entropy
S(P)=−∑ipilogpi
SK axioms serve as a starting point for other axiomatic schemes
Several axiomatic schemes generalize axiom SK4.
One possibility is to generalize additivity. The most prominent example is q-additivity
S(A∪B)=S(A)⊕qS(B∣A)
where x⊕qy=x+y+(1−q)xy is q-addition
S(B∣A)=∑iρi(q)AS(B∣A=ai) is conditional entropy
and ρi=piq/∑kpkq is escort distribution.
This uniquely determines Tsallis entropy
Sq(p)=1−q1(i∑piq−1)
Abe, Phys. Lett. A 271 (2000) 74.
Another possibility is to consider a different type of averaging
In the original SK axioms, the conditional entropy is defined as the arithmetic average of S(B∣A=ai)
We can use alternative averaging, as Kolmogoro-Nagumo average
⟨X⟩f=f−1(i∑pif(xi))
By keeping addivity, but taking S(B∣A)=f−1(∑iρi(q)Af(S(B∣A=ai))
for f(x)=1−qe(1−q)x−1 we uniquely obtain Rényi entropy
Rq(p)=1−q1logi∑piq
Jizba, Arimitsu, Annals of Physics 312 (1) (2004)17-59
We have been mentioning the issue of extensivity before
Let us see how the multiplicity and entropy scales with size N
This allows us to introduce a classification of entropies
How the sample space changes when we rescale its size N↦λN?
The ratio behaves like W(N)W(λN)∼λc0 for N→∞
the exponent can be extracted by dλd∣λ=1: c0=lim→∞W(N)NW′(N)
For the leading term we have W(N)∼Nc0.
Is it only possible scaling? We have W(N)W(λN)(λN)c0Nc0∼1
Let us use the other rescaling N↦Nλ
The we get that W(N)W(Nλ)Nλc0Nc0∼λc1
First correction is W(N)∼Nc0(logN)c1
It is the same scaling like for (c,d)-entropy
Can we go further?
J.K., R.H., S.T. New J. Phys. 20 (2018) 093007
Process | S(W) | |||
---|---|---|---|---|
Random walk |
0 |
1 |
0 |
|
Aging random walk |
0 |
2 |
0 |
|
Magnetic coins * |
0 |
1 |
-1 |
|
Random network |
0 |
1/2 |
0 |
|
Random walk cascade |
0 |
0 |
1 |
logW
(logW)2
(logW)1/2
loglogW
d0
d1
d2
logW/loglogW
* H. Jensen et al. J. Phys. A: Math. Theor. 51 375002
W(N)=2N
W(N)≈2N/2∼2N1/2
W(N)≈NN/2 e2N∼eNlogN
W(N)=2(2N)∼2N2
W(N)=22N−1∼22N
How does it change for one more scaling exponent?
R.H., S.T. EPL 93 (2011) 20006
To fulfill SK axiom 2 (maximality): dl>0, to fulfill SK axiom 3 (expandability): d0<1
P.J., J.K. Phys. Rev. Lett. 122 (2019), 120601
J. E. Shore, R. W. Johnson. Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy. IEEE Trans. Inf. Theor. 26(1) (1980), 26. - only Shannon
J. Uffink, Can the Maximum Entropy Principle be Explained as a Consistency Requirement? Stud. Hist. Phil. Mod. Phys. 26(3), (1995), 223. - larger class of entropies including Tsallis, Rényi, ..
S. Pressé, K. Ghosh, J. Lee, K.A. Dill, Nonadditive Entropies Yield Probability Distributions with Biases not Warranted by the Data. Phys. Rev. Lett., 111 (2013), 180604. - only Shannon - not Tsallis
C. Tsallis, Conceptual Inadequacy of the Shore and Johnson Axioms for Wide Classes of Complex Systems. Entropy 17(5), (2015), 2853. - S.-J. axioms are not adequate
S. Pressé K. Ghosh, J. Lee, K.A. Dill, Reply to C. Tsallis’ Conceptual Inadequacy of the Shore and Johnson Axioms for Wide Classes of Complex Systems. Entropy 17(7), (2015), 5043. - S.-J. axioms are adequate
B. Bagci, T. Oikonomou, Rényi entropy yields artificial biases not in the data and incorrect updating due to the finite-size data Phys. Rev. E 99 (2019) 032134 - only Shannon - not Rényi
P. Jizba, J.K. Phys. Rev. Lett. 122 (2019), 120601 - Uffink is correct!
(and the show goes on)
Are the axioms set by theory of information and statistical inference different or can we find some overlap?
Let us consider the 4th SK axiom
in the form equivalent to composability axiom by P. Tempesta:
4. S(A∪B)=f[f−1(S(A))⋅f−1(S(B∣A))]
S(B∣A)=S(B) if B is independent of A.
Entropies fulfilling SK and SJ: Sqf(P)=f(i∑piq)1/(1−q)=f[expq(i∑pilogq(1/pi))]
Phys. Rev. E 101, 042126 (2020)
In ST lecture, you saw that Shannon entropy fulfills the second law of thermodynamics for linear Markov dynamics with detailed balance.
But is it the only possible entropy?
Our axioms are:
1. Linear Markov evolution - p˙m=∑n(wmnpn−wnmpm)
2. Detailed balance - wmnpnst=wnmpmst
3. Second law of thermodynamics: S˙=S˙i+S˙e
where S˙e=βQ, and S˙i≥0 where S˙i=0⇔p=pst
New J. Phys. 23 (2021) 033049
Then S=−∑mpmlogpm
This is a special case of more general result connecting non-linear master equations and generalized entropies