Thermodynamics of exponential kolmogorovnagumo Averages
Jan Korbel
Pablo Morales
Fernando RoSAS
REFERENCE PAPER: 10.1088/13672630/ace4eb (Published in New JOurnal of Physics)
SLIDES AVAILABLE AT: SLIDES.COM/JANKORBEL
outline

Motivation

KOLMOGOROVNAGUMO AVERAGES

Rényi entropy

EQUILIBRIUM THERMODYNAMICS

APPLICATIONS TO MULTIFRACTALS

NONEQUILIBRIUM THERMODYNAMICS
MOTIVATION
 Complexity science is a contemporary field that studies systems with many interacting subsystems and emergent behavior
 For many complex systems, the standard approaches known from thermodynamics and information theory do not apply
 One way to overcome these issues is to introduce the generalization of ordinary variables known from thermodynamics and information theory
 These "generalized measures" can provide a solution for the description of these systems
 Warning: There is no general measure that can be used for all types of complex systems
MOTIVATION
 Example: some generalized entropies are used because they are maximized by "nonBoltzmann distributions" (e.g., heavy tails)
 There are, however, many more thermodynamic and information properties that are connected to the given entropic functional
 Note: "Three faces of entropy" (a system can be described by more than one entropy)
 Shannon entropy can lead to nonBoltzmann distribution and generalized entropies can lead to Boltzmann distribution
 The role of constraints
 One has to start with the particular system, not with the particular entropy
KOLMOGOROVNAGUMO AVERAGES
 In ancient Greece, three types of averages were studied:
 arithmetic mean \(\frac{1}{n} \sum_i x_i\)
 geometric mean \( \sqrt[n]{\prod_i x_i}\)
 harmonic mean \( n (\sum_i \frac{1}{x_i})^{1}\)
 These three means have many applications in physics, geometry, and music.
 All three can be generalized into a Hölder mean \( \left(\frac{1}{n} \sum_i x_i^p \right)^{1/p} \)
 The Hölder mean is used in many disciplines, including functional calculus, probability or signal processing
KOLMOGOROVNAGUMO AVERAGES
 In 1930, Andrey Kolmogorov and Mitio Nagumo independently introduced a generalization called fmean for an arbitrary continuous and injective function
$$\langle X \rangle_f := f^{1} \left(\frac{1}{n} \sum_i f(x_i)\right)$$
 The basic property is that the mean is invariant to the transformation \(f(x) \mapsto a f(x) + b\)
 The Hölder mean is obtained for \(f(x) = x^p\)
KOLMOGOROVNAGUMO AVERAGES
 The original arithmetic mean can be obtained for \(f(x) = x\)
 The arithmetic mean can be uniquely determined from the class of KNmean by the following two properties:
 Homogeneity \(\langle a X \rangle = a \langle X \rangle \)
 Translation invariance \( \langle X +c \rangle = \langle X \rangle +c\)
 Note that the Hölder means satisfy Property 1 but not property 2
 What is the class of means that satisfies Property 2?
EXPONENTIAL KN AVERAGES
 The solution is given by the function \(f(x) = (e^{\gamma x}1)/\gamma\) and the inverse function is \(f^{1}(x) = \frac{1}{\gamma} \ln (1+\gamma x)\)
 This leads to the exponential KN mean$$ \langle X \rangle_\gamma = \frac{1}{\gamma} \ln \left(\sum_i \exp(\gamma x_i)\right) $$
 We recover the arithmetic mean by \(\gamma \rightarrow 0\)
 Property 2 is important for thermodynamics since the average energy \(\langle E \rangle\) does not depend on the energy of the ground state
 The additivity property is preserved for independent variables
$$\langle X + Y \rangle_\gamma = \langle X \rangle_\gamma + \langle Y \rangle_\gamma \Leftrightarrow X \perp \!\!\! \perp Y$$
EXPONENTIAL KN AVERAGES
 A general additivity formula is given as $$ \langle X + Y \rangle_\gamma = \langle X + \langle YX \rangle_\gamma\rangle_\gamma$$ where the conditional mean is given as $$ \langle X  Y \rangle_\gamma = \frac{1}{\gamma} \ln \left(\sum_i p_i \exp(\gamma \langle YX = x_i\rangle_\gamma ) \right) $$
 The exponential KN mean is closely related to the cumulant generating function $$M_\gamma(X) = \ln \langle \exp(\gamma X)\rangle = \gamma \langle X \rangle_\gamma $$
RéNYI ENTROPY AS an EXPONENTIAL KN AVERAGE
 Rényi entropy can be naturally obtained as the exponential KN average of the Hartley information \(\ln 1/p_k\)
 We define the Rényi entropy as $$ R_\gamma(P) = \frac{1}{1\gamma} \langle \ln 1/p_k \rangle_\gamma = \frac{1}{\gamma(1\gamma)} \ln \sum_k p_k^{1\gamma}$$
 The role of the prefactor will be clear later, but one of the reasons is that the limit \(\gamma \rightarrow 1\) is the welldefined Burg entropy \( \sum_k \ln p_k\)
 We also naturally obtain the conditional entropy $$ R_\gamma(X,Y) = R_\gamma(X) + R_\gamma(YX)$$
EQUILIBRIUM THERMODYNAMICS OF EXPONENTIAL KN AVERAGES
 Let us now consider thermodynamics where all quantities are exponential KN averages
 We consider that the system entropy is described by Rényi entropy
 The internal energy is describe as the KN average of the energy spectrum \(\epsilon\) $$U_\gamma^\beta := \frac{1}{\beta} \langle \beta \epsilon \rangle_\gamma = \frac{1}{\beta \gamma} \ln \sum_i p_i \exp(\beta \gamma \epsilon_i)$$
 The inverse temperature \(\beta = \frac{1}{k_B T}\) ensures that the energy has correct units
MAXENT DISTRIBUTION
 Let us now calculate the MaxEnt distribution obtained from Rényi entropy with given constrained of internal energy \(U^\beta_\gamma \)
 The Lagrange function is $$ \mathcal{L} = R_\gamma  \alpha_0 \sum_i p_i  \alpha_1 \frac{1}{\beta}\langle \beta \epsilon \rangle_\gamma$$
 The MaxEnt distribution \(\pi_i\) can be obtained from
$$ \frac{1}{\gamma} \frac{\pi_i^{\gamma}}{\sum_k \pi_k^{1\gamma}}  \alpha_0  \frac{\alpha_1}{\beta \gamma} \frac{e^{\gamma \beta \epsilon_i}}{\sum_k \pi_k e^{\gamma \beta \epsilon_k}} = 0$$
 By multiplying by \(\pi_i\) and summing over \(i\) we obtain \(\alpha_0 = \frac{1\alpha_1}{\gamma}\) and therefore $$\pi_i = \frac{\left(\sum_k \pi_k e^{\gamma \beta \epsilon_k}\right)^{1/\gamma}}{\left(\sum_k \pi_k^{1\gamma}\right)^{1/\gamma}} \exp(\beta \epsilon_i) = \frac{\exp(\beta \epsilon_i)}{Z^\beta}$$
MAXENT DISTRIBUTION
 We obtained Boltzmann distribution from nonShannonian entropy with nonarithmetic constraint
 We immediately obtain that $$\ln \pi_i =  \beta(\epsilon_i  U_\gamma^\beta(\pi))  (1\gamma) R_\gamma(\pi)$$ which leads to $$\ln \sum_k e^{\beta \epsilon_k} = (1\gamma) R_\gamma  \beta U^\beta_\gamma = \Psi_\gamma  \gamma R_\gamma$$ where \(\Psi_\gamma = R_\gamma  \beta U_\gamma^\beta\) is the Massieu function
 The Helmholtz free energy can be expressed as $$F^\beta_\gamma(\pi) = U_\gamma^\beta(\pi) \frac{1}{\beta} R_\gamma(\pi) = \frac{1}{(\gamma1) \beta} \ln \sum_k e^{(\gamma1)\beta \epsilon_k}.$$
 Thus, although we get the same distribution as for the ordinary thermodynamics, the thermodynamic relations are different!
THERMODYNAMIC INTERPRETATION
 What is the interpretation of equilibrium thermodynamics of exponential KN averages?
 Let us denote the equilibrium quantities as
$$\mathcal{U}_\gamma^\beta = U_\gamma^\beta(\pi)$$
$$\mathcal{R}_\gamma^\beta = R_\gamma(\pi)$$
$$\mathcal{F}_\gamma^\beta = \mathcal{U}_\gamma^\beta  \frac{1}{\beta} \mathcal{R}_\gamma^\beta$$
 We can now express each quantity to find its interpretation
THERMODYNAMIC INTERPRETATION
 We start with Rényi entropy $$ \mathcal{R}_\gamma^\beta = \frac{1}{\gamma(1\gamma)} \ln \sum_i \left(\frac{e^{\beta \epsilon_i}}{Z^\beta}\right)^{1\gamma} = \frac{1}{\gamma(1\gamma)} \ln \sum_i e^{(1\gamma)\beta}  \frac{1}{\gamma} \ln Z^\beta $$
 Thus we get that Rényi entropy of a Boltzmann distribution is $$\mathcal{R}_\gamma^\beta = \frac{\beta}{\gamma} \left(\mathcal{F}^{(1\gamma) \beta}  \mathcal{F}^{\beta}\right)$$
 By defining \(\beta' = (1\gamma)\beta \quad \Rightarrow \quad \gamma = 1\frac{\beta'}{\beta}\) we can express it as
$$ \mathcal{R}_\gamma^\beta = \beta^2 \, \frac{\mathcal{F}^{\beta'}  \mathcal{F}^\beta}{\beta'\beta} $$
which is the \(\beta\) rescaling of the free energy difference.
 This can be interpreted as the maximum amount of work the system can perform by quenching the system from inverse temperature \(\beta\) to inverse temperature \(\beta'\).
 This has been first discovered by John Baez.
THERMODYNAMIC INTERPRETATION
 By taking \(\gamma \rightarrow 0\) we recover the relation between entropy and free energy $$\mathcal{S}^\beta = \beta^2 \left(\frac{\partial \mathcal{F}^\beta}{\partial \beta}\right)$$
 As shown before, the free energy can be expressed as $$ \mathcal{F}_\gamma^\beta = \mathcal{F}^{(1\gamma)\beta}$$ so it is clear that the temperature is rescaled from \(\beta\) to \((1\gamma)\beta\)
 Finally the internal energy can be expressed as $$ \mathcal{U}^\beta_\gamma = \frac{\beta' \mathcal{F}^{\beta'}  \beta \mathcal{F}^\beta}{\beta'\beta} =  \frac{\Psi^{\beta'}\Psi^\beta}{\beta'\beta}$$
 Again, by taking \(\gamma \rightarrow 0\) we recover
$$ \mathcal{U}^\beta =  \left(\frac{\partial \Psi^\beta}{\partial \beta}\right)\, $$
APPLICATION TO MULTIFRACTALS
 One of the main applications of Rényi entropy is multifractals
 Let us first remind go through the fundamentals of multifractal analysis
 Let us consider a physical system whose state space is parcelled into distinct regions \(k_i(s)\) indexed by \(i\in\mathcal{I}\)
 Consider the probability of observing the system within region \(k_i(s)\), which is denoted by \(p_i(s)\).
 Let us assume that this probability observes a scaling property of the form
$$p_i(s) = \frac{s^{\alpha_i}}{z(s)}$$
APPLICATION TO MULTIFRACTALS
 For small scales, i.e. \(s \rightarrow 0\), let us assume that the frequency of the scaling exponent \(\alpha_i\) is given by a continuous probability distribution, whose density \(\rho\) has the form
$$\rho(\alpha,s) \mathrm{d} \alpha = c(\alpha) s^{f(\alpha)} \mathrm{d} \alpha$$
where \(f(\alpha)\) is called as multifractal spectrum
 Let us define the generalized dimension of Rényi entropy as $$\lim_{s \rightarrow 0} \frac{(1\gamma) R_\gamma(s)}{\ln s} = D_\gamma$$
 We can express Rényi entropy as $$R_\gamma(s) = \frac{1}{\gamma(1\gamma)} \ln \frac{\int s^{(1\gamma)\alpha} c(\alpha) s^{f(\alpha)} \mathrm{d} \alpha}{\left(\int s^{\alpha} c(\alpha) s^{f(\alpha)} \mathrm{d} \alpha\right)^{1\gamma}}$$
 The relation between the multifractal spectrum and generalized dimension is given by using the stepest descent approximation for \(s \rightarrow 0\)
APPLICATION TO MULTIFRACTALS
 By using the steepest descent, we define $$\alpha_\gamma = \arg\min_\alpha\big\{(1\gamma) \alpha  f(\alpha)\big\}$$
 We introduce the Legendre transform of multifractal spectrum $$\tau_\gamma = (1\gamma)\alpha_\gamma  f(\alpha_\gamma)$$
 Thus, Rényi entropy can be expressed as $$R_\gamma = \frac{\tau_\gamma  (1\gamma) \tau_1}{\gamma(1\gamma)} \ln s + \mathcal{O}(1)$$
 Therefore, the relation to generalized dimension is $$D_\gamma = \frac{\tau_\gamma  (1\gamma) \tau_1}{\gamma}$$
MULTIFRACTAL ENERGY CASCADES
 Let us now make a connection with turbulence cascades.
 We assume that the characteristic energy \(\epsilon(s)\) scales as $$\left\langle \epsilon(s)^{\gamma} \right\rangle \sim s^{ M_\gamma(\epsilon)}$$
 The cascade spectrum can be connected to the multifractal spectrum by using the fact that the MaxEnt distribution is Boltzmann, i.e., $$\pi_i \propto s^{\alpha_i} \propto e^{\beta \epsilon_i}$$ which leads to $$\alpha_i \ln s =  \beta \epsilon_i$$
 Thus, the scale is connected with the temperature and the scaling exponent with the energy
 Rescaling of the temperature \(\beta \mapsto (1\gamma) \beta\) corresponds to the change of scale \(s \mapsto s^{1\gamma}\)
MULTIFRACTAL ENERGY CASCADES
Finally, the scaling exponents of thermodynamic potentials are
$$\beta U_\beta^{\gamma}(s) = \frac{\tau_\gamma  \tau_1}{\gamma}\ln s + \mathcal{O}(1)$$
$$\beta F_\beta^{\gamma}(s) = \frac{\tau_\gamma}{\gamma1} \ln s + \mathcal{O}(1) $$
$$M_\gamma = \gamma U_\gamma^\beta(s) = \tau_1  \tau_\gamma + o(1) $$
LEGENDRE STRUCTURE
 We consider now a general nonequilibrium state and try to calculate the work done between that state and the equilibrium state
 The minimum work is given by the free energy difference $$W^{rev} = F_\gamma(p)F_\gamma(\pi) = \Delta U_\gamma^\beta  \tfrac{1}{\beta} \Delta R_\gamma$$
 The difference between nonequilibrium and equilibrium internal energy can be expressed as
$$U^\beta_\gamma(p)  U^\beta_\gamma(\pi) = \frac{1}{\beta \gamma} \ln \frac{\sum_i p_i e^{\gamma \beta_0 \epsilon_i}}{\sum_i \pi_i e^{\gamma \beta \epsilon_i}} = \frac{1}{\beta}{\gamma} \ln \left(1 + \frac{\sum_i (p_i\pi_i) \pi_i^{\gamma}}{\sum_i \pi_i^{1\gamma}} \right)$$ $$ = \frac{1}{\beta \gamma} \ln \left(1+ \gamma \, \nabla R_\gamma \cdot (p\pi)\right)$$
LEGENDRE STRUCTURE
 Thus, the free energy difference can be expressed as $$\frac{1}{\beta} \left(F_\gamma(p)  F_\gamma(\pi)\right) =R_\gamma(\pi)  \frac{1}{\beta} R_\gamma(p) + \frac{1}{\beta \gamma} \ln \left(1+ \gamma \nabla R_\gamma \cdot (p\pi)\right)$$
 This can be expressed as the RényiBregmann divergence which is a generalized Bregmann divergence $$\mathcal{D}_{\gamma}(p  \pi) = R_\gamma(\pi)  R_\gamma(p)  C(\nabla R_\gamma,(p\pi))$$ where \(C(x,y)\) is the link function, here \(C(x,y) = \frac{1}{\gamma} \ln (1+ \gamma x \cdot y)\)
 We can rewrite it as $$\mathcal{D}_{\gamma}(p  \pi) =  R_\gamma(p)  R_\gamma(p,q)$$ where \(R_\gamma(p,q)\) is the Rényi cross entropy
 We can also express it as $$\mathcal{D}_{\gamma}(p  \pi) = R_\gamma(\pi)  R_\gamma(p) + \frac{1}{\gamma} \ln \sum p_i \Pi_i^{(\gamma)}$$ where \(\Pi_i^{(\gamma)} = \frac{\pi_i^{\gamma}}{\sum_i \pi_i^{1\gamma}}\) is the ratio of escort distribution and original distribution
NONEQUILIBRIUM THERMODYNAMICS
 Let us now consider the case of arbitrary nonequilibrium state and its evolution in time
 We consider the standard master equation $$\dot{p}_i(t) = \sum_j \Big(w_{ij}(t) p_j(t)  w_{ji}(t) p_i(t)\Big)$$
 Since the equilibrium distribution is Boltzmann, we obtain the standard detailed balance $$\frac{w_{ij}(t)}{w_{ji}(t)} = \frac{\pi_i(t)}{\pi_j(t)} = e^{\beta (\epsilon_j(t)  \epsilon_i(t))}$$
 It is possible to show that the second law of thermodynamics hold: $$\dot{\Sigma}_\gamma = \dot{R}_\gamma  \beta \dot{Q}_\gamma^\beta \geq 0$$
 For relaxation processes (\(\epsilon_i\) does not depend on time), we can show that the Hfunction \(H_\gamma(p):= \mathcal{D}_\gamma(p\pi) = \beta (F_\gamma(p)  F_\gamma(\pi))\) does not increase in time \(\dot{H}_\gamma \leq 0\)
ENSEMBLE ENTROPY PRODUCTION
 Finally, we focus on trajectory thermodynamics
 Trajectory entropy production can be expressed as $$\ln \frac{\pmb{P}\big[\pmb{x}\big]}{\tilde{\pmb{P}}\big[\tilde{\pmb{x}}\big]} = \Delta s[\pmb{x}] + \beta \pmb{q}[\pmb{x}] = \pmb{\sigma}[\pmb{x}]$$
 Here \(\pmb{x}\) is a trajectory, \(\tilde{\pmb{x}}\) is its time reverse, \(\pmb{P}\) is trajectory probability, \(\Delta s = \ln p(x_i,t_i)  \ln p(x_f,t_f)\) and \(q\) is the trajectory heat
 The ensemble average can be obtained as $$ \langle \langle \pmb{\sigma} \rangle \rangle_\gamma = \frac{1}{\gamma} \ln \int \mathcal{D} \pmb{x} \pmb{P}[\pmb{x}] \exp\left(\gamma \ln \frac{\pmb{P}\big[\pmb{x}\big]}{\tilde{\pmb{P}}\big[\tilde{\pmb{x}}\big]}\right)$$ $$=\frac{1}{\gamma} \ln \int \mathcal{D} \pmb{x} (\pmb{P}[\pmb{x}])^{\gamma+1} (\tilde{\pmb{P}}[\tilde{\pmb{x}}])^{\gamma} = \mathfrak{D}_\gamma\big(\pmb{P}[{\pmb{x}}]\,\,\tilde{\pmb{P}}[\tilde{\pmb{x}}]\big) \geq 0$$
 Here \(\mathfrak{D}_\gamma\) is the RényiCsiszár divergence
Other topics in the paper:
 Trajectory thermodynamics and fluctuation theorems
 Information geometry and thermodynamic length