Thermodynamics of exponential kolmogorov-nagumo Averages

Jan Korbel

Pablo Morales

Fernando RoSAS

REFERENCE PAPER: 10.1088/1367-2630/ace4eb (Published in New JOurnal of Physics)

SLIDES AVAILABLE AT: SLIDES.COM/JANKORBEL

outline

• Motivation

• KOLMOGOROV-NAGUMO AVERAGES

• Rényi entropy 

• EQUILIBRIUM THERMODYNAMICS

• APPLICATIONS TO MULTIFRACTALS

• NONEQUILIBRIUM THERMODYNAMICS

MOTIVATION

• Complexity science is a contemporary field that studies systems with many interacting subsystems and emergent behavior 
• For many complex systems, the standard approaches known from thermodynamics and information theory do not apply
• One way to overcome these issues is to introduce the generalization of ordinary variables known from thermodynamics and information theory
• These "generalized measures" can provide a solution for the description of these systems
• Warning: There is no general measure that can be used for all types of complex systems

MOTIVATION

• Example: some generalized entropies are used because they are maximized by "non-Boltzmann distributions" (e.g., heavy tails)
• There are, however, many more thermodynamic and information properties that are connected to the given entropic functional
• Note: "Three faces of entropy" (a system can be described by more than one entropy)
• Shannon entropy can lead to non-Boltzmann distribution and generalized entropies can lead to Boltzmann distribution
  • The role of constraints
• One has to start with the particular system, not with the particular entropy 

KOLMOGOROV-NAGUMO AVERAGES

• In ancient Greece, three types of averages were studied:
  • arithmetic mean \(\frac{1}{n} \sum_i x_i\)
  • geometric mean \( \sqrt[n]{\prod_i x_i}\)
  • harmonic mean \( n (\sum_i \frac{1}{x_i})^{-1}\)
• These three means have many applications in physics, geometry, and music.
• All three can be generalized into a Hölder mean \( \left(\frac{1}{n} \sum_i x_i^p \right)^{1/p} \)
• The Hölder mean is used in many disciplines, including functional calculus, probability or signal processing

KOLMOGOROV-NAGUMO AVERAGES

• In 1930, Andrey Kolmogorov and Mitio Nagumo independently introduced a generalization called f-mean for an arbitrary continuous and injective function

$$\langle X \rangle_f :=  f^{-1} \left(\frac{1}{n} \sum_i f(x_i)\right)$$

• The basic property is that the mean is invariant to the transformation \(f(x) \mapsto a f(x) + b\) 
• The Hölder mean is obtained for \(f(x) = x^p\)

KOLMOGOROV-NAGUMO AVERAGES

• The original arithmetic mean can be obtained for \(f(x) = x\)
• The arithmetic mean can be uniquely determined from the class of KN-mean by the following two properties:

1. Homogeneity \(\langle a X \rangle = a \langle X \rangle \)
2. Translation invariance \( \langle X +c \rangle = \langle X \rangle +c\)

• Note that the Hölder means satisfy Property 1 but not property 2
• What is the class of means that satisfies Property 2?

EXPONENTIAL KN AVERAGES

• The solution is given by the function \(f(x) = (e^{\gamma x}-1)/\gamma\)             and the inverse function is \(f^{-1}(x) = \frac{1}{\gamma} \ln (1+\gamma x)\)
• This leads to the exponential KN mean$$ \langle X \rangle_\gamma = \frac{1}{\gamma} \ln \left(\sum_i \exp(\gamma x_i)\right) $$
• We recover the arithmetic mean by \(\gamma \rightarrow 0\)
• Property 2 is important for thermodynamics since the average energy \(\langle E \rangle\) does not depend on the energy of the ground state
• The additivity property is preserved for independent variables 

$$\langle X + Y \rangle_\gamma = \langle X \rangle_\gamma + \langle Y \rangle_\gamma \Leftrightarrow X \perp \!\!\! \perp  Y$$

EXPONENTIAL KN AVERAGES

•   A general additivity formula is given as $$ \langle X + Y \rangle_\gamma = \langle X + \langle Y|X \rangle_\gamma\rangle_\gamma$$ where the conditional mean is given as $$ \langle X | Y \rangle_\gamma = \frac{1}{\gamma} \ln \left(\sum_i p_i \exp(\gamma \langle Y|X = x_i\rangle_\gamma ) \right) $$
• The exponential KN mean is closely related to the cumulant generating function  $$M_\gamma(X) = \ln \langle \exp(\gamma X)\rangle = \gamma \langle X \rangle_\gamma $$

RéNYI ENTROPY AS an EXPONENTIAL KN AVERAGE

• Rényi entropy can be naturally obtained as the exponential KN average of the Hartley information \(\ln 1/p_k\)
• We define the Rényi entropy as $$ R_\gamma(P) = \frac{1}{1-\gamma} \langle \ln 1/p_k \rangle_\gamma = \frac{1}{\gamma(1-\gamma)} \ln \sum_k p_k^{1-\gamma}$$
• The role of the prefactor will be clear later, but one of the reasons is that the limit \(\gamma \rightarrow 1\) is the well-defined Burg entropy \(- \sum_k \ln p_k\)
• We also naturally obtain the conditional entropy $$ R_\gamma(X,Y) = R_\gamma(X) + R_\gamma(Y|X)$$

EQUILIBRIUM THERMODYNAMICS OF EXPONENTIAL KN AVERAGES

• Let us now consider thermodynamics where all quantities are exponential KN averages
• We consider that the system entropy is described by Rényi entropy 
• The internal energy is describe as the KN average of the energy spectrum \(\epsilon\) $$U_\gamma^\beta := \frac{1}{\beta} \langle \beta \epsilon \rangle_\gamma = \frac{1}{\beta \gamma} \ln \sum_i p_i \exp(\beta \gamma \epsilon_i)$$
• The inverse temperature \(\beta = \frac{1}{k_B T}\) ensures that the energy has correct units

MAXENT DISTRIBUTION

• Let us now calculate the MaxEnt distribution obtained from Rényi entropy with given constrained of internal energy \(U^\beta_\gamma \)
• The Lagrange function is $$ \mathcal{L} = R_\gamma - \alpha_0 \sum_i p_i - \alpha_1 \frac{1}{\beta}\langle \beta \epsilon \rangle_\gamma$$
• The MaxEnt distribution \(\pi_i\) can be obtained from

$$ \frac{1}{\gamma} \frac{\pi_i^{-\gamma}}{\sum_k \pi_k^{1-\gamma}} - \alpha_0 - \frac{\alpha_1}{\beta \gamma} \frac{e^{\gamma \beta \epsilon_i}}{\sum_k \pi_k e^{\gamma \beta \epsilon_k}} = 0$$

• By multiplying by \(\pi_i\) and summing over \(i\) we obtain \(\alpha_0 = \frac{1-\alpha_1}{\gamma}\) and therefore $$\pi_i =  \frac{\left(\sum_k \pi_k e^{\gamma \beta \epsilon_k}\right)^{1/\gamma}}{\left(\sum_k \pi_k^{1-\gamma}\right)^{1/\gamma}}  \exp(-\beta \epsilon_i) = \frac{\exp(-\beta \epsilon_i)}{Z^\beta}$$

MAXENT DISTRIBUTION

• We obtained Boltzmann distribution from non-Shannonian entropy with non-arithmetic constraint
• We immediately obtain that $$\ln \pi_i = - \beta(\epsilon_i - U_\gamma^\beta(\pi)) - (1-\gamma) R_\gamma(\pi)$$ which leads to $$\ln \sum_k e^{-\beta \epsilon_k} = (1-\gamma) R_\gamma - \beta U^\beta_\gamma = \Psi_\gamma - \gamma R_\gamma$$ where \(\Psi_\gamma = R_\gamma - \beta U_\gamma^\beta\) is the Massieu function 
• The Helmholtz free energy can be expressed as $$F^\beta_\gamma(\pi) = U_\gamma^\beta(\pi)- \frac{1}{\beta} R_\gamma(\pi) = \frac{1}{(\gamma-1) \beta} \ln \sum_k e^{(\gamma-1)\beta \epsilon_k}.$$
• Thus, although we get the same distribution as for the ordinary thermodynamics, the thermodynamic relations are different!

THERMODYNAMIC INTERPRETATION

• What is the interpretation of equilibrium thermodynamics of exponential KN averages?
• Let us denote the equilibrium quantities as 

$$\mathcal{U}_\gamma^\beta = U_\gamma^\beta(\pi)$$   
$$\mathcal{R}_\gamma^\beta = R_\gamma(\pi)$$
$$\mathcal{F}_\gamma^\beta = \mathcal{U}_\gamma^\beta - \frac{1}{\beta} \mathcal{R}_\gamma^\beta$$

• We can now express each quantity to find its interpretation

THERMODYNAMIC INTERPRETATION

• We start with Rényi entropy $$ \mathcal{R}_\gamma^\beta = \frac{1}{\gamma(1-\gamma)} \ln \sum_i \left(\frac{e^{-\beta \epsilon_i}}{Z^\beta}\right)^{1-\gamma} = \frac{1}{\gamma(1-\gamma)} \ln \sum_i e^{-(1-\gamma)\beta} - \frac{1}{\gamma} \ln Z^\beta $$
• Thus we get that Rényi entropy of a Boltzmann distribution is $$\mathcal{R}_\gamma^\beta  = -\frac{\beta}{\gamma} \left(\mathcal{F}^{(1-\gamma) \beta} -  \mathcal{F}^{\beta}\right)$$
• By defining \(\beta' = (1-\gamma)\beta  \quad \Rightarrow \quad \gamma = 1-\frac{\beta'}{\beta}\) we can express it as 

$$  \mathcal{R}_\gamma^\beta =  \beta^2 \, \frac{\mathcal{F}^{\beta'} - \mathcal{F}^\beta}{\beta'-\beta} $$

which is the \(\beta\) rescaling of the free energy difference.

• This can be interpreted as the maximum amount of work the system can perform by quenching the system from inverse temperature \(\beta\) to inverse temperature \(\beta'\).
• This has been first discovered by John Baez.

THERMODYNAMIC INTERPRETATION

• By taking \(\gamma \rightarrow 0\) we recover the relation between entropy and free energy $$\mathcal{S}^\beta = \beta^2 \left(\frac{\partial \mathcal{F}^\beta}{\partial \beta}\right)$$
• As shown before, the free energy can be expressed as $$ \mathcal{F}_\gamma^\beta = \mathcal{F}^{(1-\gamma)\beta}$$ so it is clear that the temperature is rescaled from \(\beta\) to \((1-\gamma)\beta\)
• Finally the internal energy can be expressed as $$ \mathcal{U}^\beta_\gamma = \frac{\beta' \mathcal{F}^{\beta'} - \beta \mathcal{F}^\beta}{\beta'-\beta} =  - \frac{\Psi^{\beta'}-\Psi^\beta}{\beta'-\beta}$$
• Again, by taking \(\gamma \rightarrow 0\) we recover

$$ \mathcal{U}^\beta =  - \left(\frac{\partial \Psi^\beta}{\partial \beta}\right)\,  $$

APPLICATION TO MULTIFRACTALS

• One of the main applications of Rényi entropy is multifractals
• Let us first remind go through the fundamentals of multifractal analysis 
• Let us consider a physical system whose state space is parcelled into distinct regions \(k_i(s)\) indexed by \(i\in\mathcal{I}\)
• Consider the probability of observing the system within region \(k_i(s)\), which is denoted by \(p_i(s)\).
• Let us assume that this probability observes a scaling property of the form
  $$p_i(s) = \frac{s^{\alpha_i}}{z(s)}$$

APPLICATION TO MULTIFRACTALS

• For small scales, i.e. \(s \rightarrow 0\), let us assume that the frequency of the scaling exponent \(\alpha_i\) is given by a continuous probability distribution, whose density \(\rho\) has the form

$$\rho(\alpha,s) \mathrm{d} \alpha = c(\alpha) s^{-f(\alpha)} \mathrm{d} \alpha$$

         where \(f(\alpha)\) is called as multifractal spectrum

• Let us define the generalized dimension of Rényi entropy as $$\lim_{s \rightarrow 0} \frac{(1-\gamma) R_\gamma(s)}{\ln s} = D_\gamma$$
• We can express Rényi entropy as $$R_\gamma(s) = \frac{1}{\gamma(1-\gamma)} \ln \frac{\int s^{(1-\gamma)\alpha} c(\alpha) s^{-f(\alpha)}  \mathrm{d} \alpha}{\left(\int s^{\alpha} c(\alpha) s^{-f(\alpha)}  \mathrm{d} \alpha\right)^{1-\gamma}}$$
• The relation between the multifractal spectrum and generalized dimension is given by using the stepest descent approximation for \(s \rightarrow 0\)

APPLICATION TO MULTIFRACTALS

MULTIFRACTAL ENERGY CASCADES

• Let us now make a connection with turbulence cascades.
• We assume that the characteristic energy \(\epsilon(s)\) scales as $$\left\langle \epsilon(s)^{\gamma} \right\rangle \sim s^{ M_\gamma(\epsilon)}$$ 
• The cascade spectrum can be connected to the multifractal spectrum by using the fact that the MaxEnt distribution is Boltzmann, i.e., $$\pi_i \propto s^{\alpha_i} \propto e^{-\beta \epsilon_i}$$ which leads to $$\alpha_i \ln s = - \beta \epsilon_i$$
• Thus, the scale is connected with the temperature and the scaling exponent with the energy
• Rescaling of the temperature \(\beta \mapsto (1-\gamma) \beta\) corresponds to the change of scale \(s \mapsto s^{1-\gamma}\)

MULTIFRACTAL ENERGY CASCADES

Finally, the scaling exponents of thermodynamic potentials are

$$\beta U_\beta^{\gamma}(s)  = \frac{\tau_\gamma - \tau_1}{\gamma}\ln s + \mathcal{O}(1)$$
$$\beta F_\beta^{\gamma}(s) = \frac{\tau_\gamma}{\gamma-1} \ln s + \mathcal{O}(1) $$
$$M_\gamma = \gamma U_\gamma^\beta(s) = \tau_1 - \tau_\gamma + o(1) $$

LEGENDRE STRUCTURE

• We consider now a general non-equilibrium state and try to calculate the work done between that state and the equilibrium state
• The minimum work is given by the free energy difference $$W^{rev} = F_\gamma(p)-F_\gamma(\pi) = \Delta U_\gamma^\beta - \tfrac{1}{\beta} \Delta R_\gamma$$
• The difference between non-equilibrium and equilibrium internal energy can be expressed as
  $$U^\beta_\gamma(p) - U^\beta_\gamma(\pi) = \frac{1}{\beta \gamma} \ln  \frac{\sum_i p_i e^{\gamma \beta_0 \epsilon_i}}{\sum_i \pi_i e^{\gamma \beta \epsilon_i}} = \frac{1}{\beta}{\gamma} \ln \left(1 + \frac{\sum_i (p_i-\pi_i) \pi_i^{-\gamma}}{\sum_i \pi_i^{1-\gamma}} \right)$$ $$ =  \frac{1}{\beta \gamma} \ln \left(1+ \gamma \,  \nabla R_\gamma \cdot (p-\pi)\right)$$

LEGENDRE STRUCTURE

•  Thus, the free energy difference can be expressed as $$\frac{1}{\beta} \left(F_\gamma(p) - F_\gamma(\pi)\right) =R_\gamma(\pi) - \frac{1}{\beta} R_\gamma(p) +  \frac{1}{\beta \gamma} \ln \left(1+ \gamma  \nabla R_\gamma \cdot (p-\pi)\right)$$
• This can be expressed as the Rényi-Bregmann divergence which is a generalized Bregmann divergence $$\mathcal{D}_{\gamma}(p || \pi) = R_\gamma(\pi) - R_\gamma(p) - C(\nabla R_\gamma,(p-\pi))$$ where \(C(x,y)\) is the link function, here \(C(x,y) = \frac{1}{\gamma}  \ln (1+ \gamma x \cdot y)\)
• We can rewrite it as $$\mathcal{D}_{\gamma}(p || \pi) =  -  R_\gamma(p) - R_\gamma(p,q)$$ where \(R_\gamma(p,q)\) is the Rényi cross entropy
• We can also express it as $$\mathcal{D}_{\gamma}(p || \pi) = R_\gamma(\pi) - R_\gamma(p) + \frac{1}{\gamma} \ln \sum p_i \Pi_i^{(\gamma)}$$ where \(\Pi_i^{(\gamma)} = \frac{\pi_i^{-\gamma}}{\sum_i \pi_i^{1-\gamma}}\) is the ratio of escort distribution and original distribution

NON-EQUILIBRIUM THERMODYNAMICS

• Let us now consider the case of arbitrary non-equilibrium state and its evolution in time
• We consider the standard master equation $$\dot{p}_i(t) = \sum_j \Big(w_{ij}(t) p_j(t) - w_{ji}(t) p_i(t)\Big)$$
• Since the equilibrium distribution is Boltzmann, we obtain the standard detailed balance $$\frac{w_{ij}(t)}{w_{ji}(t)} = \frac{\pi_i(t)}{\pi_j(t)} = e^{\beta (\epsilon_j(t) - \epsilon_i(t))}$$
• It is possible to show that the second law of thermodynamics hold: $$\dot{\Sigma}_\gamma = \dot{R}_\gamma -  \beta \dot{Q}_\gamma^\beta \geq 0$$
• For relaxation processes (\(\epsilon_i\) does not depend on time), we can show that the H-function \(H_\gamma(p):= \mathcal{D}_\gamma(p||\pi) = \beta (F_\gamma(p) - F_\gamma(\pi))\) does not increase in time \(\dot{H}_\gamma  \leq 0\)

ENSEMBLE ENTROPY PRODUCTION

• Finally, we focus on trajectory thermodynamics
• Trajectory entropy production can be expressed as $$\ln \frac{\pmb{P}\big[\pmb{x}\big]}{\tilde{\pmb{P}}\big[\tilde{\pmb{x}}\big]} = \Delta s[\pmb{x}]  + \beta \pmb{q}[\pmb{x}] = \pmb{\sigma}[\pmb{x}]$$
• Here \(\pmb{x}\) is a trajectory, \(\tilde{\pmb{x}}\) is its time reverse, \(\pmb{P}\) is trajectory probability, \(\Delta s = \ln p(x_i,t_i) - \ln p(x_f,t_f)\) and \(q\) is the trajectory heat
• The ensemble average can be obtained as $$ \langle \langle \pmb{\sigma} \rangle \rangle_\gamma = \frac{1}{\gamma} \ln \int \mathcal{D} \pmb{x} \pmb{P}[\pmb{x}] \exp\left(\gamma  \ln \frac{\pmb{P}\big[\pmb{x}\big]}{\tilde{\pmb{P}}\big[\tilde{\pmb{x}}\big]}\right)$$ $$=\frac{1}{\gamma} \ln \int \mathcal{D} \pmb{x} (\pmb{P}[\pmb{x}])^{\gamma+1} (\tilde{\pmb{P}}[\tilde{\pmb{x}}])^{-\gamma} = \mathfrak{D}_\gamma\big(\pmb{P}[{\pmb{x}}]\,||\,\tilde{\pmb{P}}[\tilde{\pmb{x}}]\big) \geq 0$$
• Here \(\mathfrak{D}_\gamma\) is the Rényi-Csiszár divergence

Other topics in the paper:

• Trajectory thermodynamics and fluctuation theorems
• Information geometry and thermodynamic length