Thermodynamics of exponential kolmogorov-nagumo Averages

Jan Korbel

Pablo Morales

Fernando RoSAS

REFERENCE PAPER: 10.1088/1367-2630/ace4eb (Published in New JOurnal of Physics)

SLIDES AVAILABLE AT: SLIDES.COM/JANKORBEL

outline

Motivation
KOLMOGOROV-NAGUMO AVERAGES
Rényi entropy
EQUILIBRIUM THERMODYNAMICS
APPLICATIONS TO MULTIFRACTALS
NONEQUILIBRIUM THERMODYNAMICS

MOTIVATION

Complexity science is a contemporary field that studies systems with many interacting subsystems and emergent behavior
For many complex systems, the standard approaches known from thermodynamics and information theory do not apply
One way to overcome these issues is to introduce the generalization of ordinary variables known from thermodynamics and information theory
These "generalized measures" can provide a solution for the description of these systems
Warning: There is no general measure that can be used for all types of complex systems

MOTIVATION

Example: some generalized entropies are used because they are maximized by "non-Boltzmann distributions" (e.g., heavy tails)
There are, however, many more thermodynamic and information properties that are connected to the given entropic functional
Note: "Three faces of entropy" (a system can be described by more than one entropy)
Shannon entropy can lead to non-Boltzmann distribution and generalized entropies can lead to Boltzmann distribution
- The role of constraints
One has to start with the particular system, not with the particular entropy

KOLMOGOROV-NAGUMO AVERAGES

In ancient Greece, three types of averages were studied:
- arithmetic mean $\frac{1}{n} \sum_i x_i$
- geometric mean $ \sqrt[n]{\prod_i x_i}$
- harmonic mean $ n (\sum_i \frac{1}{x_i})^{-1}$
These three means have many applications in physics, geometry, and music.
All three can be generalized into a Hölder mean $ \left(\frac{1}{n} \sum_i x_i^p \right)^{1/p} $
The Hölder mean is used in many disciplines, including functional calculus, probability or signal processing

KOLMOGOROV-NAGUMO AVERAGES

In 1930, Andrey Kolmogorov and Mitio Nagumo independently introduced a generalization called f-mean for an arbitrary continuous and injective function

$$\langle X \rangle_f := f^{-1} \left(\frac{1}{n} \sum_i f(x_i)\right)$$

The basic property is that the mean is invariant to the transformation $f(x) \mapsto a f(x) + b$
The Hölder mean is obtained for $f(x) = x^p$

KOLMOGOROV-NAGUMO AVERAGES

The original arithmetic mean can be obtained for $f(x) = x$
The arithmetic mean can be uniquely determined from the class of KN-mean by the following two properties:

Homogeneity $\langle a X \rangle = a \langle X \rangle $
Translation invariance $ \langle X +c \rangle = \langle X \rangle +c$

Note that the Hölder means satisfy Property 1 but not property 2
What is the class of means that satisfies Property 2?

EXPONENTIAL KN AVERAGES

The solution is given by the function $f(x) = (e^{\gamma x}-1)/\gamma$ and the inverse function is $f^{-1}(x) = \frac{1}{\gamma} \ln (1+\gamma x)$
This leads to the exponential KN mean$$ \langle X \rangle_\gamma = \frac{1}{\gamma} \ln \left(\sum_i \exp(\gamma x_i)\right) $$
We recover the arithmetic mean by $\gamma \rightarrow 0$
Property 2 is important for thermodynamics since the average energy $\langle E \rangle$ does not depend on the energy of the ground state
The additivity property is preserved for independent variables

$$\langle X + Y \rangle_\gamma = \langle X \rangle_\gamma + \langle Y \rangle_\gamma \Leftrightarrow X \perp \!\!\! \perp Y$$

EXPONENTIAL KN AVERAGES

A general additivity formula is given as $$ \langle X + Y \rangle_\gamma = \langle X + \langle Y|X \rangle_\gamma\rangle_\gamma$$ where the conditional mean is given as $$ \langle X | Y \rangle_\gamma = \frac{1}{\gamma} \ln \left(\sum_i p_i \exp(\gamma \langle Y|X = x_i\rangle_\gamma ) \right) $$
The exponential KN mean is closely related to the cumulant generating function $$M_\gamma(X) = \ln \langle \exp(\gamma X)\rangle = \gamma \langle X \rangle_\gamma $$

RéNYI ENTROPY AS an EXPONENTIAL KN AVERAGE

Rényi entropy can be naturally obtained as the exponential KN average of the Hartley information $\ln 1/p_k$
We define the Rényi entropy as $$ R_\gamma(P) = \frac{1}{1-\gamma} \langle \ln 1/p_k \rangle_\gamma = \frac{1}{\gamma(1-\gamma)} \ln \sum_k p_k^{1-\gamma}$$
The role of the prefactor will be clear later, but one of the reasons is that the limit $\gamma \rightarrow 1$ is the well-defined Burg entropy $- \sum_k \ln p_k$
We also naturally obtain the conditional entropy $$ R_\gamma(X,Y) = R_\gamma(X) + R_\gamma(Y|X)$$

EQUILIBRIUM THERMODYNAMICS OF EXPONENTIAL KN AVERAGES

Let us now consider thermodynamics where all quantities are exponential KN averages
We consider that the system entropy is described by Rényi entropy
The internal energy is describe as the KN average of the energy spectrum $\epsilon$ $$U_\gamma^\beta := \frac{1}{\beta} \langle \beta \epsilon \rangle_\gamma = \frac{1}{\beta \gamma} \ln \sum_i p_i \exp(\beta \gamma \epsilon_i)$$
The inverse temperature $\beta = \frac{1}{k_B T}$ ensures that the energy has correct units

MAXENT DISTRIBUTION

Let us now calculate the MaxEnt distribution obtained from Rényi entropy with given constrained of internal energy $U^\beta_\gamma $
The Lagrange function is $$ \mathcal{L} = R_\gamma - \alpha_0 \sum_i p_i - \alpha_1 \frac{1}{\beta}\langle \beta \epsilon \rangle_\gamma$$
The MaxEnt distribution $\pi_i$ can be obtained from

$$ \frac{1}{\gamma} \frac{\pi_i^{-\gamma}}{\sum_k \pi_k^{1-\gamma}} - \alpha_0 - \frac{\alpha_1}{\beta \gamma} \frac{e^{\gamma \beta \epsilon_i}}{\sum_k \pi_k e^{\gamma \beta \epsilon_k}} = 0$$

By multiplying by $\pi_i$ and summing over $i$ we obtain $\alpha_0 = \frac{1-\alpha_1}{\gamma}$ and therefore $$\pi_i = \frac{\left(\sum_k \pi_k e^{\gamma \beta \epsilon_k}\right)^{1/\gamma}}{\left(\sum_k \pi_k^{1-\gamma}\right)^{1/\gamma}} \exp(-\beta \epsilon_i) = \frac{\exp(-\beta \epsilon_i)}{Z^\beta}$$

MAXENT DISTRIBUTION

We obtained Boltzmann distribution from non-Shannonian entropy with non-arithmetic constraint
We immediately obtain that $$\ln \pi_i = - \beta(\epsilon_i - U_\gamma^\beta(\pi)) - (1-\gamma) R_\gamma(\pi)$$ which leads to $$\ln \sum_k e^{-\beta \epsilon_k} = (1-\gamma) R_\gamma - \beta U^\beta_\gamma = \Psi_\gamma - \gamma R_\gamma$$ where $\Psi_\gamma = R_\gamma - \beta U_\gamma^\beta$ is the Massieu function
The Helmholtz free energy can be expressed as $$F^\beta_\gamma(\pi) = U_\gamma^\beta(\pi)- \frac{1}{\beta} R_\gamma(\pi) = \frac{1}{(\gamma-1) \beta} \ln \sum_k e^{(\gamma-1)\beta \epsilon_k}.$$
Thus, although we get the same distribution as for the ordinary thermodynamics, the thermodynamic relations are different!

THERMODYNAMIC INTERPRETATION

What is the interpretation of equilibrium thermodynamics of exponential KN averages?
Let us denote the equilibrium quantities as

$$\mathcal{U}_\gamma^\beta = U_\gamma^\beta(\pi)$$
$$\mathcal{R}_\gamma^\beta = R_\gamma(\pi)$$
$$\mathcal{F}_\gamma^\beta = \mathcal{U}_\gamma^\beta - \frac{1}{\beta} \mathcal{R}_\gamma^\beta$$

We can now express each quantity to find its interpretation

THERMODYNAMIC INTERPRETATION

We start with Rényi entropy $$ \mathcal{R}_\gamma^\beta = \frac{1}{\gamma(1-\gamma)} \ln \sum_i \left(\frac{e^{-\beta \epsilon_i}}{Z^\beta}\right)^{1-\gamma} = \frac{1}{\gamma(1-\gamma)} \ln \sum_i e^{-(1-\gamma)\beta} - \frac{1}{\gamma} \ln Z^\beta $$
Thus we get that Rényi entropy of a Boltzmann distribution is $$\mathcal{R}_\gamma^\beta = -\frac{\beta}{\gamma} \left(\mathcal{F}^{(1-\gamma) \beta} - \mathcal{F}^{\beta}\right)$$
By defining $\beta' = (1-\gamma)\beta \quad \Rightarrow \quad \gamma = 1-\frac{\beta'}{\beta}$ we can express it as

$$ \mathcal{R}_\gamma^\beta = \beta^2 \, \frac{\mathcal{F}^{\beta'} - \mathcal{F}^\beta}{\beta'-\beta} $$

which is the $\beta$ rescaling of the free energy difference.

This can be interpreted as the maximum amount of work the system can perform by quenching the system from inverse temperature $\beta$ to inverse temperature $\beta'$.
This has been first discovered by John Baez.

THERMODYNAMIC INTERPRETATION

By taking $\gamma \rightarrow 0$ we recover the relation between entropy and free energy $$\mathcal{S}^\beta = \beta^2 \left(\frac{\partial \mathcal{F}^\beta}{\partial \beta}\right)$$
As shown before, the free energy can be expressed as $$ \mathcal{F}_\gamma^\beta = \mathcal{F}^{(1-\gamma)\beta}$$ so it is clear that the temperature is rescaled from $\beta$ to $(1-\gamma)\beta$
Finally the internal energy can be expressed as $$ \mathcal{U}^\beta_\gamma = \frac{\beta' \mathcal{F}^{\beta'} - \beta \mathcal{F}^\beta}{\beta'-\beta} = - \frac{\Psi^{\beta'}-\Psi^\beta}{\beta'-\beta}$$
Again, by taking $\gamma \rightarrow 0$ we recover

$$ \mathcal{U}^\beta = - \left(\frac{\partial \Psi^\beta}{\partial \beta}\right)\, $$

APPLICATION TO MULTIFRACTALS

One of the main applications of Rényi entropy is multifractals
Let us first remind go through the fundamentals of multifractal analysis
Let us consider a physical system whose state space is parcelled into distinct regions $k_i(s)$ indexed by $i\in\mathcal{I}$
Consider the probability of observing the system within region $k_i(s)$, which is denoted by $p_i(s)$.
Let us assume that this probability observes a scaling property of the form
$$p_i(s) = \frac{s^{\alpha_i}}{z(s)}$$

APPLICATION TO MULTIFRACTALS

For small scales, i.e. $s \rightarrow 0$, let us assume that the frequency of the scaling exponent $\alpha_i$ is given by a continuous probability distribution, whose density $\rho$ has the form

$$\rho(\alpha,s) \mathrm{d} \alpha = c(\alpha) s^{-f(\alpha)} \mathrm{d} \alpha$$

where $f(\alpha)$ is called as multifractal spectrum

Let us define the generalized dimension of Rényi entropy as $$\lim_{s \rightarrow 0} \frac{(1-\gamma) R_\gamma(s)}{\ln s} = D_\gamma$$
We can express Rényi entropy as $$R_\gamma(s) = \frac{1}{\gamma(1-\gamma)} \ln \frac{\int s^{(1-\gamma)\alpha} c(\alpha) s^{-f(\alpha)} \mathrm{d} \alpha}{\left(\int s^{\alpha} c(\alpha) s^{-f(\alpha)} \mathrm{d} \alpha\right)^{1-\gamma}}$$
The relation between the multifractal spectrum and generalized dimension is given by using the stepest descent approximation for $s \rightarrow 0$

APPLICATION TO MULTIFRACTALS

By using the steepest descent, we define $$\alpha_\gamma = \arg\min_\alpha\big\{(1-\gamma) \alpha - f(\alpha)\big\}$$
We introduce the Legendre transform of multifractal spectrum $$\tau_\gamma = (1-\gamma)\alpha_\gamma - f(\alpha_\gamma)$$
Thus, Rényi entropy can be expressed as $$R_\gamma = \frac{\tau_\gamma - (1-\gamma) \tau_1}{\gamma(1-\gamma)} \ln s + \mathcal{O}(1)$$
Therefore, the relation to generalized dimension is $$D_\gamma = \frac{\tau_\gamma - (1-\gamma) \tau_1}{\gamma}$$

MULTIFRACTAL ENERGY CASCADES

Let us now make a connection with turbulence cascades.
We assume that the characteristic energy $\epsilon(s)$ scales as $$\left\langle \epsilon(s)^{\gamma} \right\rangle \sim s^{ M_\gamma(\epsilon)}$$
The cascade spectrum can be connected to the multifractal spectrum by using the fact that the MaxEnt distribution is Boltzmann, i.e., $$\pi_i \propto s^{\alpha_i} \propto e^{-\beta \epsilon_i}$$ which leads to $$\alpha_i \ln s = - \beta \epsilon_i$$
Thus, the scale is connected with the temperature and the scaling exponent with the energy
Rescaling of the temperature $\beta \mapsto (1-\gamma) \beta$ corresponds to the change of scale $s \mapsto s^{1-\gamma}$

MULTIFRACTAL ENERGY CASCADES

Finally, the scaling exponents of thermodynamic potentials are

$$\beta U_\beta^{\gamma}(s) = \frac{\tau_\gamma - \tau_1}{\gamma}\ln s + \mathcal{O}(1)$$
$$\beta F_\beta^{\gamma}(s) = \frac{\tau_\gamma}{\gamma-1} \ln s + \mathcal{O}(1) $$
$$M_\gamma = \gamma U_\gamma^\beta(s) = \tau_1 - \tau_\gamma + o(1) $$

LEGENDRE STRUCTURE

We consider now a general non-equilibrium state and try to calculate the work done between that state and the equilibrium state
The minimum work is given by the free energy difference $$W^{rev} = F_\gamma(p)-F_\gamma(\pi) = \Delta U_\gamma^\beta - \tfrac{1}{\beta} \Delta R_\gamma$$
The difference between non-equilibrium and equilibrium internal energy can be expressed as
$$U^\beta_\gamma(p) - U^\beta_\gamma(\pi) = \frac{1}{\beta \gamma} \ln \frac{\sum_i p_i e^{\gamma \beta_0 \epsilon_i}}{\sum_i \pi_i e^{\gamma \beta \epsilon_i}} = \frac{1}{\beta}{\gamma} \ln \left(1 + \frac{\sum_i (p_i-\pi_i) \pi_i^{-\gamma}}{\sum_i \pi_i^{1-\gamma}} \right)$$ $$ = \frac{1}{\beta \gamma} \ln \left(1+ \gamma \, \nabla R_\gamma \cdot (p-\pi)\right)$$

LEGENDRE STRUCTURE

Thus, the free energy difference can be expressed as $$\frac{1}{\beta} \left(F_\gamma(p) - F_\gamma(\pi)\right) =R_\gamma(\pi) - \frac{1}{\beta} R_\gamma(p) + \frac{1}{\beta \gamma} \ln \left(1+ \gamma \nabla R_\gamma \cdot (p-\pi)\right)$$
This can be expressed as the Rényi-Bregmann divergence which is a generalized Bregmann divergence $$\mathcal{D}_{\gamma}(p || \pi) = R_\gamma(\pi) - R_\gamma(p) - C(\nabla R_\gamma,(p-\pi))$$ where $C(x,y)$ is the link function, here $C(x,y) = \frac{1}{\gamma} \ln (1+ \gamma x \cdot y)$
We can rewrite it as $$\mathcal{D}_{\gamma}(p || \pi) = - R_\gamma(p) - R_\gamma(p,q)$$ where $R_\gamma(p,q)$ is the Rényi cross entropy
We can also express it as $$\mathcal{D}_{\gamma}(p || \pi) = R_\gamma(\pi) - R_\gamma(p) + \frac{1}{\gamma} \ln \sum p_i \Pi_i^{(\gamma)}$$ where $\Pi_i^{(\gamma)} = \frac{\pi_i^{-\gamma}}{\sum_i \pi_i^{1-\gamma}}$ is the ratio of escort distribution and original distribution

NON-EQUILIBRIUM THERMODYNAMICS

Let us now consider the case of arbitrary non-equilibrium state and its evolution in time
We consider the standard master equation $$\dot{p}_i(t) = \sum_j \Big(w_{ij}(t) p_j(t) - w_{ji}(t) p_i(t)\Big)$$
Since the equilibrium distribution is Boltzmann, we obtain the standard detailed balance $$\frac{w_{ij}(t)}{w_{ji}(t)} = \frac{\pi_i(t)}{\pi_j(t)} = e^{\beta (\epsilon_j(t) - \epsilon_i(t))}$$
It is possible to show that the second law of thermodynamics hold: $$\dot{\Sigma}_\gamma = \dot{R}_\gamma - \beta \dot{Q}_\gamma^\beta \geq 0$$
For relaxation processes ($\epsilon_i$ does not depend on time), we can show that the H-function $H_\gamma(p):= \mathcal{D}_\gamma(p||\pi) = \beta (F_\gamma(p) - F_\gamma(\pi))$ does not increase in time $\dot{H}_\gamma \leq 0$

ENSEMBLE ENTROPY PRODUCTION

Finally, we focus on trajectory thermodynamics
Trajectory entropy production can be expressed as $$\ln \frac{\pmb{P}\big[\pmb{x}\big]}{\tilde{\pmb{P}}\big[\tilde{\pmb{x}}\big]} = \Delta s[\pmb{x}] + \beta \pmb{q}[\pmb{x}] = \pmb{\sigma}[\pmb{x}]$$
Here $\pmb{x}$ is a trajectory, $\tilde{\pmb{x}}$ is its time reverse, $\pmb{P}$ is trajectory probability, $\Delta s = \ln p(x_i,t_i) - \ln p(x_f,t_f)$ and $q$ is the trajectory heat
The ensemble average can be obtained as $$ \langle \langle \pmb{\sigma} \rangle \rangle_\gamma = \frac{1}{\gamma} \ln \int \mathcal{D} \pmb{x} \pmb{P}[\pmb{x}] \exp\left(\gamma \ln \frac{\pmb{P}\big[\pmb{x}\big]}{\tilde{\pmb{P}}\big[\tilde{\pmb{x}}\big]}\right)$$ $$=\frac{1}{\gamma} \ln \int \mathcal{D} \pmb{x} (\pmb{P}[\pmb{x}])^{\gamma+1} (\tilde{\pmb{P}}[\tilde{\pmb{x}}])^{-\gamma} = \mathfrak{D}_\gamma\big(\pmb{P}[{\pmb{x}}]\,||\,\tilde{\pmb{P}}[\tilde{\pmb{x}}]\big) \geq 0$$
Here $\mathfrak{D}_\gamma$ is the Rényi-Csiszár divergence

Thermodynamics of exponential kolmogorov-nagumo Averages

Jan Korbel

Pablo Morales

Fernando RoSAS

REFERENCE PAPER: 10.1088/1367-2630/ace4eb (Published in New JOurnal of Physics)

SLIDES AVAILABLE AT: SLIDES.COM/JANKORBEL

outline

Motivation

KOLMOGOROV-NAGUMO AVERAGES

Rényi entropy

EQUILIBRIUM THERMODYNAMICS

APPLICATIONS TO MULTIFRACTALS

NONEQUILIBRIUM THERMODYNAMICS

MOTIVATION

MOTIVATION

KOLMOGOROV-NAGUMO AVERAGES

KOLMOGOROV-NAGUMO AVERAGES

KOLMOGOROV-NAGUMO AVERAGES

EXPONENTIAL KN AVERAGES

EXPONENTIAL KN AVERAGES

RéNYI ENTROPY AS an EXPONENTIAL KN AVERAGE

EQUILIBRIUM THERMODYNAMICS OF EXPONENTIAL KN AVERAGES

MAXENT DISTRIBUTION

MAXENT DISTRIBUTION

THERMODYNAMIC INTERPRETATION

THERMODYNAMIC INTERPRETATION

THERMODYNAMIC INTERPRETATION

APPLICATION TO MULTIFRACTALS

APPLICATION TO MULTIFRACTALS

APPLICATION TO MULTIFRACTALS

MULTIFRACTAL ENERGY CASCADES

MULTIFRACTAL ENERGY CASCADES

LEGENDRE STRUCTURE

LEGENDRE STRUCTURE

NON-EQUILIBRIUM THERMODYNAMICS

ENSEMBLE ENTROPY PRODUCTION