outline

• Motivation

• KOLMOGOROV-NAGUMO AVERAGES

• Rényi entropy 

• EQUILIBRIUM THERMODYNAMICS

• APPLICATIONS TO MULTIFRACTALS

• NONEQUILIBRIUM THERMODYNAMICS

MOTIVATION

• Complexity science is a contemporary field that studies systems with many interacting subsystems and emergent behavior 
• For many complex systems, the standard approaches known from thermodynamics and information theory do not apply
• One way to overcome these issues is to introduce the generalization of ordinary variables known from thermodynamics and information theory
• These "generalized measures" can provide a solution for the description of these systems
• Warning: There is no general measure that can be used for all types of complex systems

MOTIVATION

• Example: some generalized entropies are used because they are maximized by "non-Boltzmann distributions" (e.g., heavy tails)
• There are, however, many more thermodynamic and information properties that are connected to the given entropic functional
• Note: "Three faces of entropy" (a system can be described by more than one entropy)
• Shannon entropy can lead to non-Boltzmann distribution and generalized entropies can lead to Boltzmann distribution
  • The role of constraints
• One has to start with the particular system, not with the particular entropy 

KOLMOGOROV-NAGUMO AVERAGES

• In ancient Greece, three types of averages were studied:
  • arithmetic mean $\frac{1}{n} \sum_i x_i$
  • geometric mean $\sqrt[n]{\prod_i x_i}$
  • harmonic mean $n (\sum_i \frac{1}{x_i})^{-1}$
• These three means have many applications in physics, geometry, and music.
• All three can be generalized into a Hölder mean $\left(\frac{1}{n} \sum_i x_i^p \right)^{1/p}$
• The Hölder mean is used in many disciplines, including functional calculus, probability or signal processing

KOLMOGOROV-NAGUMO AVERAGES

• In 1930, Andrey Kolmogorov and Mitio Nagumo independently introduced a generalization called f-mean for an arbitrary continuous and injective function

$$\langle X \rangle_f :=  f^{-1} \left(\frac{1}{n} \sum_i f(x_i)\right)$$

• The basic property is that the mean is invariant to the transformation $f(x) \mapsto a f(x) + b$ 
• The Hölder mean is obtained for $f(x) = x^p$

KOLMOGOROV-NAGUMO AVERAGES

• The original arithmetic mean can be obtained for $f(x) = x$
• The arithmetic mean can be uniquely determined from the class of KN-mean by the following two properties:

1. Homogeneity $\langle a X \rangle = a \langle X \rangle$
2. Translation invariance $\langle X +c \rangle = \langle X \rangle +c$

• Note that the Hölder means satisfy Property 1 but not property 2
• What is the class of means that satisfies Property 2?

EXPONENTIAL KN AVERAGES

• The solution is given by the function $f(x) = (e^{\gamma x}-1)/\gamma$             and the inverse function is $f^{-1}(x) = \frac{1}{\gamma} \ln (1+\gamma x)$
• This leads to the exponential KN mean$$\langle X \rangle_\gamma = \frac{1}{\gamma} \ln \left(\sum_i \exp(\gamma x_i)\right)$$
• We recover the arithmetic mean by $\gamma \rightarrow 0$
• Property 2 is important for thermodynamics since the average energy $\langle E \rangle$ does not depend on the energy of the ground state
• The additivity property is preserved for independent variables 

$$\langle X + Y \rangle_\gamma = \langle X \rangle_\gamma + \langle Y \rangle_\gamma \Leftrightarrow X \perp \!\!\! \perp  Y$$

EXPONENTIAL KN AVERAGES

•   A general additivity formula is given as $$\langle X + Y \rangle_\gamma = \langle X + \langle Y|X \rangle_\gamma\rangle_\gamma$$ where the conditional mean is given as $$\langle X | Y \rangle_\gamma = \frac{1}{\gamma} \ln \left(\sum_i p_i \exp(\gamma \langle Y|X = x_i\rangle_\gamma ) \right)$$
• The exponential KN mean is closely related to the cumulant generating function  $$M_\gamma(X) = \ln \langle \exp(\gamma X)\rangle = \gamma \langle X \rangle_\gamma$$

RéNYI ENTROPY AS an EXPONENTIAL KN AVERAGE

• Rényi entropy can be naturally obtained as the exponential KN average of the Hartley information $\ln 1/p_k$
• We define the Rényi entropy as $$R_\gamma(P) = \frac{1}{1-\gamma} \langle \ln 1/p_k \rangle_\gamma = \frac{1}{\gamma(1-\gamma)} \ln \sum_k p_k^{1-\gamma}$$
• The role of the prefactor will be clear later, but one of the reasons is that the limit $\gamma \rightarrow 1$ is the well-defined Burg entropy $- \sum_k \ln p_k$
• We also naturally obtain the conditional entropy $$R_\gamma(X,Y) = R_\gamma(X) + R_\gamma(Y|X)$$

EQUILIBRIUM THERMODYNAMICS OF EXPONENTIAL KN AVERAGES

• Let us now consider thermodynamics where all quantities are exponential KN averages
• We consider that the system entropy is described by Rényi entropy 
• The internal energy is describe as the KN average of the energy spectrum $\epsilon$ $$U_\gamma^\beta := \frac{1}{\beta} \langle \beta \epsilon \rangle_\gamma = \frac{1}{\beta \gamma} \ln \sum_i p_i \exp(\beta \gamma \epsilon_i)$$
• The inverse temperature $\beta = \frac{1}{k_B T}$ ensures that the energy has correct units

MAXENT DISTRIBUTION

• Let us now calculate the MaxEnt distribution obtained from Rényi entropy with given constrained of internal energy $U^\beta_\gamma$
• The Lagrange function is $$\mathcal{L} = R_\gamma - \alpha_0 \sum_i p_i - \alpha_1 \frac{1}{\beta}\langle \beta \epsilon \rangle_\gamma$$
• The MaxEnt distribution $\pi_i$ can be obtained from

$$\frac{1}{\gamma} \frac{\pi_i^{-\gamma}}{\sum_k \pi_k^{1-\gamma}} - \alpha_0 - \frac{\alpha_1}{\beta \gamma} \frac{e^{\gamma \beta \epsilon_i}}{\sum_k \pi_k e^{\gamma \beta \epsilon_k}} = 0$$

• By multiplying by $\pi_i$ and summing over $i$ we obtain $\alpha_0 = \frac{1-\alpha_1}{\gamma}$ and therefore $$\pi_i =  \frac{\left(\sum_k \pi_k e^{\gamma \beta \epsilon_k}\right)^{1/\gamma}}{\left(\sum_k \pi_k^{1-\gamma}\right)^{1/\gamma}}  \exp(-\beta \epsilon_i) = \frac{\exp(-\beta \epsilon_i)}{Z^\beta}$$

MAXENT DISTRIBUTION

• We obtained Boltzmann distribution from non-Shannonian entropy with non-arithmetic constraint
• We immediately obtain that $$\ln \pi_i = - \beta(\epsilon_i - U_\gamma^\beta(\pi)) - (1-\gamma) R_\gamma(\pi)$$ which leads to $$\ln \sum_k e^{-\beta \epsilon_k} = (1-\gamma) R_\gamma - \beta U^\beta_\gamma = \Psi_\gamma - \gamma R_\gamma$$ where $\Psi_\gamma = R_\gamma - \beta U_\gamma^\beta$ is the Massieu function 
• The Helmholtz free energy can be expressed as $$F^\beta_\gamma(\pi) = U_\gamma^\beta(\pi)- \frac{1}{\beta} R_\gamma(\pi) = \frac{1}{(\gamma-1) \beta} \ln \sum_k e^{(\gamma-1)\beta \epsilon_k}.$$
• Thus, although we get the same distribution as for the ordinary thermodynamics, the thermodynamic relations are different!

THERMODYNAMIC INTERPRETATION

• What is the interpretation of equilibrium thermodynamics of exponential KN averages?
• Let us denote the equilibrium quantities as 

$$\mathcal{U}_\gamma^\beta = U_\gamma^\beta(\pi)$$   
$$\mathcal{R}_\gamma^\beta = R_\gamma(\pi)$$
$$\mathcal{F}_\gamma^\beta = \mathcal{U}_\gamma^\beta - \frac{1}{\beta} \mathcal{R}_\gamma^\beta$$

• We can now express each quantity to find its interpretation

THERMODYNAMIC INTERPRETATION

• We start with Rényi entropy $$\mathcal{R}_\gamma^\beta = \frac{1}{\gamma(1-\gamma)} \ln \sum_i \left(\frac{e^{-\beta \epsilon_i}}{Z^\beta}\right)^{1-\gamma} = \frac{1}{\gamma(1-\gamma)} \ln \sum_i e^{-(1-\gamma)\beta} - \frac{1}{\gamma} \ln Z^\beta$$
• Thus we get that Rényi entropy of a Boltzmann distribution is $$\mathcal{R}_\gamma^\beta  = -\frac{\beta}{\gamma} \left(\mathcal{F}^{(1-\gamma) \beta} -  \mathcal{F}^{\beta}\right)$$
• By defining $\beta' = (1-\gamma)\beta  \quad \Rightarrow \quad \gamma = 1-\frac{\beta'}{\beta}$ we can express it as 

$$\mathcal{R}_\gamma^\beta =  \beta^2 \, \frac{\mathcal{F}^{\beta'} - \mathcal{F}^\beta}{\beta'-\beta}$$

which is the $\beta$ rescaling of the free energy difference.

• This can be interpreted as the maximum amount of work the system can perform by quenching the system from inverse temperature $\beta$ to inverse temperature $\beta'$.
• This has been first discovered by John Baez.

THERMODYNAMIC INTERPRETATION

• By taking $\gamma \rightarrow 0$ we recover the relation between entropy and free energy $$\mathcal{S}^\beta = \beta^2 \left(\frac{\partial \mathcal{F}^\beta}{\partial \beta}\right)$$
• As shown before, the free energy can be expressed as $$\mathcal{F}_\gamma^\beta = \mathcal{F}^{(1-\gamma)\beta}$$ so it is clear that the temperature is rescaled from $\beta$ to $(1-\gamma)\beta$
• Finally the internal energy can be expressed as $$\mathcal{U}^\beta_\gamma = \frac{\beta' \mathcal{F}^{\beta'} - \beta \mathcal{F}^\beta}{\beta'-\beta} =  - \frac{\Psi^{\beta'}-\Psi^\beta}{\beta'-\beta}$$
• Again, by taking $\gamma \rightarrow 0$ we recover

$$\mathcal{U}^\beta =  - \left(\frac{\partial \Psi^\beta}{\partial \beta}\right)\,$$

APPLICATION TO MULTIFRACTALS

• One of the main applications of Rényi entropy is multifractals
• Let us first remind go through the fundamentals of multifractal analysis 
• Let us consider a physical system whose state space is parcelled into distinct regions $k_i(s)$ indexed by $i\in\mathcal{I}$
• Consider the probability of observing the system within region $k_i(s)$, which is denoted by $p_i(s)$.
• Let us assume that this probability observes a scaling property of the form
  $$p_i(s) = \frac{s^{\alpha_i}}{z(s)}$$

APPLICATION TO MULTIFRACTALS

• For small scales, i.e. $s \rightarrow 0$, let us assume that the frequency of the scaling exponent $\alpha_i$ is given by a continuous probability distribution, whose density $\rho$ has the form

$$\rho(\alpha,s) \mathrm{d} \alpha = c(\alpha) s^{-f(\alpha)} \mathrm{d} \alpha$$

         where $f(\alpha)$ is called as multifractal spectrum

• Let us define the generalized dimension of Rényi entropy as $$\lim_{s \rightarrow 0} \frac{(1-\gamma) R_\gamma(s)}{\ln s} = D_\gamma$$
• We can express Rényi entropy as $$R_\gamma(s) = \frac{1}{\gamma(1-\gamma)} \ln \frac{\int s^{(1-\gamma)\alpha} c(\alpha) s^{-f(\alpha)}  \mathrm{d} \alpha}{\left(\int s^{\alpha} c(\alpha) s^{-f(\alpha)}  \mathrm{d} \alpha\right)^{1-\gamma}}$$
• The relation between the multifractal spectrum and generalized dimension is given by using the stepest descent approximation for $s \rightarrow 0$

APPLICATION TO MULTIFRACTALS

MULTIFRACTAL ENERGY CASCADES

• Let us now make a connection with turbulence cascades.
• We assume that the characteristic energy $\epsilon(s)$ scales as $$\left\langle \epsilon(s)^{\gamma} \right\rangle \sim s^{ M_\gamma(\epsilon)}$$ 
• The cascade spectrum can be connected to the multifractal spectrum by using the fact that the MaxEnt distribution is Boltzmann, i.e., $$\pi_i \propto s^{\alpha_i} \propto e^{-\beta \epsilon_i}$$ which leads to $$\alpha_i \ln s = - \beta \epsilon_i$$
• Thus, the scale is connected with the temperature and the scaling exponent with the energy
• Rescaling of the temperature $\beta \mapsto (1-\gamma) \beta$ corresponds to the change of scale $s \mapsto s^{1-\gamma}$

MULTIFRACTAL ENERGY CASCADES

Finally, the scaling exponents of thermodynamic potentials are

$$\beta U_\beta^{\gamma}(s)  = \frac{\tau_\gamma - \tau_1}{\gamma}\ln s + \mathcal{O}(1)$$
$$\beta F_\beta^{\gamma}(s) = \frac{\tau_\gamma}{\gamma-1} \ln s + \mathcal{O}(1)$$
$$M_\gamma = \gamma U_\gamma^\beta(s) = \tau_1 - \tau_\gamma + o(1)$$

LEGENDRE STRUCTURE

• We consider now a general non-equilibrium state and try to calculate the work done between that state and the equilibrium state
• The minimum work is given by the free energy difference $$W^{rev} = F_\gamma(p)-F_\gamma(\pi) = \Delta U_\gamma^\beta - \tfrac{1}{\beta} \Delta R_\gamma$$
• The difference between non-equilibrium and equilibrium internal energy can be expressed as
  $$U^\beta_\gamma(p) - U^\beta_\gamma(\pi) = \frac{1}{\beta \gamma} \ln  \frac{\sum_i p_i e^{\gamma \beta_0 \epsilon_i}}{\sum_i \pi_i e^{\gamma \beta \epsilon_i}} = \frac{1}{\beta}{\gamma} \ln \left(1 + \frac{\sum_i (p_i-\pi_i) \pi_i^{-\gamma}}{\sum_i \pi_i^{1-\gamma}} \right)$$ $$=  \frac{1}{\beta \gamma} \ln \left(1+ \gamma \,  \nabla R_\gamma \cdot (p-\pi)\right)$$

LEGENDRE STRUCTURE

•  Thus, the free energy difference can be expressed as $$\frac{1}{\beta} \left(F_\gamma(p) - F_\gamma(\pi)\right) =R_\gamma(\pi) - \frac{1}{\beta} R_\gamma(p) +  \frac{1}{\beta \gamma} \ln \left(1+ \gamma  \nabla R_\gamma \cdot (p-\pi)\right)$$
• This can be expressed as the Rényi-Bregmann divergence which is a generalized Bregmann divergence $$\mathcal{D}_{\gamma}(p || \pi) = R_\gamma(\pi) - R_\gamma(p) - C(\nabla R_\gamma,(p-\pi))$$ where $C(x,y)$ is the link function, here $C(x,y) = \frac{1}{\gamma}  \ln (1+ \gamma x \cdot y)$
• We can rewrite it as $$\mathcal{D}_{\gamma}(p || \pi) =  -  R_\gamma(p) - R_\gamma(p,q)$$ where $R_\gamma(p,q)$ is the Rényi cross entropy
• We can also express it as $$\mathcal{D}_{\gamma}(p || \pi) = R_\gamma(\pi) - R_\gamma(p) + \frac{1}{\gamma} \ln \sum p_i \Pi_i^{(\gamma)}$$ where $\Pi_i^{(\gamma)} = \frac{\pi_i^{-\gamma}}{\sum_i \pi_i^{1-\gamma}}$ is the ratio of escort distribution and original distribution

NON-EQUILIBRIUM THERMODYNAMICS

• Let us now consider the case of arbitrary non-equilibrium state and its evolution in time
• We consider the standard master equation $$\dot{p}_i(t) = \sum_j \Big(w_{ij}(t) p_j(t) - w_{ji}(t) p_i(t)\Big)$$
• Since the equilibrium distribution is Boltzmann, we obtain the standard detailed balance $$\frac{w_{ij}(t)}{w_{ji}(t)} = \frac{\pi_i(t)}{\pi_j(t)} = e^{\beta (\epsilon_j(t) - \epsilon_i(t))}$$
• It is possible to show that the second law of thermodynamics hold: $$\dot{\Sigma}_\gamma = \dot{R}_\gamma -  \beta \dot{Q}_\gamma^\beta \geq 0$$
• For relaxation processes ($\epsilon_i$ does not depend on time), we can show that the H-function $H_\gamma(p):= \mathcal{D}_\gamma(p||\pi) = \beta (F_\gamma(p) - F_\gamma(\pi))$ does not increase in time $\dot{H}_\gamma  \leq 0$

ENSEMBLE ENTROPY PRODUCTION

• Finally, we focus on trajectory thermodynamics
• Trajectory entropy production can be expressed as $$\ln \frac{\pmb{P}\big[\pmb{x}\big]}{\tilde{\pmb{P}}\big[\tilde{\pmb{x}}\big]} = \Delta s[\pmb{x}]  + \beta \pmb{q}[\pmb{x}] = \pmb{\sigma}[\pmb{x}]$$
• Here $\pmb{x}$ is a trajectory, $\tilde{\pmb{x}}$ is its time reverse, $\pmb{P}$ is trajectory probability, $\Delta s = \ln p(x_i,t_i) - \ln p(x_f,t_f)$ and $q$ is the trajectory heat
• The ensemble average can be obtained as $$\langle \langle \pmb{\sigma} \rangle \rangle_\gamma = \frac{1}{\gamma} \ln \int \mathcal{D} \pmb{x} \pmb{P}[\pmb{x}] \exp\left(\gamma  \ln \frac{\pmb{P}\big[\pmb{x}\big]}{\tilde{\pmb{P}}\big[\tilde{\pmb{x}}\big]}\right)$$ $$=\frac{1}{\gamma} \ln \int \mathcal{D} \pmb{x} (\pmb{P}[\pmb{x}])^{\gamma+1} (\tilde{\pmb{P}}[\tilde{\pmb{x}}])^{-\gamma} = \mathfrak{D}_\gamma\big(\pmb{P}[{\pmb{x}}]\,||\,\tilde{\pmb{P}}[\tilde{\pmb{x}}]\big) \geq 0$$
• Here $\mathfrak{D}_\gamma$ is the Rényi-Csiszár divergence

Other topics in the paper:

• Trajectory thermodynamics and fluctuation theorems
• Information geometry and thermodynamic length