# MOTIVATION

• Complexity science is a contemporary field that studies systems with many interacting subsystems and emergent behavior
• For many complex systems, the standard approaches known from thermodynamics and information theory do not apply
• One way to overcome these issues is to introduce the generalization of ordinary variables known from thermodynamics and information theory
• These "generalized measures" can provide a solution for the description of these systems
• Warning: There is no general measure that can be used for all types of complex systems

# MOTIVATION

• Example: some generalized entropies are used because they are maximized by "non-Boltzmann distributions" (e.g., heavy tails)
• There are, however, many more thermodynamic and information properties that are connected to the given entropic functional
• Note: "Three faces of entropy" (a system can be described by more than one entropy)
• Shannon entropy can lead to non-Boltzmann distribution and generalized entropies can lead to Boltzmann distribution
• The role of constraints
• One has to start with the particular system, not with the particular entropy

# KOLMOGOROV-NAGUMO AVERAGES

• In ancient Greece, three types of averages were studied:
• arithmetic mean $$\frac{1}{n} \sum_i x_i$$
• geometric mean $$\sqrt[n]{\prod_i x_i}$$
• harmonic mean $$n (\sum_i \frac{1}{x_i})^{-1}$$
• These three means have many applications in physics, geometry, and music.
• All three can be generalized into a Hölder mean $$\left(\frac{1}{n} \sum_i x_i^p \right)^{1/p}$$
• The Hölder mean is used in many disciplines, including functional calculus, probability or signal processing

# KOLMOGOROV-NAGUMO AVERAGES

• In 1930, Andrey Kolmogorov and Mitio Nagumo independently introduced a generalization called f-mean for an arbitrary continuous and injective function

$$\langle X \rangle_f := f^{-1} \left(\frac{1}{n} \sum_i f(x_i)\right)$$

• The basic property is that the mean is invariant to the transformation $$f(x) \mapsto a f(x) + b$$
• The Hölder mean is obtained for $$f(x) = x^p$$

# KOLMOGOROV-NAGUMO AVERAGES

• The original arithmetic mean can be obtained for $$f(x) = x$$
• The arithmetic mean can be uniquely determined from the class of KN-mean by the following two properties:
1. Homogeneity $$\langle a X \rangle = a \langle X \rangle$$
2. Translation invariance $$\langle X +c \rangle = \langle X \rangle +c$$
• Note that the Hölder means satisfy Property 1 but not property 2
• What is the class of means that satisfies Property 2?

# EXPONENTIAL KN AVERAGES

• The solution is given by the function $$f(x) = (e^{\gamma x}-1)/\gamma$$             and the inverse function is $$f^{-1}(x) = \frac{1}{\gamma} \ln (1+\gamma x)$$
• This leads to the exponential KN mean$$\langle X \rangle_\gamma = \frac{1}{\gamma} \ln \left(\sum_i \exp(\gamma x_i)\right)$$
• We recover the arithmetic mean by $$\gamma \rightarrow 0$$
• Property 2 is important for thermodynamics since the average energy $$\langle E \rangle$$ does not depend on the energy of the ground state
• The additivity property is preserved for independent variables

$$\langle X + Y \rangle_\gamma = \langle X \rangle_\gamma + \langle Y \rangle_\gamma \Leftrightarrow X \perp \!\!\! \perp Y$$

# EXPONENTIAL KN AVERAGES

•   A general additivity formula is given as $$\langle X + Y \rangle_\gamma = \langle X + \langle Y|X \rangle_\gamma\rangle_\gamma$$ where the conditional mean is given as $$\langle X | Y \rangle_\gamma = \frac{1}{\gamma} \ln \left(\sum_i p_i \exp(\gamma \langle Y|X = x_i\rangle_\gamma ) \right)$$
• The exponential KN mean is closely related to the cumulant generating function  $$M_\gamma(X) = \ln \langle \exp(\gamma X)\rangle = \gamma \langle X \rangle_\gamma$$

# RéNYI ENTROPY AS an EXPONENTIAL KN AVERAGE

• Rényi entropy can be naturally obtained as the exponential KN average of the Hartley information $$\ln 1/p_k$$
• We define the Rényi entropy as $$R_\gamma(P) = \frac{1}{1-\gamma} \langle \ln 1/p_k \rangle_\gamma = \frac{1}{\gamma(1-\gamma)} \ln \sum_k p_k^{1-\gamma}$$
• The role of the prefactor will be clear later, but one of the reasons is that the limit $$\gamma \rightarrow 1$$ is the well-defined Burg entropy $$- \sum_k \ln p_k$$
• We also naturally obtain the conditional entropy $$R_\gamma(X,Y) = R_\gamma(X) + R_\gamma(Y|X)$$

# EQUILIBRIUM THERMODYNAMICS OF EXPONENTIAL KN AVERAGES

• Let us now consider thermodynamics where all quantities are exponential KN averages
• We consider that the system entropy is described by Rényi entropy
• The internal energy is describe as the KN average of the energy spectrum $$\epsilon$$ $$U_\gamma^\beta := \frac{1}{\beta} \langle \beta \epsilon \rangle_\gamma = \frac{1}{\beta \gamma} \ln \sum_i p_i \exp(\beta \gamma \epsilon_i)$$
• The inverse temperature $$\beta = \frac{1}{k_B T}$$ ensures that the energy has correct units

# MAXENT DISTRIBUTION

• Let us now calculate the MaxEnt distribution obtained from Rényi entropy with given constrained of internal energy $$U^\beta_\gamma$$
• The Lagrange function is $$\mathcal{L} = R_\gamma - \alpha_0 \sum_i p_i - \alpha_1 \frac{1}{\beta}\langle \beta \epsilon \rangle_\gamma$$
• The MaxEnt distribution $$\pi_i$$ can be obtained from

$$\frac{1}{\gamma} \frac{\pi_i^{-\gamma}}{\sum_k \pi_k^{1-\gamma}} - \alpha_0 - \frac{\alpha_1}{\beta \gamma} \frac{e^{\gamma \beta \epsilon_i}}{\sum_k \pi_k e^{\gamma \beta \epsilon_k}} = 0$$

• By multiplying by $$\pi_i$$ and summing over $$i$$ we obtain $$\alpha_0 = \frac{1-\alpha_1}{\gamma}$$ and therefore $$\pi_i = \frac{\left(\sum_k \pi_k e^{\gamma \beta \epsilon_k}\right)^{1/\gamma}}{\left(\sum_k \pi_k^{1-\gamma}\right)^{1/\gamma}} \exp(-\beta \epsilon_i) = \frac{\exp(-\beta \epsilon_i)}{Z^\beta}$$

# MAXENT DISTRIBUTION

• We obtained Boltzmann distribution from non-Shannonian entropy with non-arithmetic constraint
• We immediately obtain that $$\ln \pi_i = - \beta(\epsilon_i - U_\gamma^\beta(\pi)) - (1-\gamma) R_\gamma(\pi)$$ which leads to $$\ln \sum_k e^{-\beta \epsilon_k} = (1-\gamma) R_\gamma - \beta U^\beta_\gamma = \Psi_\gamma - \gamma R_\gamma$$ where $$\Psi_\gamma = R_\gamma - \beta U_\gamma^\beta$$ is the Massieu function
• The Helmholtz free energy can be expressed as $$F^\beta_\gamma(\pi) = U_\gamma^\beta(\pi)- \frac{1}{\beta} R_\gamma(\pi) = \frac{1}{(\gamma-1) \beta} \ln \sum_k e^{(\gamma-1)\beta \epsilon_k}.$$
• Thus, although we get the same distribution as for the ordinary thermodynamics, the thermodynamic relations are different!

# THERMODYNAMIC INTERPRETATION

• What is the interpretation of equilibrium thermodynamics of exponential KN averages?
• Let us denote the equilibrium quantities as

$$\mathcal{U}_\gamma^\beta = U_\gamma^\beta(\pi)$$
$$\mathcal{R}_\gamma^\beta = R_\gamma(\pi)$$
$$\mathcal{F}_\gamma^\beta = \mathcal{U}_\gamma^\beta - \frac{1}{\beta} \mathcal{R}_\gamma^\beta$$

• We can now express each quantity to find its interpretation

# THERMODYNAMIC INTERPRETATION

• We start with Rényi entropy $$\mathcal{R}_\gamma^\beta = \frac{1}{\gamma(1-\gamma)} \ln \sum_i \left(\frac{e^{-\beta \epsilon_i}}{Z^\beta}\right)^{1-\gamma} = \frac{1}{\gamma(1-\gamma)} \ln \sum_i e^{-(1-\gamma)\beta} - \frac{1}{\gamma} \ln Z^\beta$$
• Thus we get that Rényi entropy of a Boltzmann distribution is $$\mathcal{R}_\gamma^\beta = -\frac{\beta}{\gamma} \left(\mathcal{F}^{(1-\gamma) \beta} - \mathcal{F}^{\beta}\right)$$
• By defining $$\beta' = (1-\gamma)\beta \quad \Rightarrow \quad \gamma = 1-\frac{\beta'}{\beta}$$ we can express it as

$$\mathcal{R}_\gamma^\beta = \beta^2 \, \frac{\mathcal{F}^{\beta'} - \mathcal{F}^\beta}{\beta'-\beta}$$

which is the $$\beta$$ rescaling of the free energy difference.

• This can be interpreted as the maximum amount of work the system can perform by quenching the system from inverse temperature $$\beta$$ to inverse temperature $$\beta'$$.
• This has been first discovered by John Baez.

# THERMODYNAMIC INTERPRETATION

• By taking $$\gamma \rightarrow 0$$ we recover the relation between entropy and free energy $$\mathcal{S}^\beta = \beta^2 \left(\frac{\partial \mathcal{F}^\beta}{\partial \beta}\right)$$
• As shown before, the free energy can be expressed as $$\mathcal{F}_\gamma^\beta = \mathcal{F}^{(1-\gamma)\beta}$$ so it is clear that the temperature is rescaled from $$\beta$$ to $$(1-\gamma)\beta$$
• Finally the internal energy can be expressed as $$\mathcal{U}^\beta_\gamma = \frac{\beta' \mathcal{F}^{\beta'} - \beta \mathcal{F}^\beta}{\beta'-\beta} = - \frac{\Psi^{\beta'}-\Psi^\beta}{\beta'-\beta}$$
• Again, by taking $$\gamma \rightarrow 0$$ we recover

$$\mathcal{U}^\beta = - \left(\frac{\partial \Psi^\beta}{\partial \beta}\right)\,$$

# APPLICATION TO MULTIFRACTALS

• One of the main applications of Rényi entropy is multifractals
• Let us first remind go through the fundamentals of multifractal analysis
• Let us consider a physical system whose state space is parcelled into distinct regions $$k_i(s)$$ indexed by $$i\in\mathcal{I}$$
• Consider the probability of observing the system within region $$k_i(s)$$, which is denoted by $$p_i(s)$$.
• Let us assume that this probability observes a scaling property of the form
$$p_i(s) = \frac{s^{\alpha_i}}{z(s)}$$

# APPLICATION TO MULTIFRACTALS

• For small scales, i.e. $$s \rightarrow 0$$, let us assume that the frequency of the scaling exponent $$\alpha_i$$ is given by a continuous probability distribution, whose density $$\rho$$ has the form

$$\rho(\alpha,s) \mathrm{d} \alpha = c(\alpha) s^{-f(\alpha)} \mathrm{d} \alpha$$

where $$f(\alpha)$$ is called as multifractal spectrum

• Let us define the generalized dimension of Rényi entropy as $$\lim_{s \rightarrow 0} \frac{(1-\gamma) R_\gamma(s)}{\ln s} = D_\gamma$$
• We can express Rényi entropy as $$R_\gamma(s) = \frac{1}{\gamma(1-\gamma)} \ln \frac{\int s^{(1-\gamma)\alpha} c(\alpha) s^{-f(\alpha)} \mathrm{d} \alpha}{\left(\int s^{\alpha} c(\alpha) s^{-f(\alpha)} \mathrm{d} \alpha\right)^{1-\gamma}}$$
• The relation between the multifractal spectrum and generalized dimension is given by using the stepest descent approximation for $$s \rightarrow 0$$

# APPLICATION TO MULTIFRACTALS

• By using the steepest descent, we define $$\alpha_\gamma = \arg\min_\alpha\big\{(1-\gamma) \alpha - f(\alpha)\big\}$$
• We introduce the Legendre transform of multifractal spectrum $$\tau_\gamma = (1-\gamma)\alpha_\gamma - f(\alpha_\gamma)$$
• Thus, Rényi entropy can be expressed as $$R_\gamma = \frac{\tau_\gamma - (1-\gamma) \tau_1}{\gamma(1-\gamma)} \ln s + \mathcal{O}(1)$$
• Therefore, the relation to generalized dimension is $$D_\gamma = \frac{\tau_\gamma - (1-\gamma) \tau_1}{\gamma}$$

• Let us now make a connection with turbulence cascades.
• We assume that the characteristic energy $$\epsilon(s)$$ scales as $$\left\langle \epsilon(s)^{\gamma} \right\rangle \sim s^{ M_\gamma(\epsilon)}$$
• The cascade spectrum can be connected to the multifractal spectrum by using the fact that the MaxEnt distribution is Boltzmann, i.e., $$\pi_i \propto s^{\alpha_i} \propto e^{-\beta \epsilon_i}$$ which leads to $$\alpha_i \ln s = - \beta \epsilon_i$$
• Thus, the scale is connected with the temperature and the scaling exponent with the energy
• Rescaling of the temperature $$\beta \mapsto (1-\gamma) \beta$$ corresponds to the change of scale $$s \mapsto s^{1-\gamma}$$

Finally, the scaling exponents of thermodynamic potentials are

$$\beta U_\beta^{\gamma}(s) = \frac{\tau_\gamma - \tau_1}{\gamma}\ln s + \mathcal{O}(1)$$
$$\beta F_\beta^{\gamma}(s) = \frac{\tau_\gamma}{\gamma-1} \ln s + \mathcal{O}(1)$$
$$M_\gamma = \gamma U_\gamma^\beta(s) = \tau_1 - \tau_\gamma + o(1)$$

# LEGENDRE STRUCTURE

• We consider now a general non-equilibrium state and try to calculate the work done between that state and the equilibrium state
• The minimum work is given by the free energy difference $$W^{rev} = F_\gamma(p)-F_\gamma(\pi) = \Delta U_\gamma^\beta - \tfrac{1}{\beta} \Delta R_\gamma$$
• The difference between non-equilibrium and equilibrium internal energy can be expressed as
$$U^\beta_\gamma(p) - U^\beta_\gamma(\pi) = \frac{1}{\beta \gamma} \ln \frac{\sum_i p_i e^{\gamma \beta_0 \epsilon_i}}{\sum_i \pi_i e^{\gamma \beta \epsilon_i}} = \frac{1}{\beta}{\gamma} \ln \left(1 + \frac{\sum_i (p_i-\pi_i) \pi_i^{-\gamma}}{\sum_i \pi_i^{1-\gamma}} \right)$$ $$= \frac{1}{\beta \gamma} \ln \left(1+ \gamma \, \nabla R_\gamma \cdot (p-\pi)\right)$$

# LEGENDRE STRUCTURE

•  Thus, the free energy difference can be expressed as $$\frac{1}{\beta} \left(F_\gamma(p) - F_\gamma(\pi)\right) =R_\gamma(\pi) - \frac{1}{\beta} R_\gamma(p) + \frac{1}{\beta \gamma} \ln \left(1+ \gamma \nabla R_\gamma \cdot (p-\pi)\right)$$
• This can be expressed as the Rényi-Bregmann divergence which is a generalized Bregmann divergence $$\mathcal{D}_{\gamma}(p || \pi) = R_\gamma(\pi) - R_\gamma(p) - C(\nabla R_\gamma,(p-\pi))$$ where $$C(x,y)$$ is the link function, here $$C(x,y) = \frac{1}{\gamma} \ln (1+ \gamma x \cdot y)$$
• We can rewrite it as $$\mathcal{D}_{\gamma}(p || \pi) = - R_\gamma(p) - R_\gamma(p,q)$$ where $$R_\gamma(p,q)$$ is the Rényi cross entropy
• We can also express it as $$\mathcal{D}_{\gamma}(p || \pi) = R_\gamma(\pi) - R_\gamma(p) + \frac{1}{\gamma} \ln \sum p_i \Pi_i^{(\gamma)}$$ where $$\Pi_i^{(\gamma)} = \frac{\pi_i^{-\gamma}}{\sum_i \pi_i^{1-\gamma}}$$ is the ratio of escort distribution and original distribution

# NON-EQUILIBRIUM THERMODYNAMICS

• Let us now consider the case of arbitrary non-equilibrium state and its evolution in time
• We consider the standard master equation $$\dot{p}_i(t) = \sum_j \Big(w_{ij}(t) p_j(t) - w_{ji}(t) p_i(t)\Big)$$
• Since the equilibrium distribution is Boltzmann, we obtain the standard detailed balance $$\frac{w_{ij}(t)}{w_{ji}(t)} = \frac{\pi_i(t)}{\pi_j(t)} = e^{\beta (\epsilon_j(t) - \epsilon_i(t))}$$
• It is possible to show that the second law of thermodynamics hold: $$\dot{\Sigma}_\gamma = \dot{R}_\gamma - \beta \dot{Q}_\gamma^\beta \geq 0$$
• For relaxation processes ($$\epsilon_i$$ does not depend on time), we can show that the H-function $$H_\gamma(p):= \mathcal{D}_\gamma(p||\pi) = \beta (F_\gamma(p) - F_\gamma(\pi))$$ does not increase in time $$\dot{H}_\gamma \leq 0$$

# ENSEMBLE ENTROPY PRODUCTION

• Finally, we focus on trajectory thermodynamics
• Trajectory entropy production can be expressed as $$\ln \frac{\pmb{P}\big[\pmb{x}\big]}{\tilde{\pmb{P}}\big[\tilde{\pmb{x}}\big]} = \Delta s[\pmb{x}] + \beta \pmb{q}[\pmb{x}] = \pmb{\sigma}[\pmb{x}]$$
• Here $$\pmb{x}$$ is a trajectory, $$\tilde{\pmb{x}}$$ is its time reverse, $$\pmb{P}$$ is trajectory probability, $$\Delta s = \ln p(x_i,t_i) - \ln p(x_f,t_f)$$ and $$q$$ is the trajectory heat
• The ensemble average can be obtained as $$\langle \langle \pmb{\sigma} \rangle \rangle_\gamma = \frac{1}{\gamma} \ln \int \mathcal{D} \pmb{x} \pmb{P}[\pmb{x}] \exp\left(\gamma \ln \frac{\pmb{P}\big[\pmb{x}\big]}{\tilde{\pmb{P}}\big[\tilde{\pmb{x}}\big]}\right)$$ $$=\frac{1}{\gamma} \ln \int \mathcal{D} \pmb{x} (\pmb{P}[\pmb{x}])^{\gamma+1} (\tilde{\pmb{P}}[\tilde{\pmb{x}}])^{-\gamma} = \mathfrak{D}_\gamma\big(\pmb{P}[{\pmb{x}}]\,||\,\tilde{\pmb{P}}[\tilde{\pmb{x}}]\big) \geq 0$$
• Here $$\mathfrak{D}_\gamma$$ is the Rényi-Csiszár divergence

Other topics in the paper:

• Trajectory thermodynamics and fluctuation theorems
• Information geometry and thermodynamic length