Finding the ground state of spin Hamiltonians with reinforcement learning
Amin Mohamadi
Feb 5 2025
Spin Glass Problems
A graphical model, used for modelling a system as a set of binary spins, connected through interactions and exhibiting a certain bias for each spin.
- TSP
- Max-Cut
- Protein Folding
The Hamiltonian
- The Hamiltonian defines the energy of the system in a given state.
- The goal is to find global minima of the Hamiltonian, called Ground State. (NP-Hard due to non-convexity)
- Simulated Annealing (SA) is the go-to approach for finding ground state.
\mathcal{H} = - \sum_{i < j} J_{ij}\,\sigma_i\,\sigma_j \;-\; \sum_{i} h_i\,\sigma_i, \qquad \sigma_i = \pm 1, J_{ij} \in \mathbb{R}
Simulated Annealing
-
Inspired by metallurgy, where controlled heating and cooling help find a stable low-energy state.
-
An Optimization approach that mimics this process to solve NP-hard problems like spin glasses.
- At high temperatures, the system explores a wide range of states, avoiding local minima.
- As the temperature decreases, the system settles into an optimal (or near-optimal) solution.
Simulated Annealing
- Let \( s = s_0 \)
- For \( k = 0 \) through \( k_{\max} \) (exclusive):
- \( T \leftarrow \text{temperature} \left( 1 - \frac{k+1}{k_{\max}} \right) \)
- Pick a random neighbor, \( s_{\text{new}} \leftarrow \text{neighbour}(s) \)
- If \( P(E(s), E(s_{\text{new}}), T) \geq \text{random}(0,1) \):
- \( s \leftarrow s_{\text{new}} \)
- Output: the final state \( s \)
The Problem
- The success of Simulated Annealing is completely dependent on the temperature schedule.
- Temperature schedules are mostly based on heuristics:
- No adaptation according to specific instances.
- Requires manual tuning for different Hamiltonians.
- Scaling to higher problem complexities is cumbersome.
The Solution: RL
- Reinforcement Learning (RL) is a proven method for optimizing a procedure with a well-defined success measure.
- Deep reinforcement learning is now used to train agents with superhuman performance that play video games, board games, conduct protein folding and chip design.
- Main Idea: Use RL to control the temperature schedule of simulated annealing.
RL for Simulated Annealing
-
An RL agent receives an input state \(s_t\) at time \(t\), and decides to take action \(a_t\) based on a learnt policy function \(\pi_\theta(a_t \vert s_t)\) parameterized by \(\theta\) to obtain a reward \(R_t\).
- In Deep RL, policy function is represented by a neural network.
- In controlling Simulated Annealing, the agent observes the spins \(\sigma_i\) at time \(t\) and decides to change the inverse temperature by \(\Delta \beta\) to obtain a reward proportionate to negative of the energy of the system.
- An RL agent receives an input state \(s_t\) at time \(t\), and decides to take action \(a_t\) based on a learnt policy function \(\pi_\theta(a_t \vert s_t)\) parameterized by \(\theta\) to obtain a reward \(R_t\).
- In Deep RL, policy function is represented by a neural network.
RL for SA
-
State:
- \(N_{reps}\) replicas of randomly initialized spin glasses with \(L\) binary spins.
-
Action:
- A change in inverse temperature, sampled from \(\mathcal{N}(\mu, \sigma^2)\), where \(\mu, \sigma^2\) are outputs of agent.
-
Reward:
- Negative of minimum energy achieved across replicas.
RL Optimization Process
-
The agent (neural network) is optimized using a well-known RL algorithm: Proximal Policy Optimization (PPO).
- \(L^{CLIP}\) represents the policy loss: how likely is the network to take actions that maximize the reward?
- \(L^{VF}\) represents the value estimation: how well can the agent estimate the future rewards?
- \(S[\pi_\theta]\) represents the entropy of the outcomes: how diverse are the actions that agent takes?
L^{\text{PPO}} (\theta) = \hat{\mathbb{E}}_t \left[ L^{\text{CLIP}} (\theta) - c_1 L^{\text{VF}} (\theta) + c_2 S[\pi_{\theta}](s_t) \right]
Experimental Results:
Weak-Strong Clusters
-
Weak-Strong Clusters (WSC) are a specific class of spin-glasses known for having a severe local minima next to their global minima.
- In evaluation, all WSC
instances were initialized
in a local minima.
- The initial temperature
can be cool or hot.
Experimental Results:
Spin Glass
-
The agent has also been tested on general spin-glass problems without specific structures.
- Apart from specific instances,
the scaling capability of the
proposed algorithm was also
investigated.
- Surprisingly, RL scales better
than classical SA schedules.
Analyzing the learnt policy
- Analysis of the learnt policy and taken actions strongly suggests that the RL agent relies on observed states to take actions.
- This indicates that the dynamical decision making has indeed achieved better results.
Conclusion
- Reinforcement Learning (RL) agents can be learnt to enhance classical Simulated Annealing (SA) strategies in solving Spin Glass problems.
- RL dynamically learns temperature schedules rather than relying on predefined heuristics that
- Achieves higher success rates in finding the ground state.
- Scales better with increasing system size.
- Generalizes well to different Hamiltonians.
Thanks!
deck
By Amin Mohamadi
