Upgrading FESTIM for High-Performance Computing with FEniCSx


Remi Delaporte-Mathurin, Jorgen Dokken, James Dark, Samuele Meschini


Plasma Science and Fusion Centre, MIT, Cambridge, MA, 02139, USA

Department of Numerical Analysis and Scientific Computing, Simula Research Laboratory, Oslo, Norway

5th Fusion HPC workshop, November 21-22 2024

H transport? Why do we care?

📈5 years of open-source development

     Built on the FEniCS library

📑16+ publications

🧑‍💻23+ contributors

⭐~100 stars on GitHub

🏛️27+ institutions using the code

🧑‍💻5 user workshops

github.com/festim-dev/FESTIM

Component design (breeding blanket, PFCs, extraction systems...)

Experimental analysis (TDS, NRA, permeation...)

Safety studies (tritium inventory, contamination...)

Fuel cycle design (components, residence time...)

FESTIM [1]

[1] Delaporte-Mathurin et al, 2024. International Journal of Hydrogen Energy 10.1016/j.ijhydene.2024.03.184

User inputs

  • Material properties
  • Trap properties
  • Geometry
  • Boundary conditions
  • Initial conditions
  •            ...
T

FESTIM

Outputs

  • H concentration fields \(c(x,t)\)
  • Temperature field \(T(x,t)\)
  • surface fluxes
  • inventories
  • average concentration
  •              ...

Heat transfer model

Hydrogen transport model

3D multiphysics examples

K. Dunnell (2024), Simulating Tritium Transport for Nuclear Fusion Extractors, 10.13140/RG.2.2.13923.16164

James Dark et al 2021 Nucl. Fusion 61 116076

DEMO WCLL

Permeation Against Vacuum extractor

 James Dark et al 2024 Nucl. Fusion 64 086026

Thermo-desorption analysis

Tritium extraction system

K. Dunnell (2024), Simulating Tritium Transport for Nuclear Fusion Extractors, 10.13140/RG.2.2.13923.16164

  • Permeation Against Vacuum
     
  • Complex 3D geometry
     
  • Coupled with fluid dynamics
     
  • Tritium extraction from permeable membranes

Breeding Blanket modelling

  • DEMO WCLL
     
  • 3D geometry
     
  • Multiphysics:
    fluid dynamics, heat transfer, neutronics (OpenMC)

James Dark et al 2021 Nucl. Fusion 61 116076

So what's the problem?

The problem: parallel scaling

100k cells

3 species

Ideal

How do we fix this?

Test case:

The solution: upgrade the finite element backend

  • FESTIM is rebuilt on FEniCSx [1]
     
  • New features and new physical models
     
  • Better HPC performances

[1] DOLFINx: The next generation FEniCS problem solving environment (2023) 10.5281/zenodo.10447666

[2] Richardson et al, Scalable computation of thermomechanical turbomachinery problems (2019) 10.1016/j.finel.2018.11.002

[3] Jones, T. D., Sime, N., & van Keken,P. E. (2021). Geochemistry, Geophysics, Geosystems,22, e2020GC009396

FEniCS problem on

45 million cells mesh [2]

FEniCSx geophysics problem [3]

Theory background

\frac{\partial c_\mathrm{m}}{\partial t} = \nabla \cdot (D \nabla c_\mathrm{m}) - \frac{\partial c_{\mathrm{t}, i}}{\partial t}
\frac{\partial c_{\mathrm{t}, i}}{\partial t} = k \ c_\mathrm{m} \ (n - c_{\mathrm{t}, i}) - p \ c_{\mathrm{t}, i}

Mobile H concentration

Trapped H concentration

H transport model

\frac{\partial c_\mathrm{m}}{\partial t} = \nabla \cdot (D \nabla c_\mathrm{m}) - \frac{\partial c_{\mathrm{t}, i}}{\partial t}
\frac{\partial c_{\mathrm{t}, i}}{\partial t} = k \ c_\mathrm{m} \ (n - c_{\mathrm{t}, i}) - p \ c_{\mathrm{t}, i}

Time derivatives

H transport model

\frac{\partial c_\mathrm{m}}{\partial t} = \nabla \cdot (D \nabla c_\mathrm{m}) - \frac{\partial c_{\mathrm{t}, i}}{\partial t}
\frac{\partial c_{\mathrm{t}, i}}{\partial t} = k \ c_\mathrm{m} \ (n - c_{\mathrm{t}, i}) - p \ c_{\mathrm{t}, i}

Diffusive term

H transport model

\frac{\partial c_\mathrm{m}}{\partial t} = \nabla \cdot (D \nabla c_\mathrm{m}) - \frac{\partial c_{\mathrm{t}, i}}{\partial t}
\frac{\partial c_{\mathrm{t}, i}}{\partial t} = k \ c_\mathrm{m} \ (n - c_{\mathrm{t}, i}) - p \ c_{\mathrm{t}, i}

Trapping term

H transport model

\frac{\partial c_\mathrm{m}}{\partial t} = \nabla \cdot (D \nabla c_\mathrm{m}) - \frac{\partial c_{\mathrm{t}, i}}{\partial t}
\frac{\partial c_{\mathrm{t}, i}}{\partial t} = k \ c_\mathrm{m} \ (n - c_{\mathrm{t}, i}) - p \ c_{\mathrm{t}, i}

Detrapping term

H transport model

H transport model

\frac{\partial c_\mathrm{m}}{\partial t} = \nabla \cdot (D \nabla c_\mathrm{m}) - \frac{\partial c_{\mathrm{t}, i}}{\partial t}
\frac{\partial c_{\mathrm{t}, i}}{\partial t} = k \ c_\mathrm{m} \ (n - c_{\mathrm{t}, i}) - p \ c_{\mathrm{t}, i}
(\frac{c_\mathrm{m}}{K_S})^- = (\frac{c_\mathrm{m}}{K_S})^+ \quad \mathrm{at \ interfaces}

This is tricky!

How are discontinuities handled in FESTIM1

Option 1: Change of variable method

\theta = \frac{c_\mathrm{m}}{K_S}
\theta^- = \theta^+ \quad \mathrm{at \ interfaces}
\frac{\partial c_\mathrm{m}}{\partial t} = \nabla \cdot (D \nabla c_\mathrm{m}) - \frac{\partial c_{\mathrm{t}, i}}{\partial t}
\frac{\partial c_{\mathrm{t}, i}}{\partial t} = k \ c_\mathrm{m} \ (n - c_{\mathrm{t}, i}) - p \ c_{\mathrm{t}, i}
\frac{\partial K_S \ \theta}{\partial t} = \nabla \cdot \left(D \nabla (K_S \ \theta) \right) - \frac{\partial c_{\mathrm{t}, i}}{\partial t}
\frac{\partial c_{\mathrm{t}, i}}{\partial t} = k \ K_S \ \theta \ (n - c_{\mathrm{t}, i}) - p \ c_{\mathrm{t}, i}
c_\mathrm{m} = K_S \ \theta

Solve for \( \theta \)

Reimplementing this method in FESTIM 2 shows better performances

❌If traps are not defined everywhere, then you have wasted dofs
 

❌Highly non-linear with dissimilar materials

3-30x

faster

Option 2: discontinuous elements everywhere

CG elements

Option 2: discontinuous elements everywhere

CG elements

DG elements

Option 2: discontinuous elements everywhere

CG elements

Duplicated degrees of freedom

❌ Not efficient!

DG elements

Option 3: Mixed domain method

Domain

Submesh 1

Submesh 2

\frac{\partial c_\mathrm{m}}{\partial t} = \nabla \cdot (D \nabla c_\mathrm{m})
\frac{\partial c_\mathrm{m}}{\partial t} = \nabla \cdot (D \nabla c_\mathrm{m})
\textcolor{#e69138}{\frac{c_\mathrm{m}}{K_S}} = \textcolor{#02506e}{\frac{c_\mathrm{m}}{K_S}}

Discontinuous Galerkin methods [1]

[1] Arnold et al, J. Numer. Anal. 2002, 10.1137/S0036142901384162

Option 3: Mixed domain method

✅Mixed domain can streamline multiphysics coupling

 

❌Some methods do not work for dissimilar materials

The mixed domain method has similar performances

3-30x

faster

  • FESTIM2 is up to 30x faster than FESTIM1
     
  • Will improve scaling by hunting inefficiencies in FESTIM and reimplementing some functions in C++ in dolfinx
     
  • FESTIM2 includes more physics (multi-species, chemical reactions...)
     
  • This paves the way for:
    • Streamlined multiphysics (fluid-solid problems, external solvers...)
    • New interface physics (trapping, interface diffusion...)

Conclusion

👉Repository for this presentation's work: github.com/RemDelaporteMathurin/festim-benchmark

FESTIM repository: github.com/FESTIM-dev/FESTIM

Change of variable

DG everywhere

Nitsche

Penalty

Lagrange multiplier

Mixed domain

➕simple implementation

➕parameter free

➖only similar materials

➖wasted DoFs

➖only similar materials

➕simple implementation

➕can do dissimilar materials
➖penalty term \(\infty\)

➕penalty term is bounded

➖only similar materials

➕Good for multiphysics

➕No wasted DoFs

Interface discontinuities

No free lunch

spack + proper HPC + a few optimisations cluster shows even better performances

Very large problem

  • either 1 MB or an array of MBs with 2M cells