AOE 5984: Introduction to Parallel Computing Applications


OpenFOAM for CFD Applications

Lecture 1:  Introduction and set-up


Professor Eric Paterson
Aerospace and Ocean Engineering, Virginia Tech
14 November 2013

Agenda

  • Lecture 1 - 12 November :  Introduction and set-up
  • Lecture 2 - 14 November :  Parallel simulation
  • Lecture 3 - 19 November :  Data analysis and visualization
  • Lecture 4 - 21 November :  Mesh generation

  • Homework 1
    • Assigned, 9 November
    • Due:  15 November

  • Homework 2
    • Assigned, 15 November
    • Due:  22 November

Homework 1

  • Available on github
# Assignment 8 - Due Friday 8 November 2013
#
# Introduction to OpenFOAM:  set-up and tutorials
#
# 1. load the OpenFOAM environment on BlueRidge using the OpenFOAM module
#
# 2. modify your .bashrc so that it is automatically loaded upon login
#
# 3. verify that the OpenFOAM installation is functional
#
# 4. identify the environment variables which are set by the 
# OpenFOAM environment. Based on these variables, identify:
# which C++ compiler was used, was OpenFOAM compiled as single- or # double-precision, which MPI library is OpenFOAM linked to? # # 5. identify important OpenFOAM linux aliases. Hint: look in
# $WM_PROJECT_DIR/etc/config/ # # 6. copy the tutorials in $FOAM_TUTORIALS to your $WORK directory # # 7. run the backward-facing-step tutorial,
# $WORK/tutorials/incompressible/simpleFoam/pitzDaily. # # 8. run a tutorial of your choice. # # # What to hand in: # # 1. A text-file "report" which documents output from parts 2, 3, 4, 5. # # 2. Plot the solution residuals from part 7 to document convergence.
# Make an image of the velocity field using ParaView. # # 3. Communicate the results from part 8 using a plot and a visualization.
# Make sure to identify the solver and case that you studied.
#

Lecture 1:  Introduction and set-up

  • Objectives
    • Provide cursory overview of OpenFOAM so that students can run tutorials
    • Set-up OpenFOAM CFD environment on Virginia Tech's BlueRidge computer
  • Outcomes
    • Students will:
      • have a basic understanding of OpenFOAM CFD, including history, capabilities, and online resources
      • understand basic file structure and use of dictionaries for controlling simulations
      • realize that solvers and tutorials are for specific physics, vs. monolithic solvers found in commercial software
      • know how to modify their .bashrc file to properly load the OpenFOAM Environment on BlueRidge

Lecture 1:  Introduction and set-up

  • Basic introduction to OpenFOAM

    • What is OpenFOAM

    • History

    • Capabilities

    • Case walkthrough
    • Online resources for help

  • Set-up on BlueRidge

    • modules

    • bashrc

    • environment variables and aliases

    • tutorials

What is OpenFOAM?

  • OpenFOAM is a free-to-use open-source numerical simulation software with extensive CFD and multi-physics capabilities

  • Free-to-use means using the software without paying for license and support, including massively parallel computers: free 10,000-CPU CFD license!

  • Software under active development, capabilities mirror those of commercial CFD

  • Substantial user base in industry, research labs, and universities

  • Possibility of extension to non-traditional, complex or coupled physics

  • Physics model implementation through equation mimicking

What is OpenFOAM?

  • Discretization: Polyhedral Finite Volume Method, second order in space and time

  • Lagrangian particle tracking (discrete element model)

  • Finite Area Method: 2-D FVM on curved surface in 3-D

  • Automatic mesh motion, support for topological changes

  • Parallelism via a domain decomposition model

Equation Mimicking

  • Flexible Handling of Arbitrary Equations Sets
  • Natural language of continuum mechanics:   partial differential equations
  • Example: turbulence kinetic energy equation

  • Objective: Represent differential equations in their natural language

  • Correspondence between implementation and the original equation is clear

History

  • Late 1980’s:   Imperial College, Prof. David Gosman’s research group

  • 1996:  H. Jasak. PhD Thesis, Imperial College, University of London (1st PhD thesis on OF)

  • 2000-2004:  Nabla, Ltd, markets commercial product, FOAM

  • 2004:   Nabla makes OpenFOAM GPL

  • 2004:  OpenCFD, Ltd (Henry Weller) and Wikki, Ltd (Hrv Jasak)

  • Jan 2006:  1st OpenFOAM Workshop

  • Nov 2007:  1st OpenFOAM Conference (Open-source CFD Conference)

  • 2009:   FOAM Documentation Project was shut-down

  • Aug 2011:  SGI purchases OpenCFD, Ltd.

  • Sep 2012:  ESI Group purchases OpenCFD from SGI

Capabilities

  • OpenFOAM is a CFD Toolbox written in C++
    • Software
    • Libraries
      • Numerical algorithms and physical models
    • Utilities:
      • meshing, pre-processing, parallel computing, post-processing, etc.
    • Solvers
      • Designed for specific classes of problems, e.g., incompressible flow, compressible flow, multiphase flow, etc.

Numerical Algorithms

  • Finite-volume discretization
    • Set in system/fvSchemes
    • Term-by-term prescription of temporal and spatial (divergence, Laplacian, gradient) operators

  • Algebraic solvers and solution parameters
    • Set in system/fvSolution
    • solvers:  PCG, PBiCG, GAMG, etc.
    • relaxationFactors
    • iteration loops

Physical modeling

  • RANS and LES turbulence models
    • variants for incompressible and compressible flow
    • Linear eddy-viscosity models and RSTM 
    • wall treatment:  wall functions, roughness

  • Thermo-physical models for liquids and gases
  • Non-Newtonian viscosity models
  • Chemical reaction library interface to Chemkin
  • Diesel spray

Utilities

  • MANY useful pre-compiled utilities
  • Categories of utilities
  • [19:55:20][egp@egpMBP:utilities]532$ ls
    mesh              parallelProcessing preProcessing     thermophysicalmiscellaneous     postProcessing     surface 
  • mesh:  generation, manipulation, conversion
  • pre- and post-processing:   self-explanatory
  • parallelProcessing:  decompose/recompose
  • surface:  geometry modification

Solvers

  • Solvers are created for each class of physics
  • Solver categories
  • [20:07:00][egp@egpMBP-2:solvers]554$ ls
    DNS              compressible     financial      lagrangian
    basic            discreteMethods  heatTransfer   multiphase
    combustion       electromagnetics incompressible stressAnalysis  
  • Important solvers for external flow
    • potentialFOAM:  ideal flow
    • Steady incompressible RANS:  simpleFOAM
    • Transient incompressible RANS:   pisoFOAM
    • Multiphase flow:  interFOAM

Case walkthrough

  • steady incompressible RANS simulation (simpleFoam) of a 2D airfoil
  • tutorial name:  $FOAM_TUTORIALS/incompressible/airFoil2D
  • inlet conditions (0/U and 0/nut)
    • U  [0 1 -1 0 0 0 0]  (25.75 3.62 0);
      nut  [0 2 -1 0 0 0 0] 0.14; 
  • turbulence model (constant/RASProperties)
RASModel        SpalartAllmaras; 
  • fluid properties (constant/transportProperties)                                                                                     
transportModel  Newtonian;
nu [ 0 2 -1 0 0 0 0 ] 1e-05; 

Geometry and Mesh

Geometry and Mesh

Case Structure

  • OpenFOAM case is a directory: each self-contained piece of heavy-weight data stored in its own file 
  • Light-weight data is presented in dictionary form: keyword-value pairs in free format. It can be changed and re-read during the run: solution steering
  • Mesh data split into components for efficient management of moving mesh cases 
  • Time directories contain solution and derived fields (one per file) 

Data I/O

Data Input and Output

  • Dictionary format: file header (IOobject) and keyword-value entry pairs
FoamFile 
{ 
     version 2.0;           format ascii;           class dictionary;           object transportProperties;}

transportModel  Newtonian; nu              nu [ 0 2 -1 0 0 0 0 ] 1e-05; 

  • Contents of dictionaries depends on their role
    • Material properties and physical model constants
    • Solution fields, initial and boundary conditions
    • Discretisation settings, solver controls I/O parameters etc. 

Controls

Basic Controls: system/controlDict

Basic controls of run start, end and write frequency

startFrom          startTime; // latestTime // firstTimestartTime          0;stopAt             endTime; // writeNow // nextWrite endTime            2500; deltaT             1; writeControl       timeStep; // runTime // clockTime // cpuTime writeInterval      50;writeFormat        ascii; // binary writePrecision     6; writeCompression   uncompressed; // compressed timeFormat         general; // fixed // scientific timePrecision      6; runTimeModifiable  yes;

Schemes

Basic Controls: system/fvSchemes

Equation discretisation controls: per-term basis 

ddtSchemes
{
    default         steadyState;
}

gradSchemes
{
    default         Gauss linear;
    grad(p)         Gauss linear;
    grad(U)         Gauss linear;
}

divSchemes
{
    default         none;
    div(phi,U)      bounded Gauss linearUpwind grad(U);
    div(phi,nuTilda) bounded Gauss linearUpwind grad(nuTilda);
    div((nuEff*dev(T(grad(U))))) Gauss linear;
}

laplacianSchemes
{
    default         none;
    laplacian(nuEff,U) Gauss linear corrected;
    laplacian((1|A(U)),p) Gauss linear corrected;
    laplacian(DnuTildaEff,nuTilda) Gauss linear corrected;
    laplacian(1,p)  Gauss linear corrected;
}

interpolationSchemes
{
    default         linear;
    interpolate(U)  linear;
}

snGradSchemes
{
    default         corrected;
}

fluxRequired
{
    default         no;
    p               ;
} 

Solution

Basic Controls: system/fvSolution  

Linear equation solver settings: per equation in top-level solver 

solvers
{
    p
    {
        solver          GAMG;
        tolerance       1e-06;
        relTol          0.1;
        smoother        GaussSeidel;
        nPreSweeps      0;
        nPostSweeps     2;
        cacheAgglomeration true;
        nCellsInCoarsestLevel 10;
        agglomerator    faceAreaPair;
        mergeLevels     1;
    }

    U
    {
        solver          smoothSolver;
        smoother        GaussSeidel;
        nSweeps         2;
        tolerance       1e-08;
        relTol          0.1;
    }

    nuTilda
    {
        solver          smoothSolver;
        smoother        GaussSeidel;
        nSweeps         2;
        tolerance       1e-08;
        relTol          0.1;
    }
}

SIMPLE
{
    nNonOrthogonalCorrectors 0;
    pRefCell        0;
    pRefValue       0;

    residualControl
    {
        p               1e-5;
        U               1e-5;
        nuTilda         1e-5;
    }
}

relaxationFactors
{
    fields
    {
        p               0.3;
    }
    equations
    {
        U               0.7;
        nuTilda         0.7;
    }
} 

Mesh

Basic Controls: Structure of Mesh Files

  • Mesh files at start of simulation located in constant/polyMesh directory
  • points, faces: basic lists of primitive entries
  • owner, neighbour: lists of face-to-cell addressing
  • Note: OpenFOAM uses strongly ordered face lists for efficiency
[01:58:35][egp@egpMBP-2:airFoil2D]586$ ls -l constant/polyMesh/
total 4480
-rw-r--r--  1 egp  staff     1202 May 17 09:19 boundary
-rw-r--r--  1 egp  staff   396698 May 17 09:19 cells
-rw-r--r--  1 egp  staff  1075986 May 17 09:19 faces
-rw-r--r--  1 egp  staff   106511 May 17 09:19 neighbour
-rw-r--r--  1 egp  staff   214431 May 17 09:19 owner
-rw-r--r--  1 egp  staff   486720 May 17 09:19 points 

    Mesh boundary file

    Basic Controls: Structure of Mesh Files

    • Boundary definition: patch types and strong ordering

    4
    (
    inlet
    {
        type patch;
        physicalType inlet;
        nFaces 134;
        startFace 21254;
    }
    
    outlet
    {
        type patch;
        physicalType outlet;
        nFaces 160;
        startFace 21388;
    }
    
    wall
    {
        type wall;
        physicalType wall;
        nFaces 78;
        startFace 21548;
    }
    
    frontAndBack
    {
        type empty;
        physicalType empty;
        nFaces 21440;
        startFace 21626;
    } 

    )

    Fields

    Field: Initial and Boundary Conditions

    • Definition of initial and boundary conditions, per-field basis
    • Fields located in time directories: 0/p, 0/U,0/nut
    • Boundary conditions defined on a per-field basis
    • Note: consistency of boundary conditions related to the physics solver 

    dimensions      [0 1 -1 0 0 0 0];
    
    internalField   uniform (25.75 3.62 0);
    
    boundaryField
    {
        inlet
        {
            type            freestream;
            freestreamValue uniform (25.75 3.62 0);
        }
    
        outlet
        {
            type            freestream;
            freestreamValue uniform (25.75 3.62 0);
        }
    
        wall
        {
            type            fixedValue;
            value           uniform (0 0 0);
        }
    
        frontAndBack
        {
            type            empty;
        }
    } 

    Online Help

    OpenFOAM Workshop


    2006:  University of Zagreb, Croatia, 50

    2007:  University of Zagreb, Croatia 90

    2008:  Polytechnique de Milano, Italy, 250

    2009:  Montreal, Canada, 125

    2010:  Chalmers, Sweden, 180

    2011:   Penn State, 230 (FIRST in USA)

    2012:   TU Darmstadt, 450

    2013:  Jeju Island (SNU), S. Korea, 225

    2014:  University of Zagreb, Croatia, ???

    Set-up on BlueRidge

    1. Modules
    2. .bashrc file
    3. Environment variables
    4. Aliases
    5. Tutorials
    6. Job submission

    Modules

    • Environment Modules package provides for the dynamic modification of a user's environment via module files
    • Modules can be loaded and unloaded dynamically
    • Widely utilized on HPC systems, including VT ARC
    • Important commands

    module availmodule listmodule load openmpi gsl OpenFOAM module swap gcc/4.7.2 intelmodule show OpenFOAM

      .bashrc

      • Bash shell start up file located in $HOME/.bashrc
      • Contains additions to $PATH, aliases, environment variables
      • For OpenFOAM, I have the following additions
      • module unload mvapich2module load openmpimodule load gslmodule load OpenFOAM. /opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0/etc/bashrc

      Environment variables

      • Environment variables set paths and flags used by software. 
      • OpenFOAM sets many variables, e.g., WM_*, FOAM*

      [17:16:16][egp@brlogin1:egp]13015$ env|grep WM
      WM_LINK_LANGUAGE=c++
      WM_ARCH=linux64
      WM_OSTYPE=POSIX
      WM_THIRD_PARTY_DIR=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/ThirdParty-2.2.0
      WM_CXXFLAGS=-m64 -fPIC
      WM_CFLAGS=-m64 -fPIC
      WM_PROJECT_VERSION=2.2.0
      WM_COMPILER_LIB_ARCH=64
      WM_PROJECT_INST_DIR=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0
      WM_CXX=g++
      WM_PROJECT_DIR=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0
      WM_PROJECT=OpenFOAM
      WM_LDFLAGS=-m64
      WM_COMPILER=Icc
      WM_MPLIB=SYSTEMOPENMPI
      WM_CC=gcc
      WM_COMPILE_OPTION=Opt
      WM_DIR=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0/wmake
      WM_PROJECT_USER_DIR=/home/egp/OpenFOAM/egp-2.2.0
      WM_OPTIONS=linux64IccDPOpt
      WM_PRECISION_OPTION=DP
      WM_ARCH_OPTION=64 


      More Environment Variables

      Find using env|grep

      [17:27:00][egp@brlogin1:egp]13017$ env|grep FOAM_
      FOAM_SOLVERS=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0/applications/solvers
      FOAM_EXT_LIBBIN=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/ThirdParty-2.2.0/platforms/linux64IccDPOpt/lib
      FOAM_APPBIN=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0/platforms/linux64IccDPOpt/bin
      FOAM_TUTORIALS=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0/tutorials
      FOAM_JOB_DIR=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/jobControl
      FOAM_SITE_APPBIN=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/site/2.2.0/platforms/linux64IccDPOpt/bin
      FOAM_APP=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0/applications
      FOAM_SITE_LIBBIN=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/site/2.2.0/platforms/linux64IccDPOpt/lib
      FOAM_SRC=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0/src
      FOAM_SIGFPE=
      FOAM_UTILITIES=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0/applications/utilities
      FOAM_USER_LIBBIN=/home/egp/OpenFOAM/egp-2.2.0/platforms/linux64IccDPOpt/lib
      FOAM_INST_DIR=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0
      OPENFOAM_BIN=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0/bin
      FOAM_MPI=openmpi-system
      FOAM_LIBBIN=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0/platforms/linux64IccDPOpt/lib
      FOAM_SETTINGS=
      FOAM_RUN=/home/egp/OpenFOAM/egp-2.2.0/run
      OPENFOAM_DIR=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0
      FOAM_USER_APPBIN=/home/egp/OpenFOAM/egp-2.2.0/platforms/linux64IccDPOpt/bin 

      Aliases

      Aliases in Linux are helpful for repetitive commands

      alias h=history
      alias ls="ls -G"
      alias work="cd $WORK"
      alias scr="cd $SCRATCH"
      alias pvload="module swap intel gcc/4.7.2 && module load qt python ParaView"
      alias ofload="module swap gcc/4.7.2 intel"
      alias pv="touch foam.foam && paraview --data=foam.foam"
      alias pvs="touch foam.foam && pvserver -ch=localhost -rc" 

      Tutorials

      • Copy tutorials from $FOAM_TUTORIALS
      • Since OpenFOAM is installed for all users, the tutorials are in a location where you do not have write permissions

      [17:27:03][egp@brlogin1:egp]13018$ echo $FOAM_TUTORIALS/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0/tutorials 
      
      [17:29:20][egp@brlogin1:egp]13019$ echo $FOAM_RUN/home/egp/OpenFOAM/egp-2.2.0/run  [17:29:53][egp@brlogin1:egp]13020$ run [17:31:23][egp@brlogin1:run]13021$ cp $FOAM_TUTORIALS tutorials

      Tutorials

      There are numerous tutorials
      They are aligned with specific solvers (flow physics)

      [17:31:23][egp@brlogin1:run]13021$ cd tutorials/
      [17:35:19][egp@brlogin1:tutorials]13024$ ls -l
      total 265
      -rwxr-xr-x  1 egp  490 Jun  2 22:00 Allclean
      -rwxr-xr-x  1 egp 2857 Jun  2 22:00 Allrun
      -rwxr-xr-x  1 egp 5991 Jun  2 22:00 Alltest
      drwxr-xr-x  5 egp   99 Jun  2 22:00 basic
      drwxr-xr-x  8 egp  159 Jun  2 22:00 combustion
      drwxr-xr-x 11 egp  289 Jun  2 22:00 compressible
      drwxr-xr-x  4 egp   61 Jun  2 22:00 discreteMethods
      drwxr-xr-x  3 egp   25 Jun  2 22:00 DNS
      drwxr-xr-x  4 egp   60 Jun  2 22:00 electromagnetics
      drwxr-xr-x  3 egp   31 Jun  2 22:00 financial
      drwxr-xr-x  8 egp  238 Jun  2 22:00 heatTransfer
      drwxr-xr-x 15 egp  423 Jun  2 22:00 incompressible
      drwxr-xr-x  9 egp  268 Jun  2 22:00 lagrangian
      drwxr-xr-x  4 egp   64 Jun  2 22:00 mesh
      drwxr-xr-x 17 egp  522 Jun  2 22:00 multiphase
      drwxr-xr-x  3 egp   26 Jun  2 22:00 resources
      drwxr-xr-x  4 egp   89 Jun  2 22:00 stressAnalysis 

      Tutorials

      • Some have Allrun and Allclean scripts!   Study them for example syntax
      • Many use tutorial run functions, which are simple bash scripts

      [17:46:04][egp@brlogin1:airFoil2D]13046$ cat Allrun 
      #!/bin/sh
      cd ${0%/*} || exit 1    # run from this directory
      
      # Source tutorial run functions
      . $WM_PROJECT_DIR/bin/tools/RunFunctions
      
      application=`getApplication`
      
      runApplication $application
      
      # ----------------------------------------------------------------- end-of-file 

      Job submission


      qsub -I -W group_list=blueridge -q normal_q -l nodes=2:ppn=16 -l walltime=5:00 -A AOE5984 
      Parallel jobs using PBS
      qsub case1.job 

      Job script for OpenFOAM

      #!/bin/bash
      
      #PBS -l walltime=24:00:00
      #PBS -l nodes=12:ppn=16
      #PBS -W group_list=blueridge
      #PBS -q normal_q
      #PBS -A AOE5984
      #PBS -M egp@vt.edu
      #PBS -m bea
      
      module purge
      module load intel openmpi OpenFOAM
      
      cd $PBS_O_WORKDIR
      pwd
      
      # Print a message before running the program 
      # echo "------------------------------------------"
      # echo "Running decomposePar!" 
      # echo "------------------------------------------"
      decomposePar -cellDist 2>&1 | tee log.decomposePar
      
      # Print a message before running the program 
      echo "------------------------------------------"
      echo "Running pisoFOAM!" 
      echo "Number of processors = " $PBS_NP
      echo "------------------------------------------"
      mpirun -bind-to-core -np $PBS_NP pisoFOAM -parallel 2>&1 | tee log.pisoFOAM
      
      # Print a message before running the program 
      echo "------------------------------------------"
      echo "Running sample utility!" 
      echo "Number of processors = " $PBS_NP
      echo "------------------------------------------"
      mpirun -bind-to-core -np $PBS_NP sample -parallel 2>&1 | tee log.sample
      
      exit; 

      incompressible/simpleFoam/motorbike tutorial
      next lecture:  parallel computing with OpenFOAM


      AOE 5984: OPENFOAM Introduction

      By egp@vt.edu

      AOE 5984: OPENFOAM Introduction

      OpenFOAM for AOE 5984, Fall 2013. These lectures will cover the following topics: 1) introduction and set-up on BlueRidge; 2) parallel computing; 3) data analysis and visualization; 4) meshing for OpenFOAM CFD simulations.

      • 7,895