AOE 5984: Introduction to Parallel Computing Applications


OpenFOAM for CFD Applications

Lecture 1:  Introduction and set-up


Professor Eric Paterson
Aerospace and Ocean Engineering, Virginia Tech
14 November 2013

Agenda

  • Lecture 1 - 12 November :  Introduction and set-up
  • Lecture 2 - 14 November :  Parallel simulation
  • Lecture 3 - 19 November :  Data analysis and visualization
  • Lecture 4 - 21 November :  Mesh generation

  • Homework 1
    • Assigned, 9 November
    • Due:  15 November

  • Homework 2
    • Assigned, 15 November
    • Due:  22 November

Homework 1

  • Available on github
# Assignment 8 - Due Friday 8 November 2013
#
# Introduction to OpenFOAM:  set-up and tutorials
#
# 1. load the OpenFOAM environment on BlueRidge using the OpenFOAM module
#
# 2. modify your .bashrc so that it is automatically loaded upon login
#
# 3. verify that the OpenFOAM installation is functional
#
# 4. identify the environment variables which are set by the 
#    OpenFOAM environment.  Based on these variables, identify:  
#    which C++ compiler was used, was OpenFOAM compiled as single- or
#    double-precision, which MPI library is OpenFOAM linked to?
#
# 5. identify important OpenFOAM linux aliases.  Hint:  look in 
#    $WM_PROJECT_DIR/etc/config/
#
# 6. copy the tutorials in $FOAM_TUTORIALS to your $WORK directory
#
# 7. run the backward-facing-step tutorial, 
#    $WORK/tutorials/incompressible/simpleFoam/pitzDaily. 
#
# 8. run a tutorial of your choice.  
#
#
#  What to hand in:
#
# 1.  A text-file "report" which documents output from parts 2, 3, 4, 5.
#
# 2.  Plot the solution residuals from part 7 to document convergence.   
#     Make an image of the velocity field using ParaView.
#
# 3.  Communicate the results from part 8 using a plot and a visualization.  
#     Make sure to identify the solver and case that you studied.  
#

Lecture 1:  Introduction and set-up

  • Objectives
    • Provide cursory overview of OpenFOAM so that students can run tutorials
    • Set-up OpenFOAM CFD environment on Virginia Tech's BlueRidge computer
  • Outcomes
    • Students will:
      • have a basic understanding of OpenFOAM CFD, including history, capabilities, and online resources
      • understand basic file structure and use of dictionaries for controlling simulations
      • realize that solvers and tutorials are for specific physics, vs. monolithic solvers found in commercial software
      • know how to modify their .bashrc file to properly load the OpenFOAM Environment on BlueRidge

Lecture 1:  Introduction and set-up

  • Basic introduction to OpenFOAM

    • What is OpenFOAM

    • History

    • Capabilities

    • Case walkthrough

    • Online resources for help

  • Set-up on BlueRidge

    • modules

    • bashrc

    • environment variables and aliases

    • tutorials

What is OpenFOAM?

  • OpenFOAM is a free-to-use open-source numerical simulation software with extensive CFD and multi-physics capabilities

  • Free-to-use means using the software without paying for license and support, including massively parallel computers: free 10,000-CPU CFD license!

  • Software under active development, capabilities mirror those of commercial CFD

  • Substantial user base in industry, research labs, and universities

  • Possibility of extension to non-traditional, complex or coupled physics

  • Physics model implementation through equation mimicking

What is OpenFOAM?

  • Discretization: Polyhedral Finite Volume Method, second order in space and time

  • Lagrangian particle tracking (discrete element model)

  • Finite Area Method: 2-D FVM on curved surface in 3-D

  • Automatic mesh motion, support for topological changes

  • Parallelism via a domain decomposition model

Equation Mimicking

  • Flexible Handling of Arbitrary Equations Sets
  • Natural language of continuum mechanics:   partial differential equations

  • Example: turbulence kinetic energy equation

  • Objective: Represent differential equations in their natural language

  • Correspondence between implementation and the original equation is clear

History

  • Late 1980’s:   Imperial College, Prof. David Gosman’s research group

  • 1996:  H. Jasak. PhD Thesis, Imperial College, University of London (1st PhD thesis on OF)

  • 2000-2004:  Nabla, Ltd, markets commercial product, FOAM

  • 2004:   Nabla makes OpenFOAM GPL

  • 2004:  OpenCFD, Ltd (Henry Weller) and Wikki, Ltd (Hrv Jasak)

  • Jan 2006:  1st OpenFOAM Workshop

  • Nov 2007:  1st OpenFOAM Conference (Open-source CFD Conference)

  • 2009:   FOAM Documentation Project was shut-down

  • Aug 2011:  SGI purchases OpenCFD, Ltd.

  • Sep 2012:  ESI Group purchases OpenCFD from SGI

Capabilities

  • OpenFOAM is a CFD Toolbox written in C++
    • Software
    • Libraries
      • Numerical algorithms and physical models
    • Utilities:
      • meshing, pre-processing, parallel computing, post-processing, etc.
    • Solvers
      • Designed for specific classes of problems, e.g., incompressible flow, compressible flow, multiphase flow, etc.

Numerical Algorithms

  • Finite-volume discretization
    • Set in system/fvSchemes
    • Term-by-term prescription of temporal and spatial (divergence, Laplacian, gradient) operators

  • Algebraic solvers and solution parameters
    • Set in system/fvSolution
    • solvers:  PCG, PBiCG, GAMG, etc.
    • relaxationFactors
    • iteration loops

Physical modeling

  • RANS and LES turbulence models
    • variants for incompressible and compressible flow
    • Linear eddy-viscosity models and RSTM 
    • wall treatment:  wall functions, roughness

  • Thermo-physical models for liquids and gases
  • Non-Newtonian viscosity models
  • Chemical reaction library interface to Chemkin
  • Diesel spray

Utilities

  • MANY useful pre-compiled utilities
  • Categories of utilities
  • [19:55:20][egp@egpMBP:utilities]532$ ls
mesh              parallelProcessing preProcessing     thermophysicalmiscellaneous     postProcessing     surface
  • mesh:  generation, manipulation, conversion
  • pre- and post-processing:   self-explanatory
  • parallelProcessing:  decompose/recompose
  • surface:  geometry modification

Solvers

  • Solvers are created for each class of physics
  • Solver categories
  • [20:07:00][egp@egpMBP-2:solvers]554$ ls
DNS              compressible     financial      lagrangian
basic            discreteMethods  heatTransfer   multiphase
combustion       electromagnetics incompressible stressAnalysis
  • Important solvers for external flow
    • potentialFOAM:  ideal flow
    • Steady incompressible RANS:  simpleFOAM
    • Transient incompressible RANS:   pisoFOAM
    • Multiphase flow:  interFOAM

Case walkthrough

  • steady incompressible RANS simulation (simpleFoam) of a 2D airfoil
  • tutorial name:  $FOAM_TUTORIALS/incompressible/airFoil2D
  • inlet conditions (0/U and 0/nut)
    • U  [0 1 -1 0 0 0 0]  (25.75 3.62 0);
nut  [0 2 -1 0 0 0 0] 0.14;
  • turbulence model (constant/RASProperties)
RASModel        SpalartAllmaras;
  • fluid properties (constant/transportProperties)                                                                                     
transportModel  Newtonian;
nu [ 0 2 -1 0 0 0 0 ] 1e-05;

Geometry and Mesh

Geometry and Mesh

Case Structure

  • OpenFOAM case is a directory: each self-contained piece of heavy-weight data stored in its own file 
  • Light-weight data is presented in dictionary form: keyword-value pairs in free format. It can be changed and re-read during the run: solution steering
  • Mesh data split into components for efficient management of moving mesh cases 
  • Time directories contain solution and derived fields (one per file) 

Data I/O

Data Input and Output

  • Dictionary format: file header (IOobject) and keyword-value entry pairs
FoamFile 
{ 
     version 2.0;           format ascii;           class dictionary;           object transportProperties;}

transportModel  Newtonian;
nu              nu [ 0 2 -1 0 0 0 0 ] 1e-05; 

  • Contents of dictionaries depends on their role
    • Material properties and physical model constants
    • Solution fields, initial and boundary conditions
    • Discretisation settings, solver controls I/O parameters etc. 

Controls

Basic Controls: system/controlDict

• Basic controls of run start, end and write frequency 

startFrom          startTime; // latestTime // firstTimestartTime          0;stopAt             endTime; // writeNow // nextWrite endTime            2500; deltaT             1; writeControl       timeStep; // runTime // clockTime // cpuTime writeInterval      50;writeFormat        ascii; // binary writePrecision     6; writeCompression   uncompressed; // compressed timeFormat         general; // fixed // scientific timePrecision      6; runTimeModifiable  yes;

Schemes

Basic Controls: system/fvSchemes

• Equation discretisation controls: per-term basis  

ddtSchemes
{
    default         steadyState;
}

gradSchemes
{
    default         Gauss linear;
    grad(p)         Gauss linear;
    grad(U)         Gauss linear;
}

divSchemes
{
    default         none;
    div(phi,U)      bounded Gauss linearUpwind grad(U);
    div(phi,nuTilda) bounded Gauss linearUpwind grad(nuTilda);
    div((nuEff*dev(T(grad(U))))) Gauss linear;
}

laplacianSchemes
{
    default         none;
    laplacian(nuEff,U) Gauss linear corrected;
    laplacian((1|A(U)),p) Gauss linear corrected;
    laplacian(DnuTildaEff,nuTilda) Gauss linear corrected;
    laplacian(1,p)  Gauss linear corrected;
}

interpolationSchemes
{
    default         linear;
    interpolate(U)  linear;
}

snGradSchemes
{
    default         corrected;
}

fluxRequired
{
    default         no;
    p               ;
}

Solution

Basic Controls: system/fvSolution  

• Linear equation solver settings: per equation in top-level solver 

solvers
{
    p
    {
        solver          GAMG;
        tolerance       1e-06;
        relTol          0.1;
        smoother        GaussSeidel;
        nPreSweeps      0;
        nPostSweeps     2;
        cacheAgglomeration true;
        nCellsInCoarsestLevel 10;
        agglomerator    faceAreaPair;
        mergeLevels     1;
    }

    U
    {
        solver          smoothSolver;
        smoother        GaussSeidel;
        nSweeps         2;
        tolerance       1e-08;
        relTol          0.1;
    }

    nuTilda
    {
        solver          smoothSolver;
        smoother        GaussSeidel;
        nSweeps         2;
        tolerance       1e-08;
        relTol          0.1;
    }
}

SIMPLE
{
    nNonOrthogonalCorrectors 0;
    pRefCell        0;
    pRefValue       0;

    residualControl
    {
        p               1e-5;
        U               1e-5;
        nuTilda         1e-5;
    }
}

relaxationFactors
{
    fields
    {
        p               0.3;
    }
    equations
    {
        U               0.7;
        nuTilda         0.7;
    }
}

Mesh

Basic Controls: Structure of Mesh Files

  • Mesh files at start of simulation located in constant/polyMesh directory
  • points, faces: basic lists of primitive entries
  • owner, neighbour: lists of face-to-cell addressing
  • Note: OpenFOAM uses strongly ordered face lists for efficiency
[01:58:35][egp@egpMBP-2:airFoil2D]586$ ls -l constant/polyMesh/
total 4480
-rw-r--r--  1 egp  staff     1202 May 17 09:19 boundary
-rw-r--r--  1 egp  staff   396698 May 17 09:19 cells
-rw-r--r--  1 egp  staff  1075986 May 17 09:19 faces
-rw-r--r--  1 egp  staff   106511 May 17 09:19 neighbour
-rw-r--r--  1 egp  staff   214431 May 17 09:19 owner
-rw-r--r--  1 egp  staff   486720 May 17 09:19 points

    Mesh boundary file

    Basic Controls: Structure of Mesh Files

    • Boundary definition: patch types and strong ordering

    4
(
inlet
{
    type patch;
    physicalType inlet;
    nFaces 134;
    startFace 21254;
}

outlet
{
    type patch;
    physicalType outlet;
    nFaces 160;
    startFace 21388;
}

wall
{
    type wall;
    physicalType wall;
    nFaces 78;
    startFace 21548;
}

frontAndBack
{
    type empty;
    physicalType empty;
    nFaces 21440;
    startFace 21626;
}

    )

    Fields

    Field: Initial and Boundary Conditions

    • Definition of initial and boundary conditions, per-field basis
    • Fields located in time directories: 0/p, 0/U,0/nut
    • Boundary conditions defined on a per-field basis
    • Note: consistency of boundary conditions related to the physics solver  

    dimensions      [0 1 -1 0 0 0 0];

internalField   uniform (25.75 3.62 0);

boundaryField
{
    inlet
    {
        type            freestream;
        freestreamValue uniform (25.75 3.62 0);
    }

    outlet
    {
        type            freestream;
        freestreamValue uniform (25.75 3.62 0);
    }

    wall
    {
        type            fixedValue;
        value           uniform (0 0 0);
    }

    frontAndBack
    {
        type            empty;
    }
}

    Online Help

    OpenFOAM Workshop


    2006:  University of Zagreb, Croatia, 50

    2007:  University of Zagreb, Croatia 90

    2008:  Polytechnique de Milano, Italy, 250

    2009:  Montreal, Canada, 125

    2010:  Chalmers, Sweden, 180

    2011:   Penn State, 230 (FIRST in USA)

    2012:   TU Darmstadt, 450

    2013:  Jeju Island (SNU), S. Korea, 225

    2014:  University of Zagreb, Croatia, ???

    Set-up on BlueRidge

    1. Modules
    2. .bashrc file
    3. Environment variables
    4. Aliases
    5. Tutorials
    6. Job submission

    Modules

    • Environment Modules package provides for the dynamic modification of a user's environment via module files
    • Modules can be loaded and unloaded dynamically
    • Widely utilized on HPC systems, including VT ARC
    • Important commands

    module availmodule listmodule load openmpi gsl OpenFOAM module swap gcc/4.7.2 intelmodule show OpenFOAM

      .bashrc

      • Bash shell start up file located in $HOME/.bashrc
      • Contains additions to $PATH, aliases, environment variables
      • For OpenFOAM, I have the following additions
      • module unload mvapich2module load openmpimodule load gslmodule load OpenFOAM. /opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0/etc/bashrc

      Environment variables

      • Environment variables set paths and flags used by software. 
      • OpenFOAM sets many variables, e.g., WM_*, FOAM*

      [17:16:16][egp@brlogin1:egp]13015$ env|grep WM
WM_LINK_LANGUAGE=c++
WM_ARCH=linux64
WM_OSTYPE=POSIX
WM_THIRD_PARTY_DIR=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/ThirdParty-2.2.0
WM_CXXFLAGS=-m64 -fPIC
WM_CFLAGS=-m64 -fPIC
WM_PROJECT_VERSION=2.2.0
WM_COMPILER_LIB_ARCH=64
WM_PROJECT_INST_DIR=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0
WM_CXX=g++
WM_PROJECT_DIR=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0
WM_PROJECT=OpenFOAM
WM_LDFLAGS=-m64
WM_COMPILER=Icc
WM_MPLIB=SYSTEMOPENMPI
WM_CC=gcc
WM_COMPILE_OPTION=Opt
WM_DIR=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0/wmake
WM_PROJECT_USER_DIR=/home/egp/OpenFOAM/egp-2.2.0
WM_OPTIONS=linux64IccDPOpt
WM_PRECISION_OPTION=DP
WM_ARCH_OPTION=64


      More Environment Variables

      Find using env|grep

      [17:27:00][egp@brlogin1:egp]13017$ env|grep FOAM_
FOAM_SOLVERS=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0/applications/solvers
FOAM_EXT_LIBBIN=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/ThirdParty-2.2.0/platforms/linux64IccDPOpt/lib
FOAM_APPBIN=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0/platforms/linux64IccDPOpt/bin
FOAM_TUTORIALS=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0/tutorials
FOAM_JOB_DIR=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/jobControl
FOAM_SITE_APPBIN=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/site/2.2.0/platforms/linux64IccDPOpt/bin
FOAM_APP=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0/applications
FOAM_SITE_LIBBIN=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/site/2.2.0/platforms/linux64IccDPOpt/lib
FOAM_SRC=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0/src
FOAM_SIGFPE=
FOAM_UTILITIES=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0/applications/utilities
FOAM_USER_LIBBIN=/home/egp/OpenFOAM/egp-2.2.0/platforms/linux64IccDPOpt/lib
FOAM_INST_DIR=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0
OPENFOAM_BIN=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0/bin
FOAM_MPI=openmpi-system
FOAM_LIBBIN=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0/platforms/linux64IccDPOpt/lib
FOAM_SETTINGS=
FOAM_RUN=/home/egp/OpenFOAM/egp-2.2.0/run
OPENFOAM_DIR=/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0
FOAM_USER_APPBIN=/home/egp/OpenFOAM/egp-2.2.0/platforms/linux64IccDPOpt/bin

      Aliases

      Aliases in Linux are helpful for repetitive commands

      alias h=history
alias ls="ls -G"
alias work="cd $WORK"
alias scr="cd $SCRATCH"
alias pvload="module swap intel gcc/4.7.2 && module load qt python ParaView"
alias ofload="module swap gcc/4.7.2 intel"
alias pv="touch foam.foam && paraview --data=foam.foam"
alias pvs="touch foam.foam && pvserver -ch=localhost -rc"

      Tutorials

      • Copy tutorials from $FOAM_TUTORIALS
      • Since OpenFOAM is installed for all users, the tutorials are in a location where you do not have write permissions

      [17:27:03][egp@brlogin1:egp]13018$ echo $FOAM_TUTORIALS/opt/apps/intel13_1/openmpi1_6/OpenFOAM/2.2.0/OpenFOAM-2.2.0/tutorials 

      [17:29:20][egp@brlogin1:egp]13019$ echo $FOAM_RUN/home/egp/OpenFOAM/egp-2.2.0/run 
[17:29:53][egp@brlogin1:egp]13020$ run
[17:31:23][egp@brlogin1:run]13021$ cp $FOAM_TUTORIALS tutorials

      Tutorials

      There are numerous tutorials
      They are aligned with specific solvers (flow physics)

      [17:31:23][egp@brlogin1:run]13021$ cd tutorials/
[17:35:19][egp@brlogin1:tutorials]13024$ ls -l
total 265
-rwxr-xr-x  1 egp  490 Jun  2 22:00 Allclean
-rwxr-xr-x  1 egp 2857 Jun  2 22:00 Allrun
-rwxr-xr-x  1 egp 5991 Jun  2 22:00 Alltest
drwxr-xr-x  5 egp   99 Jun  2 22:00 basic
drwxr-xr-x  8 egp  159 Jun  2 22:00 combustion
drwxr-xr-x 11 egp  289 Jun  2 22:00 compressible
drwxr-xr-x  4 egp   61 Jun  2 22:00 discreteMethods
drwxr-xr-x  3 egp   25 Jun  2 22:00 DNS
drwxr-xr-x  4 egp   60 Jun  2 22:00 electromagnetics
drwxr-xr-x  3 egp   31 Jun  2 22:00 financial
drwxr-xr-x  8 egp  238 Jun  2 22:00 heatTransfer
drwxr-xr-x 15 egp  423 Jun  2 22:00 incompressible
drwxr-xr-x  9 egp  268 Jun  2 22:00 lagrangian
drwxr-xr-x  4 egp   64 Jun  2 22:00 mesh
drwxr-xr-x 17 egp  522 Jun  2 22:00 multiphase
drwxr-xr-x  3 egp   26 Jun  2 22:00 resources
drwxr-xr-x  4 egp   89 Jun  2 22:00 stressAnalysis

      Tutorials

      • Some have Allrun and Allclean scripts!   Study them for example syntax
      • Many use tutorial run functions, which are simple bash scripts

      [17:46:04][egp@brlogin1:airFoil2D]13046$ cat Allrun 
#!/bin/sh
cd ${0%/*} || exit 1    # run from this directory

# Source tutorial run functions
. $WM_PROJECT_DIR/bin/tools/RunFunctions

application=`getApplication`

runApplication $application

# ----------------------------------------------------------------- end-of-file

      Job submission


      qsub -I -W group_list=blueridge -q normal_q -l nodes=2:ppn=16 -l walltime=5:00 -A AOE5984
      Parallel jobs using PBS
      qsub case1.job

      Job script for OpenFOAM

      #!/bin/bash

#PBS -l walltime=24:00:00
#PBS -l nodes=12:ppn=16
#PBS -W group_list=blueridge
#PBS -q normal_q
#PBS -A AOE5984
#PBS -M egp@vt.edu
#PBS -m bea

module purge
module load intel openmpi OpenFOAM

cd $PBS_O_WORKDIR
pwd

# Print a message before running the program 
# echo "------------------------------------------"
# echo "Running decomposePar!" 
# echo "------------------------------------------"
decomposePar -cellDist 2>&1 | tee log.decomposePar

# Print a message before running the program 
echo "------------------------------------------"
echo "Running pisoFOAM!" 
echo "Number of processors = " $PBS_NP
echo "------------------------------------------"
mpirun -bind-to-core -np $PBS_NP pisoFOAM -parallel 2>&1 | tee log.pisoFOAM

# Print a message before running the program 
echo "------------------------------------------"
echo "Running sample utility!" 
echo "Number of processors = " $PBS_NP
echo "------------------------------------------"
mpirun -bind-to-core -np $PBS_NP sample -parallel 2>&1 | tee log.sample

exit;

      incompressible/simpleFoam/motorbike tutorial
      next lecture:  parallel computing with OpenFOAM


      AOE 5984: OPENFOAM Introduction

      By egp@vt.edu

      Made with Slides.com

      AOE 5984: OPENFOAM Introduction

      OpenFOAM for AOE 5984, Fall 2013. These lectures will cover the following topics: 1) introduction and set-up on BlueRidge; 2) parallel computing; 3) data analysis and visualization; 4) meshing for OpenFOAM CFD simulations.

      • 8,002

      More from egp@vt.edu