Cluster practices

OpenFoam® has been designed to run in a distributed memory system. This means that each core or node has a memory of its own and does not require locks like shared memory. Distributed memory uses a collection of independent core memory pairs that synchronize using a network.

OpenMPI is an implementation of the Message Passing Interface (MPI) used to permit at the system to communicate via a distributed memory.

Environment setup

Check what modules are installed:

module avail

A module refers to a collection of software packages or libraries that is available on a system. The module load command is used to load specific modules into the user’s environment.

The default environment is typically a minimal set of software packages and libraries needed to run basic commands. However, there may be many additional software packages and libraries available on the cluster, and these can be loaded into the user’s environment as needed using the “module load” command.

Once a module is loaded, its associated executables, libraries, and environment variables become available to the user’s session.

#!/bin/bash
#SBATCH --job-name=OF-Job
#SBATCH --ntasks=36
#SBATCH --output=%x_%j.out
#SBATCH --constraint=c5.18xlarge

#------------------------------------------------------------------------------

module load openmpi                                         # MPI module upload
source /fsx/programs/OpenFOAM®/OpenFOAM®-v2112/etc/bashrc   # Source OpenFOAM® binaries
source ${WM_PROJECT_DIR:?}/bin/tools/RunFunctions           # Source OpenFOAM® run functions

#------------------------------------------------------------------------------

rm -r log; mkdir log
restore0Dir
decomposePar -force > ./log/decomposePar.log 2>&1
mpirun -np $SLURM_NTASKS checkMesh -parallel  > ./log/checkMesh.log 2>&1
mpirun -np $SLURM_NTASKS redistributePar -parallel -overwrite  > ./log/redistributePar.log 2>&1
mpirun -np $SLURM_NTASKS renumberMesh -parallel -overwrite -constant  > ./log/renumberMesh.log 2>&1
mpirun -np $SLURM_NTASKS patchSummary -parallel  > ./log/patchSummary.log 2>&1
mpirun -np $SLURM_NTASKS potentialFoam -parallel  > ./log/potentialFoam.log 2>&1

#------------------------------------------------------------------------------

mpirun -np $SLURM_NTASKS $(getApplication) -parallel   > ./log/$(getApplication).log 2>&1