Compiling CUDA solvers in Foam Extend v 3.2

Today's computing world is slowly and steadily shifting from CPU to GPU. With the availability of resources at much lower rates then some years ago and the need for faster + efficient computing has resulted computational scientist to broaden their view of using GPU for computation and not see it just for gaming purpose.

The OpenFOAM extend group has contributed to this by adding libraries for solvers to harness the power of GPU. I have an NVIDIA GeForce GTX 650 Ti 2GB Graphics card and first wanted to compile the cudaSolvers in foam-extend.
So some of the basic steps which are required for compilation are as shown below :

1. Install NVIDIA cuda libraries for your system ( https://developer.nvidia.com/cuda-downloads ). 

2. After this before compiling foam-extend source the environment variable for compiling cuda solvers. On the terminal type the command below and hit enter

export CUDA_ARCH=sm_30

3. Compile foam-extend by running the ./Allwmake.firstInstall script

4. Once the compilation is finished we can now compile the CUDA solvers separately by running the Allwmake file inside $FOAM_SRC/cudaSolvers

5. But wait, this might still throw up some errors. So we need to complete a few more steps before compiling cudaSolvers

a ) Set the path for CUDA nvcc. You can save these lines in you etc/bashrc file so you can save time exporting these everytime you need to run the cuda based solvers

export set PATH=/usr/local/cuda-7.5/bin:$PATH

export set LD_LIBRARY_PATH=/usr/local/cuda-7.5/lib64:$LD_LIBRARY_PATH

b ) You will also require CUSP for compiling the cuda libraries

cusp library v0.4.0

Download it from : https://github.com/cusplibrary/cusplibrary.git

Rename the zip file as " cusp " .This library needs to be saved inside the cuda directory in /usr/local/cuda-7.5

c ) I had some trouble while compiling the cudaSolvers, but when I checked options file in Make directory my error was solved. Basically we need to set the CUDA_INCLUDE_DIR path to include all the required files for running cudaSolvers.

export CUDA_INCLUDE_DIR=$CUDA_DIR/include

d ) Now compile the cudaSolvers , and once it says that

'libcudaSolvers.so' is up to date

Your work is done

Now my next task would be to work with these solvers and check how fast the cuda libraries work for OpenFOAM extend.

Compiling CUDA solvers in Foam Extend v 3.2

Today's computing world is slowly and steadily shifting from CPU to GPU. With the availability of resources at much lower rates then some years ago and the need for faster + efficient computing has resulted computational scientist to broaden their view of using GPU for computation and not see it just for gaming purpose.

The OpenFOAM extend group has contributed to this by adding libraries for solvers to harness the power of GPU. I have an NVIDIA GeForce GTX 650 Ti 2GB Graphics card and first wanted to compile the cudaSolvers in foam-extend.
So some of the basic steps which are required for compilation are as shown below :

1. Install NVIDIA cuda libraries for your system ( https://developer.nvidia.com/cuda-downloads ). 

2. After this before compiling foam-extend source the environment variable for compiling cuda solvers. On the terminal type the command below and hit enter

export CUDA_ARCH=sm_30

3. Compile foam-extend by running the ./Allwmake.firstInstall script

4. Once the compilation is finished we can now compile the CUDA solvers separately by running the Allwmake file inside $FOAM_SRC/cudaSolvers

5. But wait, this might still throw up some errors. So we need to complete a few more steps before compiling cudaSolvers

a ) Set the path for CUDA nvcc. You can save these lines in you etc/bashrc file so you can save time exporting these everytime you need to run the cuda based solvers

export set PATH=/usr/local/cuda-7.5/bin:$PATH

export set LD_LIBRARY_PATH=/usr/local/cuda-7.5/lib64:$LD_LIBRARY_PATH

b ) You will also require CUSP for compiling the cuda libraries

cusp library v0.4.0

Download it from : https://github.com/cusplibrary/cusplibrary.git

Rename the zip file as " cusp " .This library needs to be saved inside the cuda directory in /usr/local/cuda-7.5

c ) I had some trouble while compiling the cudaSolvers, but when I checked options file in Make directory my error was solved. Basically we need to set the CUDA_INCLUDE_DIR path to include all the required files for running cudaSolvers.

export CUDA_INCLUDE_DIR=$CUDA_DIR/include

d ) Now compile the cudaSolvers , and once it says that

'libcudaSolvers.so' is up to date

Your work is done

Now my next task would be to work with these solvers and check how fast the cuda libraries work for OpenFOAM extend.

12 step guide on Installing OpenFOAM on Mac OS ( El-Capitan 10.11.6 )

I recently purchased a Mac-Book Pro 13'' 2012 Model, an old model but functioning as good as new. Getting a Mac for myself was in my bucket-list and finally I own it now.

Now the First Task was to Install OpenFOAM on this and let me tell you it wasn't easy at all. And finally after almost a week's struggle I have finally been able to Install it on Mac and that too the latest version of OpenFOAM 4.0 with Paraview-5.1.0 !!!!!

As it was a good long struggle for me and since there is no one stop solution for installation let me help you with the Installation. Here are the steps for installing OpenFOAM on Mac OS.

1. Install the homebrew package ( brew.sh ).  This basically is like a onestop package manager for all the dependencies required for Installing OF.

2. Now Install these packages one by one

 $ brew tap hombre/science

 $ brew install open-mpi  --disable-fortran

 $ brew install scotch

 $ brew install cgal

 $ brew install flex
 
 $ brew install boost --without-single --with-mpi

 $ brew install metis

 $ brew install parmetis

 $ brew install https://raw.githubusercontent.com/mrklein/openfoam-os-x/master/formulae/parmgridgen.rb

3. Once done with Installing all these packages ( would take about and hour or more depending on your Internet speed ), Download the latest version of OpenFOAM tar file from the OpenFOAM website or git or bitbucket.

4. Create a disk image using the CLI interface from the command shown below.

$ hdiutil create -size 8.3g -type SPARSEBUNDLE -fs HFSX -volname OpenFOAM -fsargs -s OpenFOAM.sparsebundle

Size key specifies size of the image, as usually I keep all OpenFOAM versions on one image, the size is rather large but as it is sparse image, its size will grow as necessary (for example size of disk image with 6 different OpenFOAM versions is around 6G). Type of the disk image is Sparse bundle as it is more convenient for backup software. Volume name is OpenFOAM, you can change it to anything you like. File system type is set with -fs flag, in the command it is just extended HFS, if you'd like, you can add J letter there to get journaling. And finally -fsargs -s supply -s option to newfs utility that forces it to create case sensitive file system.

5. Mount the Disk Image

First create a directory with the name OpenFOAM

$ mkdir -p OpenFOAM

Now mount the disk image on the newly created folder

$ hdiutil attach -mountpoint $HOME/OpenFOAM OpenFOAM.sparsebundle

6. Extract the tar file into your OpenFOAM directory in home folder

$ cd OpenFOAM && tar xzf ~/Download/OpenFOAM-4.x.tgz

7. Download the Mac OS patch for the specific version you are using and move it into OpenFOAM-4.x directory in your OpenFOAM folder in $HOME

$ cd OpenFOAM-4.x && curl -L https://raw.githubusercontent.com/mrklein/openfoam-os-x/master/OpenFOAM-<VER>.patch > OpenFOAM-<VER>.patch

Here VER refers to your version number

8. Apply the downloaded patch

$ git apply OpenFOAM-4.0.patch

9. Building OpenFOAM

Follow these few steps before you start compiling OpenFOAM

$ mkdir -p $HOME/.OpenFOAM

$ echo 'WM_COMPILER=Clang' > $HOME/.OpenFOAM/prefs.sh

$ echo 'WM_COMPILE_OPTION=Opt' >> $HOME/.OpenFOAM/prefs.sh

$ echo 'WM_MPLIB=SYSTEMOPENMPI' >> $HOME/.OpenFOAM/prefs.sh

$ echo 'export WM_NCOMPPROCS=$(sysctl -n hw.ncpu)' >> $HOME/.OpenFOAM/prefs.sh

$ echo 'export WM_SILENT_RULES=Y' >> $HOME/.OpenFOAM/prefs.sh

$ echo 'WM_LABEL_SIZE=32' >> $HOME/.OpenFOAM/prefs.sh

$ source etc/bashrc

$ [ "$(ulimit -n)" -lt "4096" ] && ulimit -n 4096

$ ./Allwmake > log.Allwmake 2>&1

COMPILATION will take about 2-3 hours


10. Testing your Installation

You can check your Installation with the regular way, by typing icoFoam -help and look out for the message or by typing

foamInstallationTest

This basically checks if all the packages are configured properly by OpenFOAM during compilation

11. Setting up Environment Variable

Sourcing OpenFOAM bashrc file each and every time you login to your system can be quite boring and time consuming. To avoid this copy past these lines into a file named as  :  

openfoam-env-setup.sh and save this on your $HOME folder. Source this file into .profile in your terminal by typing source $HOME/openfoam-env-setup.sh

#!/bin/sh

# If you'd like to setup environment every time terminal is launched
# create .OpenFOAM/OpenFOAM-release file in your home folder. In the file put
# the string with a version you'd like to use. This can be done with:
# $ mkdir -p .OpenFOAM
# $ cat '2.X.Y' > .OpenFOAM/OpenFOAM-release
# If you'd like to switch environment between versions use of2xy commands.

readonly FOAM_MOUNT_POINT="${FOAM_MOUNT_POINT:-"$HOME/OpenFOAM"}"
readonly FOAM_RELEASE_FILE="${FOAM_RELESE_FILE:-"$HOME/.OpenFOAM/OpenFOAM-release"}"
readonly FOAM_DISK_IMAGE="${FOAM_DISK_IMAGE:-"$HOME/OpenFOAM.sparsebundle"}"

mount_disk_image () {
 local oldpwd="$(pwd)"
 cd "$HOME"
 # Attempt to mount image
 hdiutil attach -quiet -mountpoint "$FOAM_MOUNT_POINT" "$FOAM_DISK_IMAGE"
 cd "$oldpwd"
 return 0
}

main () {
 [ -f "$FOAM_RELEASE_FILE" ] || return 1

 local release="$(cat "$FOAM_RELEASE_FILE")"
 local bashrc="$FOAM_MOUNT_POINT/OpenFOAM-$release/etc/bashrc"

 [ -f "$bashrc" ] || mount_disk_image

 if [ -f "$bashrc" ]
    then
  source "$bashrc" WM_NCOMPPROCS="$(sysctl -n hw.ncpu)"
 else
  echo "OpenFOAM $release doesn't seem to be installed."
 fi
}

# Reset environment variables for specified version
ofxxx () {
 local release="$1"
 [ -n "$WM_PROJECT_DIR" ] && . "$WM_PROJECT_DIR/etc/config/unset.sh"
 local bashrc="$FOAM_MOUNT_POINT/OpenFOAM-$release/etc/bashrc"
 if [ -f "$bashrc" ]; then
  source "$bashrc" WM_NCOMPPROCS="$(sysctl -n hw.ncpu)"
 else
  mount_disk_image
  if [ -f "$bashrc" ]; then
   source "$bashrc" WM_NCOMPPROCS="$(sysctl -n hw.ncpu)"
  else
   echo "OpenFOAM $release doesn't seem to be installed."
  fi
 fi
}

of22x () {
 ofxxx "2.2.x"
}

export -f of22x

of231 () {
 ofxxx "2.3.1"
}

export -f of231

of23x () {
 ofxxx "2.3.x"
}

export -f of23x

of240 () {
 ofxxx "2.4.0"
}

export -f of240

of24x () {
 ofxxx "2.4.x"
}

export -f of24x

of300() {
 ofxxx "3.0.0"
}

export -f of300

of301() {
 ofxxx "3.0.1"
}

export -f of301

of30x() {
 ofxxx "3.0.x"
}

export -f of30x

of40() {
    ofxxx '4.0'
}

export -f of40

of4x() {
    ofxxx '4.x'
}

export -f of4x

ofdev() {
    ofxxx "dev"
}

export -f ofdev

pf () {
 paraFoam -builtin > /dev/null 2>&1 &
}

main

12. Restart Terminal !!!!

Now every time you start the terminal just type the alias related to your version

eg. of40

Now, this is for all the MAC users. Do not install packages Macports, it will create issues while compilation since we have installed packages from brew. ( from personal experience )

The Foam version Confusion

Hi Guys,

The Open source CFD seems to be getting more competitive these days and guess what we are now having three different versions of OpenFOAM. Isn't that great but again adding to a bit of confusion as well, which version should I really use. Well lets clarify some of the things for now beginning with the Foam hierarchy as shown below 

Figure taken from a LinkedIn article on OpenFOAM who's who.

Well so as we can see we have three different versions released by the OpenFOAM Foundation, OpenCFD-ESI Group and Foam-Extend. So what really differs in each of these, let find out

1. OpenFOAM 4.0 - This is the official Foam release and has a lot of new features added with it from its previous version. OpenFOAM foundation releases this version and this one seems to be the quickest release ( 6 months ) then all the prior ones. I would always prefer for going for this release to keep the originality.

2. OpenFOAM v1606+ - This version is released by OpenCFD-ESI Group and comes with a lot of      add on solvers used by Industry and some really cool post-processing options as well. The version naming is done as year and month of release with a +. One can look at the features offered by this software and use it for solving problems.

3. foam-extend - Here you will get some of the most extensive solvers which are not available in         either of the above Foam. The solvers are mainly contributed by OpenFOAM users / developers globally.

So finally to end the confusion it really doesn't matter much about what Foam version you want to use till the time it really solves your engineering problem. Once again, happy Foaming!!! :)