1. Overview
  2. Hardware
  3. Setting up your environment
  4. Compiling
  5. Job Submission


InfiniBand is a high-speed, low-latency communication network connecting some of the stacpolly worker nodes.

Making use of the Infiniband connexions is currently limited to parallel codes that use the Message Passing Interface (MPI) library to communicate between processes. A minor change to your environment and the use of an InfiniBand-aware MPI implementation is required. Also, nodes with InfiniBand must be requested at job-submission time.


The eight compute nodes worker049 to worker056 are equipped with FDR InfiniBand connexions with throughputs of up to 14 Gb/s and latencies of 1 us.

The 27 compute nodes worker057 to worker083 are equipped with DDR InfiniBand connexions with throughputs of up to 4 Gb/s and latencies of 2.5 us.

These numbers should be compared to those achieved with the default TCP (ethernet) connexions which provide throughputs of up to 100 Mb/s and latencies of 24 us.

The two groups of InfiniBand-connected nodes are also connected and can be used in any combination, but will, of course, default to the slowest connexion speed.

Setting up your environment

InfiniBand requires unlimited access to locked memory. However, your shell will default to a limited amount of locked memory. Fortunately, the default value can be overridden. Each shell will have it's own method for resetting limits and a number of places to do it. I recommend setting the limit in your .tcshrc or .bashrc, depending on whether your default shell is CSH/TCSH or BASH.

Add to the non-interactive part of your ~/.tcshrc file, normally near the top:
limit memorylocked unlimited
Add to the non-interactive part of your ~/.bashrc file, normally near the top:
ulimit -l unlimited

Compiling for infiniband

The paths listed here are liable to change. A separate build of OpenMPI was done to avoid breaking tools in use. Once verified that an InfiniBand-aware OpenMPI still manages to build for non-InfiniBand nodes, the separate branch will disappear.

Point your makefile to use the MPI tools in

Currently, InfiniBand-aware OpenMPI-1.8.4 has been built for gcc-4.7.2 only. Support for the Intel compilers will be coming.

Job submission

InfiniBand-aware nodes are selected with the infiniband property. See Job Submission Properties for details.

Last modified 2015-May-19 by Eric Tittley