stacpolly

Job Submission

  1. Overview
  2. Setting up your environment
  3. Usage
  4. Job Scripts
    1. Serial
    2. MPI
    3. OpenMP
    4. Hybrid MPI/OpenMP
    5. Job Arrays
  5. Memory
  6. Reserving an entire node
  7. Requesting a specific node
  8. Requesting hardware properties

Overview

Torque 2.3.5 is the queuing software. Folks familiar with PBS will be at home with Torque.

All parallel tasks require submission to the queue. Submission of jobs is done with the qsub command. The queue can be queried with showq or qstat.

Setting up your environment

Follow the instructions in Environment.

Usage

Submit jobs with
qsub job_script

Display the queue status with
qstat
or
showq

Cancel jobs with
qdel job_id
where job_id can be retrieved from showq or qstat.

Other useful commands are:

showbf
see how many nodes are available for use.
mdiag (aka diagnose)
explain why jobs aren't running.
checkjob
check the status of a particular job.
pbsnodes
list nodes and their state.
pbsnodes -l "up"
list all nodes that are up.
/home/ert/bin/NodeUsage
Find the loads on each node.
/home/brs/tools/queue_stat/queue_stat
Queue summary

Job Scripts

Serial job

# The shell
#PBS -S /bin/tcsh

# The number of nodes and the number of processes per node
#PBS -l nodes=1

# The wall time for the calculation in HH:MM:SS
#PBS -l walltime=720:00:00

# Where the errors go
#PBS -e /home/foo/070419a/log.error

# Where the messages go
#PBS -o /home/foo/070419a/log

set DIR = /home/foo/070419a

# The work directory
cd $DIR

# Run the executable
/home/foo/bin/my.executable < my.input > my.output

For running an octave job: 

octave --silent --no-window-system Job.m > Job.log

MPI job

The following will run PMRT on 16 cores spread over 2 nodes. Note, you are unable to request more than 8 processors per node (ppn). There are 17 nodes (16 8-way and 1 6-way). 

# The shell
#PBS -S /bin/tcsh

# The number of nodes and the number of processes per node
#PBS -l nodes=2:ppn=8

# The wall time for the calculation in HH:MM:SS
#PBS -l walltime=720:00:00

# Where the errors go
#PBS -e $HOME/PMRT/Runs/Polar/L=1e21/090107f/log.error

# Where the messages go
#PBS -o $HOME/PMRT/Runs/Polar/L=1e21/090107f/log

set DIR = $HOME/PMRT/Runs/Polar/L=1e21/090107f
set EXE = $HOME/bin/PMRT_Polar_HY_zi=8_zt=5.5
set IC  = $HOME/PMRT/ICs/ic_256_z=009.bin.min

# The work directory
cd $DIR

module load openmpi

# Run the executable
mpirun $EXE $IC 1 1 >& PMRT.log

OpenMP job

# The shell
#PBS -S /bin/tcsh

# The number of nodes and the number of processes per node
# OpenMP cannot request more than one node, and no more than the number of
# cores on the largest node, currently 64, but less for most nodes
#PBS -l nodes=1:ppn=8

# The wall time for the calculation in HH:MM:SS
#PBS -l walltime=720:00:00

# Where the errors go
#PBS -e log.error

# Where the messages go
#PBS -o log

set DIR = $HOME/PMRT/Runs/Polar/L=1e21/090107f
set EXE = $HOME/bin/PMRT_Polar_HY_zi=8_zt=5.5
set IC  = $HOME/PMRT/ICs/ic_256_z=009.bin.min

# The work directory
cd $DIR

setenv OMP_NUM_THREADS 8

# Run the executable
$EXE $IC 1 1 >& PMRT.log

Hybrid MPI/OpenMP job

# The shell
#PBS -S /bin/tcsh

# The number of nodes and the number of processes per node
# 4 nodes, with 8 processors per node, so 4 MPI jobs, with 8 threads each
#PBS -l nodes=4:ppn=8

# The wall time for the calculation in HH:MM:SS
#PBS -l walltime=1:00:00

# Where the errors go
#PBS -e log.error

# Where the messages go
#PBS -o log

set DIR = $HOME/Comp/test_HybridJob
set EXE = $DIR/MPI_OpenMP_test

# The work directory
cd $DIR

# Each process can generate 8 threads
setenv OMP_NUM_THREADS 8

# Run the executable on 4 nodes, with only one process per node
mpirun -n 4 --map-by ppr:1:node --bind-to none $EXE >& Log

Note, for the last line you can use the deprecated but clearer: 

mpirun -n 4 -npernode 1 $EXE >& Log

Job Arrays

To submit a number of very similar jobs, perhaps varying an input data file or a parameter, create a job array.

Submit with:
qsub -t 0-10 JobArrayScript.pbs
or
qsub -t 1,2,5,10 JobArrayScript.pbs

JobArrayScript.pbs is just like any of the job submission scripts above, but will be called 11 times in the first example and 4 times in the second with the environment variable $PBS_ARRAYID set appropriately.

Memory

The amount of memory required can be set with the mem parameter. This is useful if you know you need an amount of memory available only on all selection of nodes. 

# The memory requested per process. The largest requested by all processes.
#PBS -l mem=30gb

Reserving an entire node to one job

On occassion, you may want to start only a few jobs on a node, but reserve the whole node for your single job. For example, you know you will need all the memory or you want to mix threading with MPI.

The following will run the job on 2 nodes, using only 1 processor per node. 

#PBS -l nodes=2:ppn=1
#PBS -W x=NACCESSPOLICY:SINGLEJOB

mpirun -n 2 MyMPIProg

To determine the number of processors available on the assigned node: 

# Determine the number of cores available on this node
set NCores=`cat /proc/cpuinfo | grep processor | wc -l`

Requesting a specific node

On occassion, you may want to have a job placed on a particular node. For example, you already have a dataset on a local drive.

The following will run the job on worker004. 

#PBS -l nodes=worker004.cluster.loc

Requesting hardware properties

Since stacpolly is a heterogenous system, properties of the nodes differ. These properties are selectable at job-submission time. Add the following resource requests as necessary:

intel
Intel CPU
amd
AMD CPU
localdatadisc
A local /data scratch directory
gpu
Two Tesla M2090 GPUs
infiniband
InfiniBand-connected nodes
weaklensing
Nodes provided by the Weak Lensing group

Examples

#PBS -l nodes=4:intel:ppn=12

Select four Intel nodes with 12 cores per node. 

#PBS -l nodes=1:gpu

Select the GPU node.

If you want to choose multiple resources, separate the properties with a ":". 

#PBS -l nodes=1:intel:localdatadisc:ppn=1

Last modified 2015-May-19 by Eric Tittley