IBM p690

Job Submission

LoadLeveler 3.4.1.0 is the queuing software. All parallel tasks require submission to the queue. Submission of jobs is done with the llsubmit command. The queue can be queried with llq.

Submit jobs with
llsubmit job_script

Display the queue status with
llq

Cancel jobs with
llcancel job_id
where job_id can be retrieved from llq.

Example Job Scripts

Serial job

#!/bin/csh -f
#
# @ executable = serial_programme
# @ output = serial_programme.out
# @ error = serial_programme.err
# @ resources        = ConsumableCpus(1)
# @ wall_clock_limit = 0:20:0
# @ queue

# Sets the number of data segments to the maximum.
LDR_CNTRL=MAXDATA=0x80000000

Parallel MPI job

#!/bin/csh -f
#
#@ job_type         = parallel
#@ output           = output.log
#@ error            = error.log
#@ node             = 1
#@ tasks_per_node   = 32
#@ resources        = ConsumableCpus(1)
#@ wall_clock_limit = 0:20:0
#@ class = parallel
#@ queue

# Sets the number of data segments to the maximum.
LDR_CNTRL=MAXDATA=0x80000000

poe ./PMRT ic_128_z=009.bin 1 1

Note, tasks_per_node sets the number of MPI processes to start while the ConsumableCpus resource sets the number of threads per MPI process.

The following attibutes may improve performance:

#@ environment      = MEMORY_AFFINITY=MCM; \
#                     MP_SHARED_MEMORY=yes; MP_WAIT_MODE=poll; \
#                     MP_SINGLE_THREAD=yes; MP_TASK_AFFINITY=MCM

OpenMP job

#!/bin/csh -f
#
#@ job_type         = serial
#@ output           = output.log
#@ error            = error.log
#@ resources        = ConsumableCpus(8)
#@ wall_clock_limit = 200:00:00
#@ class = parallel
#@ queue

set DIR=/my/run/dir

cd $DIR

setenv OMP_SCHEDULE guided
setenv OMP_NUM_THREADS 8

$DIR/myprog_openmp my_arguments

Troubleshooting

Try:

llq
llq -s JID
llclass

The Royal Observatory Edinburgh comprises the UK Astronomy Technology Centre of the Science and Technology Facilities Council, the Institute for Astronomy of the University of Edinburgh and the ROE Visitor Centre.