The Slurm workload scheduler is used to manage the compute nodes on the cluster. Jobs must be submitted through the scheduler to have access to compute resources on the system. There are two groups of partitions (aka "queues") currently configured on the system. One group is specifically designated for CPU jobs, i.e., non-GPU jobs: defq a general partition, short, tiny and nano compute nodes, highThru high throughput or highMem high memory. NOTE: Running jobs on the login nodes is prohibited and may result in account suspension.
- highThru – Nodes in this category have large memory (384gb), high CPU frequency nodes with a low core count. These nodes are used for single threaded jobs that cannot take advantage of multiple cores.
- highMem –Nodes in this category have large memory – 3tb and are for jobs that require more memory intensive jobs.
- debug-cpu - please see DebugPartition for more details. These nodes have 96gb of memory
- defq - With 384gb of memory, this partition is designed for longer running jobs - 14 days. Shorter jobs can be submitted here but it's advised to really modify your job run time in your submit script and submit it to the appropriate partition for max efficiency.
- 384gb - With 384gb of memory, this partition is designed for longer running jobs - 14 days. Some nodes here overlap with defq.
- short - With 96gb of memory, this partition is designed for quicker turnaround of shorter running jobs – 1 day. Some nodes here overlap with defq.
- short-384gb - Similar to the other short, except with 384gb of memory. Designed for quicker turnaround of shorter running jobs – 1 day. Some nodes here overlap with defq
- tiny - With 96gb of memory, this partition is designed for even quicker turnaround of shorter running jobs – 4 hours. Some nodes here overlap with defq.
- nano - With 96gb of memory, this partition is designed for the fastest turnaround of shorter running jobs - 30 min. These nodes don't overlap with other partitions.
The other is specifically for jobs requiring GPU resources and should not be used for jobs that do not require GPU resources:
- small-gpu – has access to the GPU nodes, each has two NVIDIA NVIDIA Tesla V100 GPU.
- large-gpu – has access to four Nvidia Tesla V100 SXM2 16GB GPUs with NVLink enabled
- debug-gpu - please see DebugPartition for more details.
Note that you must set a time limit (with the -t flag, for example -t 1:00:00 would set a limit of one hour) for your jobs when submitting, otherwise they will be immediately rejected. You must also set a partition (with the -p flag for example -p short would submit it to the short partition) for your jobs otherwise they will be immediately rejected. This allows the Slurm scheduler to keep the system busy by backfilling when trying to allocate resources for larger jobs. The maximum time-limit for any job is 14 days. We will not under any circumstances increase time limits for jobs already begun. Please estimate your required time carefully and request a time limit accordingly. Longer running processes should checkpoint and restart to avoid losing significant amounts of outstanding work if there is a problem with the hardware or cluster configuration. We will provide guidance to help you better understand your job's requirements if you are unsure. The time-node limit for a single job may not exceed 224 node*days which prevents any single job from allocating more the 16 nodes for 14 days.
As an example of a simple MPI job script, which could be submitted as sbatch job_script.sh:
#!/bin/sh # one hour timelimit: #SBATCH --time 1:00:00 # default queue, 32 processors (two nodes worth) #SBATCH -p defq -n 32 module load openmpi mpirun ./test
The Slurm documentation has further documentation of some of the advanced features. The Slurm Quick-Start User Guide provides a good overview. The use of Job Arrays (Job Array Support) is mandatory for people submitting a large quantity of similar jobs.
Matlab Example
#!/bin/bash
# set output and error output filenames, %j will be replaced by Slurm with the jobid
#SBATCH -o testing%j.out
#SBATCH -e testing%j.err
# single node in the "short" partition
#SBATCH -N 1
#SBATCH -p short
# half hour timelimit
#SBATCH -t 0:30:00
module load matlab/2022a
# test.m is your matlab code
matlab -nodesktop -r "run('test.m'), exit"
