Running GPU jobs

To schedule a job on a GPU, provide the --gres=gpu:<count> option in the submission script. The following example schedules a job on a GPU node then lists the GPU card it was assigned

Note

Make sure to include -p gpu/-p interruptible_gpu in either the submission script or the command line when submitting the job or requesting a session (srun/salloc).

#!/bin/bash -l
#SBATCH --output=/scratch/users/%u/%j.out
#SBATCH --job-name=gpu
#SBATCH --gres=gpu:2
echo "Hello, World! From $HOSTNAME"
nvidia-debugdump -l
sleep 15
echo "Goodbye, World! From $HOSTNAME"

In the above example, it calls for 2 GPUs.

k1234567@erc-hpc-login1:~$ sbatch -p gpu hellogpu.sh
Submitted batch job 12087
k1234567@erc-hpc-login1:~$ cat /scratch/users/k1234567/12087.out
Hello, World! From erc-hpc-comp-032
Found 2 NVIDIA devices
  Device ID:              0
  Device name:            NVIDIA A100-PCIE-40GB
  GPU internal ID:        1324120038830

  Device ID:              1
  Device name:            NVIDIA A100-PCIE-40GB
  GPU internal ID:        1324120040003

Goodbye, World! From erc-hpc-comp-032

Note

The maximum number of 8 GPUs can be concurrently requested. For further scheduler policy information, see the following.

Note

A GPU enabled application will mostly likely make use of the NVidia CUDA libraries, to load the CUDA module use module load cuda in the job submission script.

Testing on the GPU

The interruptible_gpu partition gives access to all GPUs in CREATE, leading to a larger pool size that serves well as both a mechanism for testing GPU scheduling and making use of unused private resources, as detailed in our scheduler policy. So it is still important to take note that, although faster scheduling may be facilitated, jobs may be cancelled at anytime. Additionally, as the interruptible_gpu can be made up of a broad mix of GPU architectures, it may be useful to provide the following --constraint scheduler option with the job submissions:

k1234567@erc-hpc-login1:~$ srun -p interruptible_gpu --gres gpu:<count> --constraint a40 --pty /bin/bash -l

or a newer version:

k1234567@erc-hpc-login1:~$ srun -p interruptible_gpu --gpus=<count> --constraint a40 --pty /bin/bash -l

Note

Remember to replace the <count> with a number.

Constraints

The --constraint flage allows the scheduler to restrict the pool of GPUs.

How to find available constraints

To list all the nodes in a given partition(s) with groups that can access them and the features available, user sinfo, e.g.

sinfo -p gpu,interruptible_gpu -o "%P %n %g %f"

Which would output:

PARTITION HOSTNAMES GROUPS AVAIL_FEATURES
gpu erc-hpc-comp034 er_hpc_create_users,hpcsys,er_hpc_admins icelake,a100,a100_40g,ib
gpu erc-hpc-comp031 er_hpc_create_users,hpcsys,er_hpc_admins icelake,a100,a100_40g,ib
...
interruptible_gpu erc-hpc-comp051 er_hpc_create_users,hpcsys,er_hpc_admins icelake,a100,a100_40g,ib
interruptible_gpu erc-hpc-comp127 er_hpc_create_users,hpcsys,er_hpc_admins icelake,a40

This lists the following information::

• %P: partition name e.g. gpu, interruptible_gpu
• %n: hostname e.g. erc-hpc-comp242
• %g: groups that can use said nodes e.g. er_hpc_create_users
• %f: available features e.g. icelake, a100, a100_40g, etc.

The above command helps list all gpu and interruptible_gpu partitions. To list all partitions, just remove the -p flag e.g.

sinfo -o "%P %n %g %f"

Example constraints

Now with the information provided we can set up example constraints:

• --constraint="icelake": The scheduler restricts it to Icelake GPU nodes.
• --constraint="icelake&a100": The scheduler restricts to nodes with both Icelake and A100.
• --constraint="icelake|a100": The scheduler restricts to nodes with either Icelake or A100.
• --constraint="[icelake|a100|a100_40g|ib]": The scheduler restricts to nodes with either Icelake, A100, A100 40GB or IB.
• --constraint=icelake*2: The scheduler needs at least 2 nodes with Icelake.

For further information on --constraints flag for sbatch, please consult the SLURM documentation.

GPU credits for priority usage

Note

Information about paying for GPU access and costs can be found in the GPU pay per use section.

Each tier in the GPU pay per user table has a corresponding partition that gives access to the GPUs in that tier:

• Tier 1:
  • Partition name:tier1_gpu
  • GPUs available: B200, H200
• Tier 2:
  • Partition name:tier2_gpu
  • GPUs available: B200, H200
• Tier 3:
  • Partition name:tier2_gpu
  • GPUs available: B200, H200

To use GPU credits, the project (account) the credits have been allocated to and a partition need to be specifed e.g.

k1234567@erc-hpc-login1:~$ sbatch --partition tier1_gpu --account <er_prj_dept_name> example_run.sh

This can be combined with a --constraint to further specify the GPU (or any other feature required).