Skip to content

Running Python on CREATE

Using modules, python/3.11.6 is the current default version on CREATE HPC.

Tip

To avoid memory limitations on the login nodes, request an interactive session to complete your installation process. Python and its' dependencies can require more memory than what is allocated as default. Please visit our requesting more resources page on how to request more resources when using CREATE HPC.

How to use Python via Modules

You can use the Python module within CREATE to run Python interactively within the terminal and load Python packages:

1
2
3
4
5
$ module load python/3.11.6-gcc-13.2.0
$ python
>>> import sys
>>> print(sys.version)
3.11.6 (main, Mar  7 2024, 18:26:40) [GCC 13.2.0]

Setting up and activating a virtual environment

1
2
3
4
5
6
$ virtualenv venv -p `which python3`
$ source venv/bin/activate
(venv) $ python
>>> import sys
>>> print(sys.version)
3.11.6 (main, Mar  7 2024, 18:26:40) [GCC 13.2.0]

Installing packages

After activating the environment, any packages installed with pip will be added to the virtual environment, not the default environment

1
2
$ pip install numpy
$ pip install pandas

Tip

An alternative to pip is conda. Please refer to our documentation here

Submitting a batch job

To execute a python script in a batch job using the newly installed virtual environment, an sbatch script similar to the following can be used:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
#!/bin/bash -l
#SBATCH --job-name=python_job
#SBATCH --output=/scratch/users/%u/%j.out
#SBATCH --partition=cpu
#SBATCH --ntasks=1
#SBATCH --mem=2G
#SBATCH --cpus-per-task=1
module load python/3.11.6-gcc-13.2.0
source venv/bin/activate
python your_file.py
deactivate
1
k1234567@erc-hpc-login1:/scratch/users/k1234567$ sbatch bash_script.sh

Using Singularity

Singularity is a containerisation tool, similar to Docker, which can be used to run python on CREATE HPC. The following command can be used to download the lastest version of python available from Docker Hub:

Tip

To avoid storage issues when downloading Singularity images, please refer to the Alternative cache location section of our Singularity documentation

1
$ singularity pull docker://python

Tags can also be used for downloading more specific versions:

1
$ singularity pull docker://python:3.11

Interactive Python containers

Use the following commands to run an interactive containerised Python version:

1
2
3
4
5
6
7
$ singularity shell python_3.11.sif python
Singularity> python
Python 3.11.10 (main, Sep  9 2024, 18:05:22) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> print(sys.version)
'3.11.13 (main, Jul  1 2025, 05:28:08) [GCC 12.2.0]'

Note

This example demonstrates shell and interactive Python use within a container. For browser-based usage, see our Jupyter usage guide.

Installing specific containers

By default, your host home directory (/users/k1234567/...) is an accessible mounted path in your Python container. Therefore, all packages you install in your container will be written to your home directory - the default library location - so to avoid using too much of your home directory storage, you can specify and bind a different HPC location for your container to install to:

1
2
singularity shell --bind /scratch/users/k1234567/software:/my-libraries python_3.11.sif
Singularity> ls /my-libraries

The above binds your host scratch path: /scratch/users/k1234567/software to a new example container path: /my-libraries, where any changes that occur on /my-libraries in your container will correspond to /scratch/users/k1234567/software, however, within the context of your container, only /my-libraries is accessible, and the path /scratch/users/k1234567/software does not exist.

You may intend to install containers that contain specific packages. Here's an example of jupyter/scipy-notebook which contains common Data Science packages like numpy and pandas:

1
$ singularity pull docker://jupyter/scipy-notebook

Running containerised Python scripts

Singularity containers can also be used to run your required Python scripts. In this example, we will create a python script named test.py located in /scratch/users/k1234567, which is bound to the following mount in the container on: /my-data. The script can will then be executed through sbatch:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
import numpy as np
import pandas as pd
data = {
 'Name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
 'Age': [24, 27, 22, 32, 29],
 'Score': [85, 91, 78, 95, 88]
}
df = pd.DataFrame(data)
mean_score = df['Score'].mean()
print("Data frame:", df)
print("Mean score:", mean_score)
1
2
3
4
5
#!/bin/bash -l
#SBATCH --job-name=ops-python-container
#SBATCH --partition=cpu
CONTAINER_PATH=/scratch/users/k1234567/scipy-notebook_latest.sif
singularity exec --bind /scratch/users/k1234567:/my-data ${CONTAINER_PATH} python /my-data/test.py

Running GUI applications with Python

For a detailed example, please refer to Testing Visualisation with Python.