Running Python on CREATE¶
Using modules, python/3.11.6 is the current default version on CREATE HPC.
Tip
To avoid memory limitations on the login nodes, request an interactive session to complete your installation process. Python and its' dependencies can require more memory than what is allocated as default. Please visit our requesting more resources page on how to request more resources when using CREATE HPC.
How to use Python via Modules¶
You can use the Python module within CREATE to run Python interactively within the terminal and load Python packages:
1 2 3 4 5 |
|
Setting up and activating a virtual environment¶
1 2 3 4 5 6 |
|
Installing packages¶
After activating the environment, any packages installed with pip
will be added to the virtual environment, not the default environment
1 2 |
|
Tip
An alternative to pip
is conda
. Please refer to our documentation here
Submitting a batch job¶
To execute a python script in a batch job using the newly installed virtual environment, an sbatch script similar to the following can be used:
1 2 3 4 5 6 7 8 9 10 11 |
|
1 |
|
Using Singularity¶
Singularity is a containerisation tool, similar to Docker, which can be used to run python on CREATE HPC. The following command can be used to download the lastest version of python available from Docker Hub:
Tip
To avoid storage issues when downloading Singularity images, please refer to the Alternative cache location section of our Singularity documentation
1 |
|
Tags can also be used for downloading more specific versions:
1 |
|
Interactive Python containers¶
Use the following commands to run an interactive containerised Python version:
1 2 3 4 5 6 7 |
|
Note
This example demonstrates shell and interactive Python use within a container. For browser-based usage, see our Jupyter usage guide.
Installing specific containers¶
By default, your host home directory (/users/k1234567/...) is an accessible mounted path in your Python container. Therefore, all packages you install in your container will be written to your home directory - the default library location - so to avoid using too much of your home directory storage, you can specify and bind a different HPC location for your container to install to:
1 2 |
|
The above binds your host scratch path: /scratch/users/k1234567/software
to a new example container path: /my-libraries
, where any changes that occur on /my-libraries
in your container will correspond to /scratch/users/k1234567/software
, however, within the context of your container, only /my-libraries
is accessible, and the path /scratch/users/k1234567/software
does not exist.
You may intend to install containers that contain specific packages. Here's an example of jupyter/scipy-notebook
which contains common Data Science packages like numpy
and pandas
:
1 |
|
Running containerised Python scripts¶
Singularity containers can also be used to run your required Python scripts.
In this example, we will create a python script named test.py
located in /scratch/users/k1234567
, which is bound to the following mount in the container on: /my-data
.
The script can will then be executed through sbatch:
1 2 3 4 5 6 7 8 9 10 11 |
|
1 2 3 4 5 |
|
Running GUI applications with Python¶
For a detailed example, please refer to Testing Visualisation with Python.