Compute

Our group has access to the Killarney compute cluster (among others). Follow the guide below to get started. Based on Scott Lowe's guide from Vector.

Getting Started

This guide assumes you are using a Unix-based OS (Mac or Linux) and know the basics of navigating a terminal. The compute clusters all run Linux, so if you are not familiar with it, you should start reading up ASAP.

Gaining Access

1. Register for a CCDB account: https://ccdb.alliancecan.ca/security/login
   1. I suggest using your UWO username (e.g. pmoore44) as your CCDB username.
   2. Use Western as your institution, not Vector.
   3. Agustinus' CCRI is hsp-434-01.
2. Log in and apply for a role.
   1. My Account > Apply for a New Role
   2. Again, use Western affiliation, not Vector.
   3. Agustinus will accept your application and add you to the AIP project.
3. Set up MFA.
   1. My Account > Multifactor Authentication
4. Set up your SSH key.

   1. My Account > SSH Keys

   2. Copy your SSH public key, typically stored at ~/.ssh/id_rsa.pub or ~/.ssh/id_ed25519.pub.

   3. If you don't have a public key, generate one using (on Linux/Mac):

      ssh-keygen -t ed25519 -C "your_email@example.com"

   4. Paste the contents into the SSH Key box and add the key.

5. Request access to Killarney.
   1. Resources > Access Systems
   2. Under the Artificial Intelligence tab, select "I request access for Killarney".
   3. Wait for access to be granted.

Connecting

1. SSH into the cluster: ssh ccdb-username@killarney.alliancecan.ca
2. Select (1) to use Duo.
3. You will land on one of the login nodes.

tip

Add an SSH host to your sshconfig (~/.ssh/config) so you don't have to type out the address all the time:

Host cc
    HostName killarney.alliancecan.ca
    User ccdb-username
    Port 22

Cluster Directory Structure

There are three directories that you can use, each with its own use case:

Layout

/
├── home/
│   └── $USER/                          HOME — 50 GB
│       │                                  configs & binaries (nvim, tmux, turm)
│       └── projects/
│           └── aip-akristi/
│               └── $USER/              PROJECT — 2000 GB (symlink)
├── project/
│   └── aip-akristi/
│       └── $USER/                      PROJECT — code & results
└── scratch/
    └── $USER/                          SCRATCH — 1000 GB
                                           datasets, weights, temp files

Home - /home/$USER/

Small quota (50 GB). Use this for configs or binaries like nvim, tmux, and turm.

Scratch - /scratch/$USER/

Large quota (1000 GB), but inactive data is automatically deleted. Use it for large files and temporary storage, e.g. a dataset or model weights.

Project - /project/aip-akristi/$USER/

Larger quota (2000 GB), but shared between all group members. This is where your code and results should live. Also reachable at /home/$USER/projects/aip-akristi/$USER.

Most of your work will be done from your project directory.

tip

Add an alias to your .bashrc that cds you to your project directory:

alias p='cd ~/projects/aip-akristi/$USER/'

Jobs

Killarney (and every other Alliance cluster) uses Slurm to fairly divide access to compute. This is achieved with a queue system: you submit a script detailing how to run your program, and it is added to the queue. Your position in the queue depends on how much compute the job needs, how much of the shared resources our group has been using, and when the job was submitted.

Monitoring Jobs

Before we get into running jobs, we need a way to monitor their status. Are they in the queue? Running? Crashed? There are a few ways of checking on them:

1. squeue

   Use squeue to view the current state of jobs in the scheduling queue.

   • List all your active jobs: squeue -u $USER
   • Check a specific job: squeue -j job_id

   A job will be in one of three states:

   1. R: Running
   2. PD: Pending (waiting for resources)
   3. CG: Completing

2. turm

   turm is a TUI (text-based user interface) for Slurm that displays job statuses, streams log output interactively, and manages jobs. This is the best way to monitor your jobs.

   To install it, make sure ~/.local/bin exists and is on your $PATH, then run:

   wget https://github.com/karimknaebel/turm/releases/latest/download/turm-x86_64-unknown-linux-musl.tar.gz -O - | tar -xz -C ~/.local/bin/

   Then run turm.

   By default this displays the entire squeue output, which is not very useful. Instead, you should also specify a user or account: turm -u pmoore44, or turm --account=aip-akristi.

tip

Create an alias in your .bashrc to quickly check on your own jobs in the queue:

alias t='turm --me'

Submitting a Job

The login nodes are not very powerful and should not be used to run any code. They are for navigating the directory structure, writing code, and submitting jobs. To submit a job, use the sbatch command with a script such as the one below.

Example single job Slurm script

#!/bin/bash

#SBATCH --job-name=run_main
#SBATCH --output=job_%j.out
#SBATCH --error=job_%j.err
#SBATCH --account=aip-akristi
#SBATCH --time=00:03:00
#SBATCH --gres=gpu:l40s:0
#SBATCH --mem=1G

# Clean existing modules and load Python or whatever other modules you need.
module purge
module load python/3.10

# Create a virtual environment (if one does not already exist).
VENV_PATH="ENV"
virtualenv --no-download --quiet $VENV_PATH

# Activate the virtual environment.
source $VENV_PATH/bin/activate

# There are prebuilt PyPI wheels for some packages already on Killarney in a
# local wheelhouse. These are optimized for the hardware we have on Killarney,
# so they should work better than generic wheels from pypi.org. But only a
# subset of possible package versions is available. You can see which versions
# are available with the `avail_wheels` command, e.g. `avail_wheels torch`.

# If it is not vital to replicate the environment exactly, I recommend
# adjusting exact version requirements (e.g. `numpy==2.2.1`) to something which
# is available in the wheelhouse (change it to `numpy==2.2.2` or `numpy~=2.2.1`
# to allow installation of the 2.2.2 version in the wheelhouse). In particular,
# you may get a speed-up for packages which handle data/compute
# (e.g. torch, numpy, scipy, Pillow, pandas).

# Upgrade pip and install requirements (if applicable).
echo "Installing Requirements..."
pip install --no-index --upgrade -q pip

pip install -q -r requirements.txt --find-links https://pypi.org/simple/ --prefer-binary
echo "Done."

# Execute the script.
python main.py

This script will submit a job requesting resources, and the job will be added to the queue. Slurm will schedule it to be run on the appropriate compute node.

important

Jobs must be submitted from within your project directory.

Resource Etiquette and Limits

Killarney has two tiers of compute nodes: standard and performance. The standard nodes are good enough for most projects. The performance nodes are useful if you need a large amount of RAM for a single process, or more VRAM. The exact specs are as follows:

Performance Tier	Nodes	Model	CPU	Cores	System Memory	Storage	GPUs per node	Total GPUs
Standard Compute	168	Dell 750xa	2 x Intel Xeon Gold 6338	64	512 GB	350 GB SSD	4 x NVIDIA L40S 48 GB	672
Performance Compute	10	Dell XE9680	2 x Intel Xeon Gold 6442Y	48	2048 GB	800 GB NVMe	8 x NVIDIA H100 SXM 80 GB	80

important

These nodes are shared across all of Vector, so try not to use more than the following:

• L40s: 128 GB RAM per GPU, 16 CPU cores per GPU
• H100: 256 GB RAM per GPU, 6 CPU cores per GPU

This allows headroom for Slurm to run other jobs concurrently on the same node, allowing everyone's jobs to run sooner.

If more is required, the hard limits are:

• L40s: 4 GPUs, 496 GB RAM, 128 CPUs
• H100: 8 GPUs, 2000 GB RAM, 48 CPUs

warning

You should set --time and --mem to as tight an upper limit as possible.

Billing is calculated at submit time, not based on actual usage. If you submit seven-day jobs on the high-power H100 nodes for a 10-minute workload, the whole group's priority will be tanked, and we will be placed at the back of the queue. Shorter jobs also start sooner.

Interactive Jobs

On the topic of cluster etiquette, there are situations where you might need to interact with a computationally expensive program ("expensive" is a strong word — the limit is quite low). Such programs degrade the experience for other users on the login node (lag, disconnections, etc.).

The solution is to run an interactive job on a compute node:

srun --account=aip-akristi --time=3:00:00 --mem=32G --gres=gpu:l40s:1 --pty bash

This will drop you onto a compute node, where you can safely perform your task. If you don't need a GPU, set --gres=gpu:l40s:0.

tip

Add a bash function to your .bashrc to quickly drop onto a compute node:

# c — grab an interactive compute job via srun
# Usage: c [srun overrides...]
# Defaults: account=aip-akristi, 1 h, 16G RAM, no GPU
c() {
  local args=(
    --account=aip-akristi
    --time="${SRUN_TIME:-1:00:00}"
    --mem="${SRUN_MEM:-16G}"
    --gres="gpu:l40s:${SRUN_GPUS:-0}"
  )
  ( cd ~/projects/aip-akristi/"$USER" || exit 1
    srun "${args[@]}" "$@" --pty bash )
}

Run it with c. If you need more than an hour, extra RAM, a GPU, etc., just override the arguments:

c --time=4:00:00 --mem=32G --gres=gpu:l40s:1

Slurm Array Jobs

There are many scenarios where you might need to run multiple variations of the same script: for example, training and testing an ML model on different random seeds to verify significance, or even running a grid sweep of hyperparameters (you should use wandb for this, but it is possible to do with arrays).

The main idea is that you submit one script which runs multiple times, each run with a different SLURM_ARRAY_TASK_ID from your specified range. You can use this to index a list of random seeds, or use it as a random seed directly. See the example script below.

Example job array Slurm script

#!/bin/bash

#SBATCH --job-name=runmain
#SBATCH --output=logs/job_%A_%a.out
#SBATCH --error=logs/job_%A_%a.err
#SBATCH --account=aip-akristi
#SBATCH --time=00:03:00
#SBATCH --gres=gpu:l40s:0
#SBATCH --mem=1G
#SBATCH --array=1-10%4

# --array=1-10%4  ->  tasks 1..10, at most 4 running at once.
# %A = array job id, %a = task id (used in the log filenames above).
# mkdir -p logs   <- do this before sbatch, or the job fails to start.

set -euo pipefail

# Build the environment in $SLURM_TMPDIR (fast node-local disk).
# This can also be done for individual jobs, but it matters much more for arrays.

module purge
module load python/3.10

virtualenv --quiet --no-download "$SLURM_TMPDIR/env"
source "$SLURM_TMPDIR/env/bin/activate"

echo "Installing Requirements..."

pip install -q --no-index --upgrade pip
pip install -q --no-index -r requirements.txt

echo "Done."

# Map task id -> parameters.
# Example: pass the ID as an argument.

python main.py $SLURM_ARRAY_TASK_ID

File Transfer

If you need to transfer a file from your local machine to the cluster, use tools like rsync or scp. Using rsync is simple:

rsync -avz /local/dir cc:/remote/dir

Uploads your local directory to the path on cc (from the sshconfig we added above). Reversing the order of the arguments downloads the remote directory to the specified location on your local machine.

Another option is to mount the cluster filesystem on your machine using sshfs. This allows you to work on your projects from the cluster as if they were stored locally. Using it is also simple:

sshfs cc:/remote/path/ /mounting/point/

And when you are finished, unmount using:

umount ~/mounting/point/

note

umount may require root on linux systems.

Development Environment

The above file transfer methods and sshfs setup allow you to seamlessly use your local Neovim configuration and plugins on the cluster. If you prefer VS Code, you can use the Remote - SSH plugin. It has similar functionality to sshfs, but is notorious for misbehaving on the login nodes (see https://docs.alliancecan.ca/wiki/Visual_Studio_Code for how to configure it properly). Agustinus recommends Neovim, but your preferred development environment is ultimately up to you.

Jupyter Notebook may also be of interest for using python notebooks. Further reading can be found at https://docs.alliancecan.ca/wiki/JupyterNotebook

Helpful Links

Vector Compute Playbook

note

For further reading, check the official Vector compute playbook:

https://github.com/VectorInstitute/vec-playbook

DM Vector admins on Slack (Sebastian, Mark, Rob, etc.) for access.

Vector Internal Links

• User Guide to Killarney for Vector Researchers [PDF]
• https://support.vectorinstitute.ai/Killarney_QuickStart
  • You need to log in with your Vector Wi-Fi account. You should get or set this up during your onboarding at Vector.
• https://support.vectorinstitute.ai/Computing
  • The general computing wiki at Vector.

CCDB Public Links

• https://alliancecan.ca/en/services/advanced-research-computing/account-management/apply-account
• https://docs.alliancecan.ca/wiki/Getting_started
• https://docs.alliancecan.ca/wiki/Killarney
• https://docs.alliancecan.ca/wiki/Python
• https://docs.alliancecan.ca/wiki/AI_and_Machine_Learning
• https://docs.alliancecan.ca/wiki/Visual_Studio_Code
• https://docs.alliancecan.ca/wiki/JupyterNotebook