Skip to content

Using SBGrid Programs on HPC Clusters with Apptainer

Overview

This document explains how members of the SBGrid Consortium can utilize Apptainer (formerly known as Singularity) to run the SBGrid software collection on High-Performance Computing (HPC) clusters.

Problem

Software titles in the SBGrid collection assume executables are available at /programs/ (either as a direct path, or by symbolic link). At HPC sites, it is sometimes not feasible to create this entry at the root filesystem for all compute nodes.

Approach

Use the SBGrid Installation Manager to install titles without the link, and use Apptainer's bind-mount feature to make it accessible to these titles when using them.

Prerequisites

  1. Apptainer available at your cluster
  2. Sufficient available space for the SBGrid titles being installed, and a minimal apptainer image
  3. Installation manager credentials

Notes on this documentation

Things that may need to be changed:

  • name of apptainer definition file, this documentation uses "sbgrid-a9-minimal.def"
  • name of apptainer container, this documentation uses "sbgrid-a9-minimal.sif"
  • any "module load" statements necessary for the HPC cluster for apptainer
  • SLURM partition names, and other SLURM parameters for GPU access at your cluster.

Things that will need to be changed:

  • path to the SBGrid installation, this documentation uses "/home/your_username/software/sbgrid"
  • SBGrid installation manager credentials, this documentation uses "your_site", "your_username", "your_key"
  • SBGrid titles and versions you'd like to install; this documentation uses "relion@5.0.1_cu12.8" (for an example GPU accelerated title) and "phenix@1.21-5207" (for an example CPU-only title)

Setup and Configuration

./sbgrid-cli activate your_site your_username your_key --target /home/your_username/software/sbgrid --no-link

Install SBGrid titles

./sbgrid-cli install relion@5.0.1_cu12.8 phenix@1.21-5207

This can be done outside the container if desired.

Build minimal container

Copy and paste the following into a text file called sbgrid-a9-minimal.def or download a copy :

BootStrap: docker
From: almalinux:9

%post
        yum install -y ncurses freetype fontconfig zlib tcsh rsync 
        yum install -y mesa-libGL which openssl libxcrypt libxcrypt-compat nss
        yum install -y libxcb xcb-util xcb-util-image xcb-util-keysyms xcb-util-renderutil xcb-util-wm xcb-util-cursor libnsl
        yum install -y glibc-langpack-en

%environment
        export LANG=en_US.UTF-8
        export LC_ALL=C

Build the minimal container:

apptainer build sbgrid-a9-minimal.sif sbgrid-a9-minimal.def

Usage

Interactive Usage

  1. Request interactive SLURM session

    srun --pty -p cpu-partition --mem=4G --time=00:30:00 /bin/bash

    For an interactive GPU session (if your cluster supports this):

    srun --pty -p gpu-partition --gres=gpu:1 --mem=4G --time=00:30:00 /bin/bash

  2. Within interactive session, once it has started

    apptainer shell --bind /home/your_username/software/sbgrid:/programs sbgrid-a9-minimal.sif

    For GPU support:

    apptainer shell --bind /home/your_username/software/sbgrid:/programs --nv sbgrid-a9-minimal.sif

  3. Within the container

    source /programs/sbgrid.shrc
sbinfo

    The output of the sbinfo command should have "Base path: /programs"

    Your ~/.bashrc (or tcsh equivalent) is typically used by apptainer shell and apptainer exec, so the environment can be automatically initialized.

Batch Usage

  1. Write or Update your SLURM script 

    #!/usr/bin/env bash

#SBATCH --job-name=ExampleBatchJob
#SBATCH --output=ExampleJob-%j.out
#SBATCH --error=ExampleJob-%j.err
#SBATCH --partition=CpuPartition
#SBATCH --nodes=1
#SBATCH --ntasks=5
#SBATCH --cpus-per-task=4

# initialize SBGrid environment
source /programs/sbgrid.shrc

# data processing commands 
sbinfo
sbwhich phenix
phenix.version

  2. Submit job

    sbatch example_job.sh