MacOS/Linux Command Line Installer
SBGrid Installation Manager CLI Installation and Usage
sbgrid-cli is the command line installation manager application for the SBGrid software collection.
The SBGrid installation manager is available in graphical and command-line flavors for macOS and Linux. These can be interchanged for the same installation.
Download
You can find all versions including GUI and CLI on our download page.
Latest sbgrid-cli download
Download sbgrid-cli for Linux Download sbgrid-cli for macOS
Usage
This page covers the usage of the installation manager graphical user interface (CLI). For usage information on the latest graphical interface (GUI), see here.
The SBGrid installation manager requires an SBGrid Consortium user account. Set up an account by registering here : SBGrid registration
Requirements and recommendations
sbgrid-cli is single binary executable and should work on most modern Linux distributions.
The SBGrid installation manager will install the software collection to /opt/sbgrid by default (though this can be changed). A symlink from /programs to the installation directory is required and this step does require admin priveleges. Admin privileges are not required to run the applications. This is only for the initial installation, specifically the creation of /opt/sbgrid (if desired) and for /programs (required).
The sbgrid-cli install manager uses 'sudo' for priveleged installation tasks. As such, it should be used from an account with sudo privileges if this functionality is desired. Alternatively, software installation can be completed without sudo and the /programs link can be added manually later. The software will not be functional without this path.
While not strictly required, we recommend a dedicated 'sbgrid' user account for software management, particularly on multiuser systems. This ensures the software can't be modified by user accounts and that data won't be mistakenly saved into the software stack and lost inadvertantly during software updates. The sbgrid-cli program will add and remove software. If user data files are accidentally saved into the software collection, they are likely to be lost.
Apple Silicon macs will need to install Rosetta2 to use sbgrid-cli and most of the SBGrid software collection. Rosetta2 can be installed with this command:
Installation
The sbgrid-cli program is a single executable and does not have any dependencies or requirements.
Untar the archive on the command line with :
where the "archive file" is the name of the tar.gz file downloaded from the links above.
MAC USERS PLEASE NOTE
If you encounter a pop-up warning about an "unidentified developer", you need to strip apple's quarantine extended attribute from the sbgrid-cli executable. This is easy to do.
First, cancel the dialog and then run this in your terminal:
You should then be able to run the application normally without warnings.
You can use curl to download the tarball directly in your terminal. This approach has the added benefit of not triggering the macOS "unidentified developer" Gatekeeper response which requires the xattr step below.
Untar and run the application to test.
General Usage
The SBGrid command line installation manager takes subcommands and arguments to those commands to perform various tasks. These can be found in the usage info shown above. Each subcommand can also take a -h or --help argument to show more detailed usage info. For example :
will produce the help info for the install subcommand.
Activation
To install the software, you must activate your installation with valid credentials provided by SBGrid. This process is only required for the initial installation of the software and is not required to add/remove/upgrade titles.
Activate with the activate subcommand
$ sbgrid-cli activate --help
SBGrid Installer 2.2.9 f593932a17 —
_______ _____ _ __
/ __/ _ )/ ___/___(_)__/ /
_\ \/ _ / (_ / __/ / _ /
/___/____/\___/_/ /_/\_,_/
C o n s o r t i u m
USAGE — activate
▸ sbgrid-cli activate [site] [user] [key] [OPTIONS...]
ARGUMENTS
[site] The installation site provided in your activation
credentials
[user] The username provided in your activation credentials
[key] The activation key provided in your activation
credentials
OPTIONS
-k, --credential-key <key> Activate with a single credential key
--darwin Specify macOS as a platform
--no-link No symlink creation, no sudo prompt.
--linux Specify Linux as a platform
--skip-folder-checking Skips checking if a folder with programs already exists
-t, --target <target> Specify a custom installation target
Some useful options here are :
--no-link
skips creatation of a symlink at /programs. Useful for installs without sudo
--skip-folder-checking
skips the check for software that is already installed. Will overwrite existing installations.
--target
Set the install directory to a different path than /opt/sbgrid
Trouble activating? See Troubleshooting options further down this page.
Restoring from a saved config using reactivate
reactivate subcommand can restore from a saved config file. This config includes all titles and versions from a saved installation. Useful for moving installations or reproducing installations.
Migrating existing accounts
The sbgrid-cli feature allows a user to change between SBGrid accounts. This can be useful in a few circumstances:
- Moving between SBGrid member labs
- To change the ownership of an existing installation
- To change from legacy SBGrid account to a new account which allows installation of BioGrids (migration guide here)
The process is does not require reinstallation, downloading new software or extensive changes to the filesystem (and should be fast).
$ sbgrid-cli migrate-account [site] [user] [key] [OPTIONS...]
ARGUMENTS
[site] The installation site provided with your new account
credentials
[user] The username provided with your new account credentials
[key] The activation key provided with your new account
credentials
For example, to move to a new account with creditials site: harvard_123i user:bjones key:2MHNCPpGreyJi48v0DaeYjH6l5duRNyO , you would use:
Alternatively, the command can be run as sbgrid-cli migrate account and the program will prompt for new site, user, key info on the command line.
Listing available titles, status
The sbgrid-cli list command will show all available titles. They are normally colored by their installation status.
Available options :
list
-a, --all-versions List all available versions for a title
-c, --collections List the available software collections
-d, --default-versions List only the default version for a title
--long-format Use the long format to display the title list
--one-column Format the output into a single column
The list subcommand has several options
USAGE — list
▸ sbgrid-cli list [title...] [OPTIONS...]
ARGUMENTS
[title...] The names of the software titles to list
OPTIONS
-a, --all-versions List all available versions for a title
-c, --collections List the available software collections
--darwin Specify macOS as a platform
-d, --default-versions List only the default version for a title
--linux Specify Linux as a platform
--long-format Use the long format to display the title list
--one-column Format the output into a single column
Note --collections can be used to see subsets of software as listed at https://sbgrid.org/software/.
updates
sbgrid-cli updates Shows available updated titles and versions
$ sbgrid-cli updates
------------------------------------------------------------------------------------
info: Retrieving package metadata...
The following installed packages have updates available (x86_64-linux):
- cryodrgn@0.3.0b => 0.3.1 (new default)
- imod@4.11.0 (patch for installed default version)
- ligplotplus@2.2 (patch for installed default version)
- r@3.6.2 (patch for installed version)
Updates are either new default versions (new default), or updates to an existing version (patch for installed version). The patch versions are usually bug or configuration fixes.
obsolete
Shows obsolete versions that will be removed. Obsolete versions are previously installed versions no longer included in the SBGrid installation.
Getting Information about titles
info
The info subcommand shows information about each title, including a description and links to documentation can be show with the sbgrid-cli info <title> command. See example below.
For example, info for RELION :
$ sbgrid-cli info relion
------------------------------------------------------------------------------------
Package information for relion (x86_64-linux):
Package: RELION
Name slug: relion
Architecture: x86_64-linux
Available version(s):
* 1.3 (installed) depends on: openmpi (1.8.4)
* 1.4 (installed) depends on: openmpi (1.8.4)
* 1.4b (installed) depends on: openmpi (1.8.4)
* 1.4-randomphase3d (installed) depends on: openmpi (1.8.4)
* 2.1_cu8.0 (installed) depends on: cuda (8.0), openmpi (2.1.2)
* 3.0.4_cu9.0 (installed) depends on: cuda (9.0), openmpi (2.1.2)
* 3.0.8 (installed) depends on: openmpi (2.1.3)
* 3.0.8_cu10.1 (installed) depends on: cuda (10.1), openmpi (2.1.2)
* 3.0.8_cu8.0 (installed) depends on: cuda (8.0), openmpi (2.1.2)
* 3.0.8_cu8.0_slurm20 (installed) depends on: cuda (8.0.44), openmpi (3.1.6_slurm-20.02.03)
* 3.0.8_cu9.0 (installed) depends on: cuda (9.0), openmpi (2.1.2)
* 3.0.8_cu9.2 (installed) depends on: cuda (9.2), openmpi (2.1.2)
* 3.1.0 (installed) depends on: openmpi (3.1.6)
* 3.1.0_cu10.2 (installed) depends on: cuda (10.2.89), openmpi (3.1.6)
* 3.1.0_cu8.0 (installed) depends on: cuda (8.0.44), openmpi (3.1.6_compat)
* 3.1.0_cu8.0_compat (installed) depends on: cuda (8.0.44), openmpi (3.1.6_compat)
* 3.1.0_cu8.0_slurm20 (installed) depends on: cuda (8.0.44), openmpi (3.1.6_slurm-20.02.03)
* 3.1.0_cu9.2 (installed) depends on: cuda (9.2.88), openmpi (3.1.6)
* 3.1.1 (installed) depends on: openmpi (3.1.6)
* 3.1.1_cu9.2 (installed, default) depends on: cuda (9.2.88), openmpi (3.1.6)
Grid(s): sbgrid
Collection(s): Electron Microscopy
Description:
(REgularised LIkelihood OptimisatioN) a stand-alone computer program for Maximum A Posteriori (MAP) refinement of (multiple) 3D reconstructions or 2D
class averages in cryo-electron microscopy.
Technical notes:
SBGrid Usage Info RELION versions > 2.0 are GPU-accelerated using Nvidia CUDA on Linux. For general information on running GPU accelerated applications from SBGrid, please see here : https://sbgrid.org/wiki/gpuCUDA builds of RELION re designated by a '_cu' suffix in the version of the application. For example, 3.1.0_cu9.2 is linked against CUDA v9.2 libraries. All builds are single precision on the GPU and can be run on 'consumer-grade' GPU hardware.RELION builds without CUDA support are compiled with Intel 2019.5 with accelation options and Intel math libraries. RELION and MPI versionsRELION uses the 'mpirun' executable to manage MPI "ranks" for parallel processing. SBGrid includes several OpenMPI versions which are matched to specific RELION versions. When called from the SBGrid capsule environment, the relion GUI is configured to use the correct version of OpenMPI to manage parallel processing. When calling RELION binaries from the command line, the `mpirun.relion` executable can be used. `mpirun.relion` is configured to used the correct version of OpenMPI for each version of RELION. The 3.1.* versions for RELION use the default configuration of OpenMPI (3.1.6 on Linux, 2.1.3 on macOS) and do not require the version to be defined. Older releases use OpenMPI 2.1.2 which should be specified in $HOME/.sbgrid.conf. To set the OpenMPI version, add : OPENMPI_X=2.1.2"to $HOME/.sbgrid.conf and open a new terminal to use the correct mpirun. As noted above, you can also use the `mpirun.relion` executable in place of `mpirun` to use the MPI that matches the currently configured version of RELION. In this case, no other configuration is needed.Version 3.1.0_cu8.0 is compiled to support legacy CPUs (pre-Haswell). OpenMPI 3.1.6_compat is also needed for this version.Versions 3.1.0_cu8.0_slurm20 3.1-beta_cu8.0_slurm20 and 3.0.8_cu8.0_slurm20 support Slurm v20 PMI library. Recent Changes2.2.9111 : Add 3.1.1 RELION versions20200703 : Add 3.1.0 release version20200629 : reconfigure RELION gui to automatically use correct 'mpirun' in capsules.20200610 : rebuild of 3.1-beta releases with OpenMPI 3.1.6 from commit 5997001.
Known Issues20200610 : Empty values in GUI during autopicking can be interpreted as Text instead of Float values. Deprecated versionsThe following verions are deprecated and will be removed in a future SBGrid update.20200706: 3.0.8_cu10.1 3.0.8_cu9.2 3.0.8_avx2 3.1-beta_cu9.2 3.1-beta_cu10.1
(c) Copyright 2008, Structural Biology Grid (www.sbgrid.org) Last Update to this document# Jason Key 0a6b896 2020-11-15 18:39:38 -0500
Links:
Website: http://www2.mrc-lmb.cam.ac.uk/relion/index.php/Main_Page
Manual: http://www2.mrc-lmb.cam.ac.uk/relion/index.php/Main_Page
Forumhelp: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=CCPEM
The info subcommand includes all version info, dependencies ( which are automatically installed), technical notes and relevant links.
Installing and removing
Install and remove titles with the sbgrid-cli install <title> and sbgrid-cli remove <title> command. Each of these commands can take multiple titles with versions at one time.
install
sbgrid-cli install [title...] [OPTIONS...]
ARGUMENTS
[title...] The name(s) of the software title(s) to install. To
specify a non-default version, append @ and the version
number to the name of the title (e.g. xyz@1.2.3).
OPTIONS
--all-titles Installs all the available titles
--all-versions This will install all the available versions for a
specified title or collection
-c, --collection <title> The title of a software collection to install
--darwin Specify macOS as a platform
--linux Specify Linux as a platform
-j, --num-workers <num-workers> Run installation tasks in parallel
number, default: 1
--reinstall Force the reinstallation of already installed packages
Installation Examples
Install a single default version of an application, adxv :
$ sbgrid-cli install adxv
- Install adxv@1.9.14 (x86_64-linux)
- Installation setup (x86_64-linux).
Continue? … yes
info: (1/2) Installing adxv@1.9.14 (x86_64-linux)...
info: Installation complete for adxv@1.9.14 (x86_64-linux)
info: (2/2) Setting up your installation...
$
Install a specific version of a adxv
Install all versions of adxv
Install the 'Electron Microscopy' collection of titles (see above list command to list collections)
Install all default versions of all titles
Install all versions of all titles ( The full collection)
Note the --reinstall flag can be added to all commands to reinstall titles that have already been installed.
remove
Same options as install subcommand above.
Updating titles
Updates are performed when default titles change to a new version or bugs are fixed in existing verisons. To update, use the update subcommand.
update
Remove obsolete titles
Use the clean subcommand to list or remove obsolete versions or titles
clean
Automated installation and updates
sbgrid-cli can be configured to update all installed versions of software. To automatically check for updates for installed versions, add this line to the user crontab file using crontab -e:
*/15 * * * * /programs/x86_64-linux/sbgrid_installer/latest/sbgrid-cli update --all-titles --yes --quiet
This will check for updates every 15 minutes and update installed versions automatically when updates are available.
sbgrid-cli can be used in an automated method to periodically install all new titles, all new versions of an updated title, and remove obsolete versions no longer included in the collection. This method is similar to the sbgrid-admin script used for SBGrid site installations, but with several notable benefits. sbgrid-cli allows rolling updates to the software collection so new titles are added continuously. Only updates are installed so data transfers are quick and efficient.
For automated installation of all versions of all titles, this command should be added to crontab for the sbgrid software management account:
To only install default versions ( older versions will not be installed but are retained ), use :
To include software for macOS on a linux host ( ideal for NFS shares )
To automate the process the commands above can be added to the crontab of the 'sbgrid' user account.
This can be done with the sbgrid-cli crontab command (see below).
For example, the crontab configuration line below will check for new versions every 15 minutes.
Run periodic installation, update and removal. Installs all new titles, updates where appropriate and removes obsolete versions
For example running
will install all default, update all installed titles and remove all obsolete versions while
will install all versions of all titles, add any new titles, update any pending updates, and remove obsolete versions.
The sbgrid-cli admin functionality replaces the legacy sbgrid-admin installation script. For a tutorial on setup and migration, please see this page.
crontab
Adds a line to the user crontab to configure periodic updates.
The default crontab configuration is to check for new titles every 15 minutes:
but can be modified with the following options.
--all-versions This will install all the available versions for all
the available titles
boolean
--darwin Specify macOS as a platform
--linux Specify Linux as a platform
-j, --num-workers <num-workers> Run installation & update tasks in parallel
number, default: 4
For more info on cron, see https://www.man7.org/linux/man-pages/man5/crontab.5.html
Rebuild an installation
rebuild
rebuild rebuilds the installation environment configuration files. Usually this step is automatic but can be run manually for troubleshooting.
save
The save subcommand saves the current configuration so it can be restored at a later stage or replicated on another machine or in the cloud.