Skip to content

SCIPION

SCIPION uses software that needs Nvidia GPU acceleration. Appropriate hardware and driver are required.

Scipion plugins included in SBGrid (30+).

Note that this plugins are already linked to SBGrid available titles (capsules)*

  • aretomo
  • bsoft
  • chimerax
  • cistem
  • cryoassess
  • cryocare
  • cryodrgn
  • cryosparc (CryoSPARC installation not included in SBGrid, you need to link your own**)
  • deepfinder
  • dynamo
  • eman2
  • emantomo
  • emfacilities
  • emready
  • fidder
  • gapstop
  • gautomatch
  • gctf
  • imod
  • membrain
  • motioncorr
  • novactf
  • pwem
  • pyworkflowtests
  • relion
  • reliontomo
  • repic
  • tardis
  • sphire
  • tomo
  • tomosegmemtv
  • warp
  • topaz
  • xmipptomo (no longer supported by Scipion devs)
  • xmipp3

IMPORTANT NOTE: if you miss any plugin or a newer version, you can always request SBGrid team to install/update it, but DO NOT try to do it on your own, as it might not work as expected (most of plugin installs need a fine-tunning to work properly with SBGrid installed software, see next section for details).

*Note that if you are a commercial user, some of the titles linked to Scipion plugin may not be available for you (Sphire/CryOLO, Chimerax, etc), in that case, you will see the plugin in Scipion but you will not be able to run the protocols as the underneath title unfortunatly is not available for you on SBGrid.

**The scipion-em-cryoSPARC2 plugin is provided in Scipion, but CryoSPARC is not available in SBGrid, so you can not use CryoSPARC directly. However, if you want to use CryoSPARC in Scipion, link a CryoSPARC installation outside of SBGrid to the scipion-em-cryoSPARC2 plugin following the instructions on its GitHub repository (skip step 1). See next section for detailed information.

For more info on available, installed and version of the plugins, check the complete list of Scipion plugins: scipion installp --help

Scipion configuration

(<SBGRID_HOME>)/programs/x86_64-linux/scipion/<VERSION>/config contains configuration files for the Scipion installation provided by SBGrid, which will be applied to everyone using it. These files are modified by the SBGrid software team to ensure the smooth working of Scipion and all its plugins inside SBGrid. We do NOT recommend the user to modify them, as it can cause issues, and they ARE OVERWRITTEN any time a Scipion update is provided by SBGrid (including updates/hot fixes in the existing Scipion version). There are two files stored here: - scipion.conf with general variables to run scipion, install and use external libraries, etc. Also, all plugin-defined variables will be added for those plugins already installed. - hosts.conf with the host configuration. This is the general template provided by Scipion developers, generally not modified by SBGrid, however, we do NOT recommend modifying it, as, again, this will be overwritten with updates.

More information can be found on Scipion webpage

How to manage your personal preferences and configuration:

If you need to modify or add variables to your Scipion configuration to set your preferences, as paths for Scipion projects, test data, etc., you need to create your own config files (scipion.conf and hosts.conf) in your user home path (you need to create the path manually if it doesn't exist yet on your system): <USER_HOME>/.config/scipion/scipion.conf and <USER_HOME>/.config/scipion/hosts.conf. They will be read by Scipion if present and will overwrite the global Scipion configuration written there. This is particularly useful for queue system (as SLURM) and CryoSPARC configurations, as we can not provide them due to the paricularities of each of your systems. Do not hesitate to reach us at bugs@sbgrid.org if you need support on this.

EXAMPLES:

  • ~/.config/scipion/scipion.conf with preferred storage locations and cryosparc configuration (detailed info on how to configure your cryosparc install can be found on the scipion-em-cryosparc2 plugin page):
[PYWORKFLOW]
SCIPION_LOG=/shared/data/projects/my_user/ScipionUserData/logs/scipion.log
SCIPION_LOGS=/shared/data/projects/my_user/ScipionUserData/logs
SCIPION_TESTS=/shared/data/projects/my_user/ScipionTestData/data/tests
SCIPION_TESTS_OUTPUT=/shared/data/projects/my_user/ScipionUserData/Tests
SCIPION_USER_DATA=/shared/data/projects/my_user/ScipionUserData
CRYOSPARC_HOME = /cryosparc_home/shared/cryosparc/
CRYOSPARC_USER =my_user@institution.com
CRYOSPARC_DEFAULT_LANE = cryosparc-lane
CRYO_PROJECTS_DIR =/shared/data/projects/cs-projects/my_user/scipion_projects

Do not forget the [PYWORKFLOW] in the first row!

  • ~/.config/scipion/hosts.conf: template for SLURM integration. SLURM needs to be available in advance on your machine. You will need to customize your hosts.conf with the queues and resources set/available on your machine (that's why we can not provide a generic one for everyone). Other queue systems can be also integrated in Scipion. For more details visit Scipion webpage
[localhost]
SCIPION_USER_DATA=/shared/data/projects/ScipionUserData/
PARALLEL_COMMAND = mpirun -np %_(PROC_COUNT)d %_(COMMAND)s
NAME = my_slurm
MANDATORY = False
SUBMIT_COMMAND = sbatch %_(JOB_SCRIPT)s
CANCEL_COMMAND = scancel %_(JOB_ID)s
CHECK_COMMAND = squeue -h -j %_(JOB_ID)s
JOB_DONE_REGEX = ""
SUBMIT_PREFIX = scipion
SUBMIT_TEMPLATE = #!/bin/bash
  ### Inherit all current environment variables
  #SBATCH --export=ALL
  ## intentionally restrict multi-node slurm jobs
  #SBATCH --nodes=1
  ### Job name
  #SBATCH -J "%_(SCIPION_PROTOCOL)s_%_(JOB_NAME)s"
  ### Outputs
  #SBATCH -o %_(JOB_SCRIPT)s.out
  #SBATCH -e %_(JOB_SCRIPT)s.err
  #SBATCH --open-mode=append
  ### Partition (queue) name
  #SBATCH -p %_(JOB_QUEUE)s
  ### Specify time, number of nodes (tasks), cores and memory(MB) for your job
  ### cores =?= JOB_THREADS
  ### memory =?= JOB_MEMORY
  ### runtime limit =?= JOB_HOURS
  #SBATCH --ntasks=%_(PROC_COUNT)s --cpus-per-task=%_(JOB_THREADS)d --gres=gpu:%_(GPU_COUNT)s
  #printenv | grep -i conda
  #sbwhich python3
  echo "slurm id: ${SLURM_JOB_ID}"
  %_(JOB_COMMAND)s
QUEUES = {
  "gpu-small": [["GPU_COUNT","1", "number of GPUs" ],["PROC_COUNT",48,"number of processes"]],
  "gpu-medium": [["GPU_COUNT","4", "number of GPUs" ],["PROC_COUNT",48,"number of processes"]],
  "cpu-medium": [["GPU_COUNT","0", "number of GPUs" ],["PROC_COUNT",120,"number of processes"]]
  }

Running tutorials without write privileges

Scipion tutorials are launched by typing, for example scipion tutorial intro into the terminal. However, these tutorials will try and download the tutorial data into the $SCIPION_HOME/data/tests directory. Many SBGrid 'site' installations will have the /programs directory set as 'read and execute' only, with no write permissions in the /programs tree.

To be able to run Scipion tutorials without write privileges in the /programs tree, Scipion needs to have its configuration file modified to save tests in a directory with write privileges.

The configuration file is located at ~/.config/scipion/scipion.conf and should look similar to this: 

[DIRS_LOCAL]
SCIPION_USER_DATA = ~/ScipionUserData
SCIPION_LOGS = %(SCIPION_USER_DATA)s/logs
SCIPION_TMP = %(SCIPION_USER_DATA)s/tmp

[PACKAGES]
EM_ROOT = software/em
XMIPP_HOME = %(EM_ROOT)s/xmipp
CHIMERA_HOME = %(EM_ROOT)s/chimera-1.13.1
CHIMERA_HEADLESS_HOME = %(EM_ROOT)s/chimera_headless
VMD_HOME = %(EM_ROOT)s/vmd

[VARIABLES]
SCIPION_NOTES_PROGRAM =
SCIPION_NOTES_ARGS =
SCIPION_NOTES_FILE = notes.txt
SCIPION_NOTIFY = False
VIEWERS = {"Volume":["pyworkflow.em.viewers.viewers_data.DataViewer"]}

Open the file in your favourite editor, and add a line

SCIPION_TESTS = ~/scipion/tests

in the "DIRS_LOCAL" section. You can specify any directory you want the tutorials to be saved, provided you have write privileges in the directory.

Your modified ~/.config/scipion/scipion.conf file should now look similar to this: 

[DIRS_LOCAL]
SCIPION_USER_DATA = ~/ScipionUserData
SCIPION_LOGS = %(SCIPION_USER_DATA)s/logs
SCIPION_TMP = %(SCIPION_USER_DATA)s/tmp
SCIPION_TESTS = ~/scipion/tests

[PACKAGES]
EM_ROOT = software/em
XMIPP_HOME = %(EM_ROOT)s/xmipp
CHIMERA_HOME = %(EM_ROOT)s/chimera-1.13.1
CHIMERA_HEADLESS_HOME = %(EM_ROOT)s/chimera_headless
VMD_HOME = %(EM_ROOT)s/vmd

[VARIABLES]
SCIPION_NOTES_PROGRAM =
SCIPION_NOTES_ARGS =
SCIPION_NOTES_FILE = notes.txt
SCIPION_NOTIFY = False
VIEWERS = {"Volume":["pyworkflow.em.viewers.viewers_data.DataViewer"]}

After saving the file, running scipion tutorial intro will save the tutorial project in the ~/scipion/tests directory, and you should be able to follow the tutorials as described.