Supported Programs ================== Program Launchers ----------------- In **emwrap**, external programs can be defined by specifying "program launchers". The idea of the launcher is to create a bash script that wraps the program call and sets up the necessary environment. For example, the launcher can load cluster modules, source bash files, or set up environment variables. In that way, the code from **emwrap** just needs to call the launcher without taking care of local installation details. Jobs implemented in **emwrap** itself are launched via ``emh-tomo --launch MODULE`` (configured as the default ``EMWRAP`` launcher in *EMWRAP_CONFIG*, typically ``$ROOT/emh-tomo --launch``). Submitting jobs from the Relion GUI uses the ``relion_dispatcher`` entry in *EMWRAP_CONFIG* (typically ``$ROOT/emh-tomo --submit``). There is a section in the *EMWRAP_CONFIG* variable related to the launchers: .. code-block:: json "programs": { "EMWRAP": {"launcher": "$ROOT/emh-tomo --launch"}, "WARP": {"launcher": "$SCRIPTS/warp_launcher.sh"}, "PYTOM": {"launcher": "$SCRIPTS/pytom_launcher.sh"}, "RELION": {"launcher": "$SCRIPTS/relion_launcher.sh"}, "MOTIONCOR2": {"launcher": "$SCRIPTS/motioncor2.sh"}, "MOTIONCOR3": {"launcher": "$SCRIPTS/motioncor3.sh"}, "ARETOMO2": {"launcher": "$SCRIPTS/aretomo2.sh"}, "ARETOMO3": {"launcher": "$SCRIPTS/aretomo3.sh"}, "IMOD": {"launcher": "$SCRIPTS/imod_launcher.sh"}, "CTFFIND": {"launcher": "$SCRIPTS/ctffind5.sh", "version": 5}, "CRYOCARE": {"launcher": "$SCRIPTS/cryocare_launcher.sh"}, "DENOISET": {"launcher": "$SCRIPTS/denoiset_launcher.sh"}, "MISSALIGNMENT": {"launcher": "$SCRIPTS/missalignment.sh"} } After the installation, there is a *scripts* folder that is created with some of the launcher scripts, but YOU MIGHT NEED TO MODIFY them to work in your environment. In the following sections, there are some examples of launchers. Warp Launcher ~~~~~~~~~~~~~ In the following example, Warp is loaded from the available modules, together with Aretomo2, version 1.0.0. .. code-block:: bash #!/bin/bash PROGRAM=$1 shift export MODULES="warp/2.0dev33-latest aretomo2/1.0.0" echo Loading modules $MODULES module load -s $MODULES $PROGRAM $@ Or, if we are loading Warp from an SBGrid installation, the launcher could be something like: .. code-block:: bash #!/bin/bash PROGRAM=$1 shift export SBGRID=/programs/sbgrid.shrc source $SBGRID echo "Loading Warp from SBGrid file: ${SBGRID}." $PROGRAM $@ Relion Launcher ~~~~~~~~~~~~~~~ In the case of the Relion launcher, the first argument is the program name, and the second is the number of MPI processes. The wrapper will take care of adding the *_mpi* suffix to the program and also the *mpirun* command. For example: .. code-block:: bash #!/bin/bash export SBGRID=/programs/sbgrid.shrc source $SBGRID echo "Loading Relion from SBGrid file: ${SBGRID}." export PROGRAM=$1 shift export MPI=$1 shift if [ "$MPI" -eq 1 ]; then export CMD="${PROGRAM} $@" else export CMD="mpirun.relion --oversubscribe -np ${MPI} ${PROGRAM}_mpi $@" fi echo Running command: ${CMD} $CMD Other Launchers ~~~~~~~~~~~~~~~ **emwrap** is still under development, and more tools will be integrated in the future. Right now, apart from Warp and Relion, it might be helpful to configure the following launchers: * PyTOM launcher: for 3D template matching * IMOD launcher: for etomo tilt-series alignment .. _supported-programs: Supported Programs ------------------ The table below lists the external programs that **EMhub-Tomo** runs under the hood, the version set in the launcher templates shipped with **emwrap** (``emwrap/config/scripts``), and where to find installation instructions. These are the versions used for development and testing. Other versions may work, but they have not been tested. Most programs are also available through `SBGrid `_, which is how several of the example launchers load them. .. list-table:: :header-rows: 1 :widths: 16 16 22 20 26 * - Program - Config key - Launcher template - Tested version - Installation * - Warp / WarpTools / M - ``WARP`` - ``warp_launcher.sh`` - 2.0.0dev36 - `Warp installation `_ * - RELION - ``RELION`` - ``relion_launcher.sh`` - 5.1.0-beta (CUDA 13.0) - `RELION installation `_ * - pytom-match-pick - ``PYTOM`` - ``pytom_launcher.sh`` - 0.13.2 - `pytom-match-pick docs `_ * - AreTomo2 - ``ARETOMO2`` - ``aretomo2.sh`` - 1.0.0 - `AreTomo2 on GitHub `_ * - AreTomo3 - ``ARETOMO3`` - ``aretomo3.sh`` - 2.2.8 - `AreTomo3 on GitHub `_ * - IMOD - ``IMOD`` - ``imod_launcher.sh`` - 5.1.9 - `IMOD download `_ * - DenoisET - ``DENOISET`` - ``denoiset.sh`` - conda env ``denoiset`` - `DenoisET on GitHub `_ * - MissAlignment - ``MISSALIGNMENT`` - ``missalignment.sh`` - conda env ``miss-alignment`` - `MissAlignment on GitHub `_ .. note:: The Warp tilt-series alignment job (``emw-warp-tsalign``) also calls AreTomo2/AreTomo3 or IMOD, depending on the alignment method selected, so those launchers must work for the Warp pipeline too. .. _supported-jobs: Supported Jobs -------------- The job types currently implemented in **emwrap** for CryoET processing, grouped by the external package they wrap. This list is generated from ``ProcessingConfig.get_jobs()`` and ``ProcessingConfig.get_packages()`` (in ``emwrap/base/config.py``); every job listed here ships with a matching form under ``emwrap/config/forms`` -- see :doc:`job_forms` for how those forms are structured. .. _Warp: https://warpem.github.io/ .. _Relion: https://relion.readthedocs.io/en/latest/STA_tutorial/Introduction.html .. _PyTOM: https://github.com/SBC-Utrecht/pytom-match-pick .. _AreTomo: https://github.com/czimaginginstitute/AreTomo3 emwrap ~~~~~~ General-purpose jobs that are not specific to any external package. .. list-table:: :header-rows: 1 :widths: 30 40 20 :width: 100% * - Job - Label - Output * - ``emw-import-ts`` - EMwrap Import TS - EMwrap * - ``emw-subset-ts`` - EMwrap Subset TS - EMwrap * - ``emw-merge-sets`` - EMwrap Merge Sets - EMwrap * - ``emw-missalignment`` - MissAlignment - MissAlignment Warp ~~~~ Jobs wrapping `Warp`_. .. list-table:: :header-rows: 1 :widths: 30 40 20 :width: 100% * - Job - Label - Output * - ``emw-warp-mctf`` - Warp Motion and CTF - WarpMctf * - ``emw-warp-tsalign`` - Warp Tilt Series Alignment - WarpTsAlign * - ``emw-warp-ctfrec`` - Warp CTF and Reconstruction - WarpCtfRec * - ``emw-warp-otf`` - Warp OTF Preprocessing - WarpOtf * - ``emw-warp-export_particles`` - Warp Export Particles - WarpExportParticles * - ``emw-warp-mtools_create`` - Warp M Create Population - WarpMCreatePopulation * - ``emw-warp-mcore`` - Warp MCore Refine - WarpMCore * - ``emw-warp-estimate_weights`` - Warp Estimate Weights - WarpEstimateWeights * - ``emw-warp-mtools_resample`` - Warp M Resample - WarpMResample Relion ~~~~~~ Jobs wrapping native `Relion`_ programs, plus a couple of thin Relion-adjacent utilities. .. list-table:: :header-rows: 1 :widths: 30 40 20 :width: 100% * - Job - Label - Output * - ``relion.reconstructtomograms`` - Relion Reconstruct Tomograms - Tomograms * - ``relion.pseudosubtomo`` - Relion Extract Subtomograms - Extract * - ``relion.reconstructparticletomo`` - Relion Tomo Reconstruct Particles - Reconstruct * - ``relion.initialmodel.tomo`` - Relion Tomo Initial Volume - InitialModel * - ``relion.class3d.tomo`` - Relion Tomo 3D Classification - Class3D * - ``relion.refine3d.tomo`` - Relion Tomo Refine - Refine3D * - ``emw-relion-symmetrize_volume`` - *(none -- falls back to the job id)* - -- * - ``emw-relion-mask_create`` - *(none -- falls back to the job id)* - -- PyTOM ~~~~~ Jobs wrapping `PyTOM`_ for 3D template matching. .. list-table:: :header-rows: 1 :widths: 30 40 20 :width: 100% * - Job - Label - Output * - ``emw-pytom-create_template`` - PyTom Create Template - PyTom * - ``emw-pytom`` - PyTom Template Matching - PyTom AreTomo ~~~~~~~ Jobs wrapping `AreTomo`_. .. list-table:: :header-rows: 1 :widths: 30 40 20 :width: 100% * - Job - Label - Output * - ``emw-aretomo3`` - Aretomo3 - Aretomo3 * - ``emw-denoiset`` - DenoisET Pipeline - DenoisET