Config Forms#

We have used (or abused) the Form data type from EMhub for configuration purposes. Forms provide a convenient way to set an attribute that is a JSON string. This has been used to store some needed configurations for EMhub. As an internal convention, all configuration forms have the name config:ConfigName.

Bookings Config Form#

Basic configuration about Bookings’ Display

Form name: config:bookings#
1{
2    "display": {
3        "show_operator": true,
4        "show_application": false,
5        "show_experiment": false
6    },
7    "local_tag": "stjude"
8}
Parameters#

display

Displaying options for Bookings.

show_operator

Show or not the Operator’s Name when displaying booking info.

show_application

Show or not the Application’s Name when displaying booking info.

show_experiment

If true, there is an Experiment form associated to each booking.

local_tag

In the case of an EMhub instance with instruments from several facilities, this tag helps to mark the ‘local’ ones.

Project Config Form#

The following form specifies what types of entries ara available in the Project page and how they are arranged in the Entries menu.

Form name: config:projects#
 1{
 2    "entries_menu": [
 3        [
 4            "access_microscopes",
 5            "Request Microscope Access"
 6        ],
 7        [
 8            "note",
 9            "Note"
10        ]
11    ],
12    "entries": {
13        "grids_storage": {
14            "label": "Grids Storage",
15            "group": 1,
16            "iconClass": "fas fa-box fa-inverse",
17            "imageClass": "img--picture",
18            "report": "report_grids_storage.html"
19        },
20        "data_acquisition": {
21            "label": "Data Acquisition",
22            "group": 2,
23            "iconClass": "far fa-image fa-inverse",
24            "imageClass": "img--location",
25            "report": "sj_report_data_acquisition.html"
26        },
27        "note": {
28            "label": "Note",
29            "group": 3,
30            "iconClass": "fas fa-sticky-note fa-inverse",
31            "imageClass": "img--picture"
32        },
33        "access_microscopes": {
34            "label": "Microscope Request",
35            "group": 2,
36            "iconClass": "fas fa-search fa-inverse",
37            "imageClass": "img--location",
38            "report": "sj_report_access_microscopes.html"
39        },
40        "booking": {
41            "label": "Booking",
42            "group": 3,
43            "iconClass": "far fa-calendar",
44            "imageClass": "img--location"
45        }
46    }
47}
Parameters#

entries_menu

List of entries that will appear in the Projects Add Entry menu. Each item should be of the following:

[entry_key, ENTRY_LABEL]

entry_key should be from entries dict. Moreover, it should be another config form with the name entry_form::entry_key defining that entry parameters. For example, for access_microscopes entry, there is a form entry_form::access_microscopes.

[]

Adding a empty list will add a separator in the menu.

entries

Specify the list of entry’s configurations that can be used in the project’s menu. Each entry config should have the following options:

label

Label to be displayed.

group

Legacy param, not in use.

iconClass

Icon to be used for this entry. See Font Awesome (version xxx) for possible options.

imageClass

Class from Concept template, possible options are: img--picture, img--location

report

Report HTML template associated with this type of entry (TODO add reference to Developer Guide)

Sessions Config Form#

The following form shows the configuration for Sessions used at St.Jude. This form will likely differ in other places since the session’s logic changes from one center to another.

Form name: config:sessions#
 1{
 2    "acquisition": {
 3        "Krios01": {
 4            "voltage": 300,
 5            "magnification": 130000,
 6            "pixel_size": 0.6485,
 7            "dose": 1.09,
 8            "cs": 2.7,
 9            "images_pattern": "Images-Disc*/GridSquare_*/Data/Foil*fractions.tiff",
10            "gain_pattern": "*{microscope}*gain*.mrc"
11        },
12        "Krios02": {
13            "voltage": 300,
14            "magnification": 165000,
15            "pixel_size": 0.724,
16            "dose": 0.038,
17            "cs": 2.7,
18            "images_pattern": "Images-Disc*/GridSquare_*/Data/Foil*EER.eer",
19            "gain_pattern": "*EER_GainReference.gain"
20        },
21        "Arctica01": {
22            "voltage": 200,
23            "magnification": 130000,
24            "pixel_size": 0.63,
25            "dose": 1.063,
26            "cs": 2.7,
27            "images_pattern": "Images-Disc*/GridSquare_*/Data/Foil*fractions.tiff",
28            "gain_pattern": "*{microscope}*gain*.mrc"
29        }
30    },
31    "data": {
32        "gain": "/jude/facility/data/gains/*{microscope}*gain*.mrc",
33        "cryolo_models": "/jude/facility/data/cryolo_models/*/*.h5",
34        "images": ""
35    },
36    "raw": {
37        "root_frames": "/mnt/EPU_frames",
38        "root": "/research/cryo_core_raw",
39        "hosts": {
40            "Krios01": "workstation01.emhub.org",
41            "Arctica01": "workstation02.emhub.org"
42        }
43    },
44    "otf": {
45        "root": "/jude/facility/appdpcryoem",
46        "relion": {
47            "command": "/software/emhub-otf/scripts/relion-otf.sh {session_id} {otf_path}",
48            "options": {
49                "do_prep": "True",
50                "do_proc": "False",
51                "prep__do_at_most": "32",
52                "prep__importmovies__angpix": "{pixel_size}",
53                "prep__importmovies__kV": "{voltage}",
54                "prep__importmovies__Cs": "{cs}",
55                "prep__importmovies__fn_in_raw": "data/Images-Disc1/GridSquare_*/Data/FoilHole_*_fractions.tiff",
56                "prep__importmovies__is_multiframe": "True",
57                "prep__motioncorr__do_own_motioncor": "False",
58                "prep__motioncorr__fn_motioncor2_exe": "/software/scipion/EM/motioncor2-1.5.0/bin/motioncor2",
59                "prep__motioncorr__dose_per_frame": "1.00",
60                "prep__motioncorr__do_save_noDW": "False",
61                "prep__motioncorr__do_save_ps": "False",
62                "prep__motioncorr__do_float16": "False",
63                "prep__motioncorr__fn_gain_ref": "./gain.mrc",
64                "prep__motioncorr__bin_factor": "1",
65                "prep__motioncorr__gpu_ids": "0:1:3:4",
66                "prep__motioncorr__nr_mpi": "4",
67                "prep__motioncorr__nr_threads": "1",
68                "prep__motioncorr__patch_x": "7",
69                "prep__motioncorr__patch_y": "5",
70                "prep__motioncorr__other_args": "--skip_logfile --do_at_most 32",
71                "prep__ctffind__fn_ctffind_exe": "/software/scipion/EM/ctffind4-4.1.13/bin/ctffind",
72                "prep__ctffind__nr_mpi": "8",
73                "prep__ctffind__use_given_ps": "False",
74                "prep__ctffind__use_noDW": "False"
75            }
76        },
77        "scipion": {
78            "command": "/software/emhub-otf/scripts/scipion-otf.sh {session_id} {otf_path}",
79            "options": {}
80        },
81        "hosts": [
82            "workstation01.emhub.org",
83            "workstation02.emhub.org"
84        ],
85        "hosts_default": {
86            "Krios01": "workstation01.emhub.org",
87            "Arctica01": "workstation02.emhub.org"
88        },
89        "workflows": [
90            "scipion",
91            "relion",
92            "none"
93        ],
94        "workflow_default": "scipion"
95    }
96}
Parameters#

acquisition

Specify acquisition parameters for each of the Microscopes. The key should be the Resource’s Name of the Microscope.

data

Specify some data locations. For example, data[‘gain’] should point to the pattern of where the gain references are. The newest one will be used for the OTF.

raw

Configuration of raw data locations and workers.

root_frames

Where raw frames will be written from the camera PC.

root

Root of offloading server where frames might be moved.

hosts

Hosts that will take care of the transfer for each microscope.

otf

Options related to the on-the-fly processing.

root

Where OTF folder for each session will be created.

relion

Options for the Relion pipeline.

scipion

Options for the Scipion pipeline.

hosts

Host options to launch the OTF (usually available only to managers).

hosts_defaults

What is the default option for each microscope.

workflow

Workflow options to choose from.

Permissions Config Form#

The following form defines what user’s roles have permissions to perform a given action. Each key will be an action and the options are specified for tags. In the following example there are two groups of permissions, one for tag microscope and the other for tag prep. In this case, only admin or manager can create bookings for microscopes but any user can create bookings for prep tagged instruments.

Form name: config:permissions#
 1{
 2    "create_booking": {
 3        "microscope": ["manager", "admin"],
 4        "prep": ["user"]
 5    },
 6    "delete_booking": {
 7        "microscope": ["manager", "admin"],
 8        "prep": ["user"]
 9    },
10    "create_session": ["manager", "admin"],
11    "content": {
12        "usage_report": ["manager", "admin"],
13        "raw": ["admin"]
14    }
15}
Parameters#

create_booking

Define what user roles can create bookings, based on resource tags.

delete_booking

Define what user roles can delete bookings, based on resource tags.

create_session

Define permissions for session creation.

Hosts Config Form#

This form is used to define a list of worker hosts that are allowed to connect with EMhub.

An alias should be defined for each hosts. After a worker is connected, it notifies the EMhub server of the hardware configuration in the worker machine (e.g. available CPU cores, RAM memory, and GPUs). See below the initial configuration and after workers have connected and sent some info.

Form name: config:hosts#
1{
2    "workstation01.emhub.org": {
3        "alias": "l1",
4    },
5    "workstation02.emhub.org": {
6        "alias": "l2",
7    }
8}
Form name: config:hosts#
 1{
 2    "workstation01.emhub.org": {
 3        "alias": "l1",
 4        "updated": "2023-07-24 12:09:29",
 5        "specs": {
 6            "CPUs": 72,
 7            "GPUs": {
 8                "NVIDIA GeForce GTX 1080 Ti": {
 9                    "count": 4,
10                    "memory": "11178 MiB"
11                }
12            }
13        }
14    },
15    "workstation02.emhub.org": {
16        "alias": "l2",
17        "updated": "2023-07-24 12:09:23",
18        "specs": {
19            "CPUs": 72,
20            "GPUs": {
21                "NVIDIA GeForce GTX 1080 Ti": {
22                    "count": 4,
23                    "memory": "11264 MiB"
24                }
25            }
26        }
27    }
28}