Relay

Relay is a web-based platform for cryo-EM data processing workflows. It provides a visual interface for building, running, and monitoring computational pipelines that process electron microscopy data — from raw micrographs through to 3D reconstructions.

Prerequisites

• .NET 10 SDK
• Python 3.10+ (for the Bakery visualization package)

Clone

git clone --recurse-submodules https://github.com/warpem/relay.git
cd relay

If you already cloned without --recurse-submodules:

git submodule update --init --recursive

Build

dotnet build

To publish a deployment using the installed .NET 10 runtime:

dotnet publish Relay -c Release -o publish/

The solution includes Relay.Runner, a standalone console supervisor for managed jobs. The standard build and publish automatically include its executable, DLL, and runtime files alongside Relay. Deploy the complete output directory; the runner needs no separate service or configuration. It depends only on the .NET runtime, with no web or scientific-library dependencies.

Install Bakery (visualization)

Bakery generates plots and thumbnails for job results. Install it into a Python environment:

pip install -e Bakery

Ensure the bakery command is on your PATH when running Relay.

Configure

Relay loads configuration from two sources:

1. Built-in defaults — Relay/appsettings.json (ships with the application)
2. Local overrides — relay.json in the working directory (your site-specific settings)

Authentication

Relay supports two authentication modes:

• native (default) — built-in username/password authentication, no external dependencies
• sso — OpenID Connect single sign-on with an external identity provider

To enable SSO, add to your relay.json:

{
  "Authentication": {
    "AuthenticationType": "sso"
  },
  "AuthService": {
    "Authority": "https://your-idp.example.com",
    "ClientId": "your-client-id"
  }
}

Run

From your project directory (where relay.json lives):

dotnet /path/to/publish/Relay.dll

Or during development:

dotnet run --project Relay

Relay will start on http://localhost:5001 by default.

Deploy (Linux server)

A management script is provided at scripts/relay.sh. It expects an environment file at ~/relay/relay.env (or the path in $RELAY_ENV_FILE).

Create ~/relay/relay.env:

RELAY_HOME="$HOME/relay"
RELAY_PORT=5001
RELAY_CERT_PATH="/path/to/cert.pfx"
RELAY_CERT_PASSWORD="your-password"

# Module system (if using Lmod)
LMOD_INIT="/path/to/lmod/init/bash"
CONDA_MODULE="miniconda3"
CONDA_ENV="relay"

# .NET and Warp native libraries
DOTNET_ROOT="/path/to/dotnet"
WARP_LIB_PATH="/path/to/warp/native/libs"

Then:

scripts/relay.sh start    # Start in background
scripts/relay.sh stop     # Stop gracefully
scripts/relay.sh status   # Check if running
scripts/relay.sh restart  # Stop + start

Configure a cluster queue

Queues are managed by an admin under Users → Queue configuration. Each queue has a name, a type (CPU, GPU, or Mixed), a scheduler, and several command templates.

Scheduler

The Scheduler setting tells Relay how to read job IDs and job states out of your commands' output. Pick Slurm, Lsf, Pbs, Sge or Flux to use the built-in parser for that scheduler, or Custom to supply your own patterns — the job ID regular expression and the pending/running/failed status patterns in the Advanced settings tab apply only to Custom queues, and are hidden otherwise.

Queues default to Slurm. Note that the parser no longer falls back across schedulers: a queue set to Slurm will reject output in another scheduler's format rather than guessing, so a non-SLURM queue must have its scheduler selected explicitly.

Managed queues (no external scheduler)

Setting Scheduler to Managed makes Relay run jobs itself, as local processes on the Relay host, instead of submitting them to SLURM, Flux or anything else. It starts a job only when the CPU cores, memory and GPUs it asks for are free. This suits a single workstation, where installing a scheduler purely to arbitrate one machine's resources is a large dependency for a small need.

Configure CPU cores, Memory (GB) and GPUs to describe the host. The submission script template is still used — for module blocks and {{ command }} — but drops the scheduler directives:

#!/bin/bash
{{ warp }}
ml warptools/latest
{{ /warp }}

umask 007

{{ command }}

Things worth knowing:

• Cores and memory are accounting, not enforcement. Relay will not start a job unless its declared requirements fit, but nothing stops a running job from exceeding them. GPUs are the exception: each job sees only its assigned devices via CUDA_VISIBLE_DEVICES.
• One managed queue per host. A second would double-book the same machine, so Relay refuses to create one or to switch an existing queue to Managed when one already exists. The rule is also applied to the queues loaded from the state file at startup, so a hand-edited or copied state carrying two does not slip past: the lowest-numbered managed queue keeps working, every other one is logged as an error and refuses to start jobs until you switch it to another scheduler or delete it. Resolving the duplication re-enables it immediately; no restart is needed.
• Managed payloads belong to Relay. A relay-runner supervisor stops its payload when Relay closes the ownership channel, including after a Relay crash. Restart marks the old managed attempt Interrupted and releases its reservation. It does not reconnect to the runner, act on saved process IDs, or hold admission waiting for the old process. This owner-bound contract relies on asynchronous EOF cleanup; it does not promise instantaneous cleanup or cleanup after simultaneous loss of both Relay and its supervisor.
• Pool workers require an external scheduler queue. A managed queue may run the pool manager, but worker groups currently support only scheduler-backed receipts.
• Admission is strict FIFO among dependency-ready jobs. Once the oldest eligible job is waiting for resources, later jobs cannot bypass it. Jobs still waiting for inputs do not block independent work.
• macOS is a development-only configuration. Without setsid, cleanup uses process-tree termination, which cannot reliably find descendants after their parent exits. Linux private process groups provide stronger containment.

Change note — 2D classification memory. Class2D now declares its real memory footprint — 16 GB per MPI worker rank, (processes - 1) x 16 GB — instead of the flat 16 GB it inherited before. A VDAM run with 8 processes therefore asks for 112 GB, which does not fit the default managed-queue total of 64 GB, so it is rejected outright with a message naming both figures rather than queued to wait for memory that will never appear. Raise Memory (GB) to match the host, or lower the job's process count.

The cores, memory and GPU totals can only be changed while the queue is idle. If jobs are still running on it, the edit is refused rather than leaving the accounting to disagree with what is on the host.

Execution state and restart

Queue definitions are stored in QueuesPath; active attempts are stored beside them in QueuesPath.executions. Each run gets a new attempt ID and a snapshot of its queue configuration. Job parameters and input connections remain fixed while an execution owns them. Clear or delete operations are rejected while active dependent jobs still use the job's files.

External scheduler jobs with saved receipts survive an ordinary Relay restart. Missing records or failed queries keep the last known state and show an observation warning in the queue view. Configure an active-status command and, where needed, a terminal-status command that can report completed jobs; an empty active listing alone cannot establish completion. Slurm has a built-in allocation-only sacct fallback. Queue edits affect future attempts, so validate these commands before submitting.

If submission reached a scheduler but its receipt was lost, Relay marks the attempt Interrupted and does not automatically submit another copy. Use {{ attempt_id }} in scheduler-visible metadata to help locate such jobs. The redesign does not import active runtime state from the old queue system; finish or stop that work before upgrading.

Changing pool size during a run

For a running pooled job, the queue card's Workers controls change the target for the current run. The minimum is one worker. Decreasing the target requests immediate cancellation of excess workers, including work in progress; those workers remain shown as stopping until the scheduler confirms exit. Increasing it starts additional workers within the run's remaining lifetime submission budget. A target that exceeds the attainable size is rejected. Resizing does not reset that budget or change the job's configured pool size for future runs.

MCP clients can call resize_job_pool(projectId, spaceId, jobId, desiredSize). The Pool field returned by get_job reports DesiredSize, Running, Pending, Stopping, Submitted, and CanResize.

Submission script template

The script template is a shell script submitted to your scheduler for each job. It uses {{ variable }} placeholders that Relay fills in at submission time:

Variable	Description
{{ command }}	The actual job command to run
{{ job_id }}	Relay's internal job identifier
{{ n_cores }}	CPU cores requested
{{ n_processes }}	MPI process count
{{ memory_gb }}	Memory in GB
{{ n_gpus }}	GPU count (GPU/Mixed queues)
{{ gpu_memory_gb }}	GPU memory in GB
{{ run_directory }}	Job working directory
{{ std_out }} / {{ std_err }}	Paths for stdout/stderr logs

Module blocks

Relay uses conditional blocks to load the right software modules depending on the job type. A block is only included in the script when that job type is being run:

{{ gpu }}
# included only for GPU jobs
{{ /gpu }}

{{ cpu }}
# included only for CPU jobs
{{ /cpu }}

{{ warp }}
# included only for Warp jobs
{{ /warp }}

{{ relion }}
# included only for RELION jobs
{{ /relion }}

{{ relion-pool }}
# included instead of {{ relion }} when a RELION job runs through the disk-based
# worker pool (CPU-only manager + CPU worker fleet). Load a RELION build that
# provides the relion_refine_pool binary. Requested by both the manager and the
# workers, alongside {{ cpu }} for the CPU partition directives.
{{ /relion-pool }}

{{ imod }}
# included only for IMOD jobs
{{ /imod }}

{{ aretomo2 }}
# included only for AreTomo2 jobs
{{ /aretomo2 }}

{{ missalignment }}
# included only for MisAlignment jobs
{{ /missalignment }}

{{ mpi }}
# included only for MPI-parallel jobs
{{ /mpi }}

Use these blocks to call your site's module system (e.g. ml modulename) so each job loads only what it needs.

A minimal SLURM example structure:

#!/bin/bash
#SBATCH -J {{ job_id }}
{{ gpu }}
#SBATCH -p <your-gpu-partition>
#SBATCH --gres=gpu:{{ n_gpus }}
{{ /gpu }}
{{ cpu }}
#SBATCH -p <your-cpu-partition>
{{ /cpu }}
#SBATCH -e {{ std_err }}
#SBATCH -o {{ std_out }}
#SBATCH --cpus-per-task {{ n_cores }}
#SBATCH --mem {{ memory_gb }}GB
#SBATCH --nodes 1
#SBATCH --ntasks-per-node {{ n_processes }}

{{ warp }}
ml warptools/latest
{{ /warp }}

{{ relion }}
ml relion/5.0
{{ /relion }}

# Preserve parent directory permissions for group members
umask 007

{{ command }}

Why umask 007: Without it, jobs create output files with default permissions that exclude the group, breaking access for other project members who share the same group. Setting umask 007 ensures files land as 660 and directories as 770, so the group can always read and write job outputs regardless of who submitted the job.

A minimal Flux example:

#!/bin/bash
#FLUX: --job-name={{ job_id }}
#FLUX: -N 1
#FLUX: -n 1
#FLUX: -x
#FLUX: --output={{ std_out }}
#FLUX: --error={{ std_err }}
{{ gpu }}
#FLUX: -g {{ n_gpus }}
{{ /gpu }}

{{ warp }}
ml warptools/latest
{{ /warp }}

umask 007

{{ command }}

Flux specifics worth knowing:

• All #FLUX: directives must be grouped at the top. Flux errors out on a directive that follows any non-blank line which isn't a comment, so module blocks that emit ml ... have to come after them. The blank lines left behind by stripped {{ gpu }} tags are fine.
• -g is GPUs per slot, not per job. With -n {{ n_processes }}, a 3-rank job asking for 1 GPU would request 3. Pairing -n 1 with -x (exclusive) sidesteps the arithmetic and gives each job the whole node — a good fit for single-node installations, at the cost of running jobs strictly one at a time.
• Flux's {{id}} mustache cannot be used in output paths, because Relay's own templating strips unrecognised {{ ... }} tags. Use {{ std_out }} / {{ std_err }}, which are already per-job absolute paths.
• Relay must be able to reach your Flux instance. A systemd-managed system instance works out of the box; a per-terminal flux start will not, since FLUX_URI won't be in the Relay service's environment.

Matching command templates:

Template	Value
Send command	{{ command }}
Submit job	flux batch {{ script_path_abs }}
Status job	flux jobs -no "{status}" {{ job_id }}
Abort job	flux cancel {{ job_id }} || true
List jobs (pools only)	flux jobs -no "{id.f58},{status}"
Cancel many jobs (pools only)	flux cancel {{ job_ids }} || true

Single braces such as {status} are Flux's own format fields and pass through Relay's templating untouched.

Command templates

Template	Variable	Purpose
Send command	{{ command }}	How to run a command on the cluster host, e.g. ssh user@host {{ command }}
Submit job	{{ script_path_abs }}	How to submit the generated script, e.g. sbatch {{ script_path_abs }}
Status job	{{ job_id }}	How to check a job's state; output must be parseable by Relay's status patterns
Abort job	{{ job_id }}	How to cancel a single job, e.g. scancel {{ job_id }}
List jobs (GPU worker pools only)	—	Lists running jobs as <id>,<state> one per line; use a comma separator (not quoted) so state survives a remote SSH hop
Cancel many jobs (GPU worker pools only)	{{ job_ids }}	Cancels a batch of jobs at once

Project structure

Directory	Description
Relay/	ASP.NET Blazor Server application (entry point)
Refund/	Core library: data model, job definitions, services
Bakery/	Python package for generating visualizations
Warp/	Submodule: Warp cryo-EM processing library
ElkSharp/	Submodule: graph layout engine for workflow diagrams
Emoji/	Submodule: Fluent UI emoji assets
scripts/	Deployment and management scripts

License

MIT License. See LICENSE for details.