[accumb3ns]

Why

Bioinformatics launchers (e.g. the RNA-Seq launcher) orchestrate work: read a samplesheet, submit one child workflow per sample, wait, then gather. Historically they ran wrapped in CWL so Seven Bridges or Arvados could execute them as a platform task. In AWS we decided not to add a CWL runner layer — the launcher image will run directly as an AWS Batch job, with CloudWatch as the log interface — and not to give the launcher a web server or status endpoint.

That removes the platform "task" record the launcher used to get for free. Nothing recorded that a launcher execution existed, and nothing linked it to the child runs it created. This PR adds that record, so launcher status and progress are derived from workflow rows this service already keeps. It is also the prerequisite for launcher restart/recovery.

Everything here is additive: no existing endpoint, schema, or callback changes shape.

What changed

A launcher execution is a WorkflowRun, not a new table — workflow_engine=awsbatch, workflow_run_id = the Batch jobId. It reuses the state machine, listing, filtering, callbacks, the client, the CLI, and the frontend's run views.

Lineage rides in as a reserved ParentRunId tag, promoted by create_run into an indexed parent_run_id column — the pattern already used for ProjectId and TaskName. POST /runs stays GA4GH-shaped, and ?filters={"parent_run_id":"…"} works through the existing generic filter with no server change. That listing is also how a restarted launcher rediscovers work it already submitted.

Parent state and child progress are deliberately separate. GET /runs/{run_id}/progress returns the launcher's own state plus a GROUP BY rollup of its direct children. They are not merged because the interesting failure is when they disagree: if the Batch job dies while 40 children run, the parent must be able to say SYSTEM_ERROR with those children still RUNNING — the orphan condition we want visible. Equally, a failed child does not fail the launcher.

Submission now dispatches by engine. Every POST /runs used to ping the HealthOmics Lambda, so a launcher run would have been pushed at HealthOmics. awsbatch runs go to ExternalDispatchSubmissionService: no AWS call, one system_logs line, left QUEUED for its submitter. Every other engine behaves exactly as before.

The Omics callback is now executor-agnostic. handle_omics_state_change became a thin adapter over handle_state_change with a per-executor status map, so the existing route, schema, and tests are untouched. POST /internal/callbacks/executor-state-change accepts either X-Internal-API-Key (the EventBridge relay Lambda) or X-Internal-Service-Key (APIServer, which is what knows the jobId from SubmitJob and reports pre-start failures) — two callers, one endpoint, two independently rotatable keys. A reported log stream becomes a CloudWatch console deep link, with the raw stream name kept in outputs for APIServer's log viewer.

Client and CLI: parent_run_id on list_runs/iter_runs, new get_run_progress and report_executor_state operations, plus wes runs list --parent, wes runs progress <id>, and wes runs tree <id>.

Who submits the Batch job

NGS360 APIServer submits; this service owns the record. APIServer already has submit_batch_job, a BatchJob table, an EventBridge-driven status/log-stream path, a paginated CloudWatch log viewer, and the IAM — and a launcher job submitted by WES would be missing from the jobs UI operators already use. If we later want WES to submit, the seam is get_submission_service: add a BatchWorkflowSubmissionService and register it. Nothing else in the design moves.

Review notes

• The migration is hand-written (autogenerate needs a live DB, which wasn't available). Verified against MySQL 8: upgrade head from empty, alembic check reports "No new upgrade operations detected." — i.e. it matches the models exactly — then downgrade -1 and re-upgrade to prove reversibility.
• ON DELETE SET NULL, not CASCADE: deleting a launcher must not delete the child workflows, whose outputs outlive the orchestration.
• Index is composite (parent_run_id, created_at) — child listings are "one parent, newest first", and it satisfies InnoDB's FK indexing requirement, so no separate index.
• Unknown parent is now 400. Related: create_run's other validation failures (missing ProjectId, unsupported workflow_type) were surfacing as 500s; the route already declared a 400.
• make lint clean, make test 360 passed.
• Based on refactor/wes-workspace-packages, not main.

Not in this PR

• ../PAML (ngs360_platform.py), without which the launcher cannot participate. Written and unit-tested on a feature/launcher-orchestration branch there, to be released and pinned separately: get_current_task() returning a task built from WES_RUN_ID; ParentRunId in submit_task's tags; a WES_SERVICE_KEY fallback credential for headless runs (a Batch job has no user session to source a token from), with the bearer token still winning when both are set; and redacting credential-shaped values from the environment dump PlatformFactory.detect_platform writes to the log when detection fails.
• Two reuse bugs found on the way, filed rather than fixed here. get_tasks_by_name does not paginate, so the task_name=None call — "all runs in this project" — is capped at the 10 newest runs, as is any sample retried more than 10 times; it fails by returning fewer rows than exist, with no way for a caller to tell truncation from a short list (NGS360Platform.get_tasks_by_name silently returns at most 10 runs (no pagination) PAML#130). And the launcher passes inputs_to_compare positionally into get_tasks_by_name's workflow parameter, so PAML receives None and skips the parameter comparison entirely, while the misplaced dict becomes a workflow_url filter that this service's filter handling silently drops — reuse therefore matches any same-named run regardless of parameters. Filed against the launcher repo; a one-word keyword-argument fix.
• ../APIServer: the submit path — resolve the AWSBatch deployment, create the parent run, build argv from workflow_params, submit as jobName=wes-<run_id>, report the binding to the callback.
• Infra: the EventBridge Batch rule branch, the launcher job definition, the WES_SERVICE_KEY secret.
• Deferred by decision: CancelRun for a launcher does not terminate the Batch job or cascade — as today it only sets CANCELING for every engine, so this is not a regression, but cancelling a launcher without cascading leaves children burning compute.

Testing

Highest remaining risk is PAML: the integration test drives child submission over HTTP, not through the launcher's actual platform abstraction, and PAML's own tests mock requests, so nothing yet exercises the two sides against each other. Running the RNA-Seq launcher image against a local service with a two-sample samplesheet is the check that would catch a mismatch. scripts/launcher_example.sh walks the flow in the meantime.

🤖 Generated with Claude Code

[accumb3ns]

Pushed b24a0d0, a docs-only change: the launcher env contract in docs/ARCHITECTURE.md now gives launchers NGS360_AUTH_TOKEN, an ngs360_ API key minted for the submitting user for one job, plus NGS360_AUTH_KEY_ID so the launcher can retire it on exit. Before, it gave them WES_SERVICE_KEY + WES_ON_BEHALF_OF.

Why the service key is the wrong credential for launchers:

• It covers only this service. The launcher also calls the NGS360 API directly (project lookup, results upload), and that API doesn't accept it.
• Anything holding it may assert any user.

The service key itself is unchanged and keeps its role for trusted sibling services.

Counterparts: NGS360/APIServer#484 mints the key; NGS360/PAML#132 retires it.

Still outstanding on this branch: it predates main's change that forwards the caller's bearer token to the NGS360 API. That arrives when the base refactor/wes-workspace-packages is brought up to date with main, and is needed for child-run workflow lookups to authenticate.

[accumb3ns]

Superseded by #233: the same launcher-orchestration work ported directly onto main, without the refactor/wes-workspace-packages package split this PR was stacked on. The package split never merged and main has moved on in the existing layout, and nothing the launcher feature needs depends on it. #233 drops the client/CLI lineage commands (they rely on the client package from the split) and uses the existing callback key on the executor callback instead of the service key. The feature/launcher-orchestration branch is left in place for reference.