Repository navigation
Conversation
A launcher (orchestrator) execution is itself a workflow run, and every child workflow it submits now points back at it. This is what lets launcher progress be derived from persisted state rather than from a status endpoint on the launcher container. The index is composite (parent_run_id, created_at) because child listings are "one parent, newest first", and a launcher fans out to hundreds of children. It also satisfies InnoDB's requirement that a foreign key column be indexed. ON DELETE SET NULL, not CASCADE: deleting a launcher run must not delete the child workflows it submitted, which are independent executions whose outputs outlive the orchestration that started them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…allback - parent_run_id on RunSummary and RunLog, so a caller can see which launcher submitted a run without a second request. - RunProgress: a run's own state plus a rollup of the runs it submitted. The two are separate fields on purpose -- a launcher whose Batch job died while its children run must be able to report SYSTEM_ERROR with 40 children still RUNNING, which is the orphan condition we want visible. - ExecutorStateChangeCallback: the executor-agnostic successor to the HealthOmics-specific callback payload, carrying the executor name so the service can pick the right status vocabulary. - TERMINAL_STATES beside State, so the progress rollup and any poller agree on which states a run does not leave. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dation as 400
Children arrive through the ordinary GA4GH POST /runs carrying a reserved
ParentRunId tag, which create_run promotes into the indexed parent_run_id
column -- the same tag-promotion pattern already used for ProjectId and
TaskName. Listing by parent then needs no server work at all, because the
generic ListRuns filter resolves any WorkflowRun attribute by name. That
listing is also how a restarted launcher rediscovers work it already
submitted.
An unknown parent is a client error, so map create_run's ValueError to 400
rather than letting it surface as a 500. Every validation failure in create_run
-- missing ProjectId, unsupported workflow_type, and now an unknown parent --
was being reported as a server fault; the route already declared a 400.
GET /runs/{run_id}/progress is one GROUP BY over the direct children, modelled
on get_system_state_counts. Direct children only: a grandchild is reached by
asking for its own parent's progress.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Omics Every POST /runs pinged LambdaWorkflowSubmissionService, so a launcher run created by NGS360 APIServer would have been pushed at HealthOmics. Runs whose engine is awsbatch now go to ExternalDispatchSubmissionService, which makes no AWS call, notes "awaiting external dispatch" in system_logs, and leaves the run QUEUED for whoever submits the Batch job. Any other engine, including None, still goes to the Lambda service exactly as before. The factory is also the seam for the deferred decision about who submits: if this service should submit Batch jobs itself, add a BatchWorkflowSubmissionService and register it here, and nothing else in the design moves. POST /runs now picks its strategy through get_submission_service, still forwarding the caller's bearer token to whichever strategy it picks. Also parameterise the hardcoded "AWSHealthOmics (us-east)" deployment lookup by the requested engine, so the same NGS360 registry lookup can serve a Batch deployment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Generalised rather than replaced: handle_omics_state_change is now a thin adapter over handle_state_change with the HealthOmics status map, so the existing route, schema, and tests keep working untouched. AWS Batch gets its own map (SUBMITTED/PENDING/RUNNABLE -> QUEUED, STARTING -> INITIALIZING, SUCCEEDED -> COMPLETE, FAILED -> EXECUTOR_ERROR, SUBMIT_FAILED -> SYSTEM_ERROR); the existing transition table already permits that progression, so it is verified by test rather than edited. POST /internal/callbacks/executor-state-change takes the same X-Internal-API-Key as /omics-state-change. Its two callers -- the relay Lambda fed by EventBridge, and NGS360 APIServer, which is what knows the jobId it just got back from SubmitJob and reports pre-start failures -- both hold that key. A reported log stream becomes a CloudWatch console deep link in stdout, with the raw stream name kept in outputs so APIServer's log viewer can still use it. batch_log_group and aws_console_region configure that link; no Batch credentials are needed, because this service does not submit Batch jobs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Service and API: ParentRunId promotion, an unknown parent rejected as 400, the parent filter selecting only that launcher's children, and a progress rollup that counts direct children while deliberately excluding a grandchild. Callbacks: the full Batch progression through to SUCCEEDED, a duplicate event_id as a no-op, FAILED recording the container exit code, a post-terminal event ignored, an unknown executor rejected, and HealthOmics still working through the generic route after the refactor. Route-level tests confirm the internal callback key is accepted and a wrong or missing one refused -- these patch get_settings where callback_auth looks it up, because that module reads settings directly rather than through FastAPI's DI, so the app fixture's override does not reach it. Integration: a launcher plus three children end to end, with the submission factory left real so the launcher genuinely takes the external-dispatch path and only the children's HealthOmics call stubbed. A failed child leaves the launcher COMPLETE, which is the separation the design asks for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ARCHITECTURE.md gains a Launcher Orchestration section: why a launcher execution is an ordinary WorkflowRun rather than a new table, how ParentRunId lineage works, why parent state and child progress are kept separate, why APIServer submits the Batch job while this service owns the record, the per-executor status maps, the launcher container env contract, and what is explicitly not done yet (cancel cascade, relaunch). The schema table, migration chain, API surface and settings table pick up parent_run_id, the two new routes and the Batch log settings. The env contract gives a launcher NGS360_AUTH_TOKEN, an ngs360_ API key minted for the submitting user for one job, plus NGS360_AUTH_KEY_ID so the launcher can retire it on exit. The launcher calls the NGS360 API directly as well as this service, which forwards the token on its own NGS360 lookups, so one user credential covers every call, each authorized against that user. The README gains a Launcher runs subsection: the launcher being started as a Batch job, and its children being tagged back to it. scripts/launcher_example.sh walks the whole flow against a local service with curl: submit a launcher, report its Batch job RUNNING, fan out two children, then read the child listing and progress back. It sends RUNNING before a terminal status because QUEUED -> COMPLETE is not a legal transition, and skips the callback steps when no key is set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The engine now selects the submission backend, so it has to be a name the service actually supports. It was neither validated nor advertised: service-info listed only cwltool, which this service cannot run, while get_submission_service sent anything it did not recognise to the HealthOmics Lambda. A launcher submitted as "aws-batch" was accepted, dispatched to HealthOmics, and failed there on a deployment it never had, reporting a HealthOmics error for a typo. service-info now advertises awsbatch and awshealthomics, and create_run validates against that same map -- the spec makes it the client's only way to discover legal values, so the check and the advertisement cannot disagree. The submission factory dispatches through a SUBMISSION_STRATEGIES registry keyed by those names and raises on an unregistered engine instead of defaulting. Omitting the engine still means this instance's default backend, so callers that predate launchers are unaffected. A submission naming an engine that is neither advertised nor omitted is now a 400 listing the accepted names. That includes cwltool, so scripts/wes_client.py stops defaulting to it and omits the engine unless one is given, and the one service test that submitted it now names awshealthomics. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rejection of unadvertised engines at both the service and the route, acceptance of an advertised name in any case or padding, and the spec's rule that an engine is required when an engine version is given. Two tests exist to stop the advertisement and the dispatch registry from drifting apart, which is what made the original bug invisible: test_registry_matches_advertised_engines compares the two sets directly, and a route test submits every engine service-info advertises and expects 200 -- the spec's discovery contract as an assertion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RunSummary and RunLog mirrored the promoted parent_run_id column as a
top-level response field. That duplicated information already present,
unmodified, in the tags field: create_run never strips ParentRunId out
of tags_dict before storing it, so tags["ParentRunId"] carries the same
value on every response. The spec's RunSummary requires only run_id,
state, start_time, end_time, and tags, so the extra field was this
service's own addition beyond the minimum GA4GH shape, unlike the
parent_run_id column itself -- an internal MySQL indexing detail the
spec has no opinion on either way, kept as-is for the same reason
ProjectId/TaskName are promoted into project/task_name.
Filtering (?filters={"parent_run_id":...}) and the progress rollup are
unaffected: both query the internal column directly and never touched
the response schema.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Pipeline launchers (e.g. the RNA-Seq launcher) are moving from CWL wrappers to plain AWS Batch jobs. Each one submits one workflow per sample and then a gather step. This PR makes a launcher execution a first-class WES run and links every child run it submits back to it, so launcher progress comes from stored state rather than from a status endpoint on the launcher container.
This replaces #183. That PR was stacked on the
refactor/wes-workspace-packagespackage split, which never merged, andmainhas kept evolving in the existing layout since. Nothing the launcher feature needs depends on that restructure, so this is the same work ported directly ontomain. If the package split goes ahead later, it can be redone on top of this.What changed
parent_run_idcolumn (FK,ON DELETE SET NULL, indexed withcreated_at) is promoted from a reservedParentRunIdtag, the same wayProjectId→projectworks. An unknown parent is a 400.GET /runs?filters={"parent_run_id": ...}. That listing is also how a restarted launcher finds work it already submitted.GET /runs/{run_id}/progressreturns the launcher's own state plus a rollup of its direct children by state.workflow_engine=awsbatchis recorded and leftQUEUED, with no AWS call. NGS360 APIServer submits the Batch job.awshealthomics, or no engine at all, goes to the Lambda exactly as before.service-infoadvertisesawsbatchandawshealthomics, andcreate_runaccepts only those, so a typo is a 400 instead of a HealthOmics failure.scripts/wes_client.pyno longer defaults tocwltool, which is now rejected.POST /internal/callbacks/executor-state-changeis executor-agnostic, with per-executor status maps. AWS Batch has its own; the HealthOmics route is unchanged and now a thin adapter over the same handler.X-Internal-API-Key.952781491f2ffollowsdd9d7ebae80e, so the chain stays linear with a single head.ARCHITECTURE.mdgains section 15 (Launcher Orchestration) and updated schema, API and settings tables; the README gets a "Launcher runs" subsection.scripts/launcher_example.shwalks the whole flow against a local service with curl.Launcher credentials
A launcher authenticates as the user who started it. APIServer mints an
ngs360_API key for the submitter, for that one job (NGS360/APIServer#484), and the launcher retires it on exit (NGS360/PAML#132). This service forwards that token on its NGS360 lookups, so one user credential covers every call the launcher makes, each authorized against that user. The internal service key is not handed to launcher containers.Not in this PR
wesCLI/client lineage commands from feat: launcher orchestration — run launchers as AWS Batch jobs with lineage in WES #183. They depend on the client package from the package split.Testing
uv run pytest: 234 passed, up from 181 onmain. Each of the 10 commits passes on its own.uv run flake8 src tests scripts: clean.alembic upgrade head --sqlagainst MySQL renders a linear chain ending in952781491f2f.scripts/launcher_example.shwas run end to end against a local server (SQLite, callbacks enabled).🤖 Generated with Claude Code