Repository navigation
feat: launcher orchestration support (current-task, ParentRunId tagging, per-launch key retirement) - #132
Open
accumb3ns wants to merge 6 commits into
Open
feat: launcher orchestration support (current-task, ParentRunId tagging, per-launch key retirement)#132accumb3ns wants to merge 6 commits into
accumb3ns wants to merge 6 commits into
Conversation
accumb3ns
marked this pull request as ready for review
August 25, 2026 17:42
|
We may want to support running the launcher as a native CWL workflow hence we get that for free. |
get_current_task() returned None with the note that "WES API doesn't have a concept of current task". That was accurate when it was written: a run could not be identified from inside its own container. The corresponding WES service change makes a launcher execution a first-class WES run, so the launcher can be told its own run id -- WES_RUN_ID, the same shape as SevenBridges' TASK_ID -- and read its own run record back. This is what the RNA-Seq launcher needs to reach its own inputs (platform.get_task_input(current_task, "factors_file")). The inputs come from request.workflow_params, which is what the run was submitted with. Raises ValueError when WES_RUN_ID is unset, matching what the SevenBridges and Arvados platforms do for their equivalent variables -- a launcher that cannot find itself is misconfigured, not idle. test_current_task_and_users loses its assertIsNone half and becomes test_project_users; the current-task behaviour now has its own two tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nothing recorded which launcher execution submitted a given run, so a launcher's progress could only be answered by asking the launcher. Adding ParentRunId to the tags submit_task already sends -- ProjectId and TaskName -- links each child back to the launcher run that created it. WES promotes ParentRunId into an indexed column, so the service can roll up launcher progress from persisted rows instead of the launcher exposing a status endpoint, and a restarted launcher can find the work it already submitted. Only set when WES_RUN_ID is present, so a launcher run outside WES (a developer's shell, an external platform) submits exactly the tags it does today. test_submit_task hides the launcher variables so its exact-tags assertion does not depend on the developer's shell. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
When platform detection fails, PlatformFactory logs every environment variable so whoever is debugging can see what was set. That dump goes wherever the container's logs go -- CloudWatch for a launcher running as an AWS Batch job -- which is a wider audience than whoever provisioned the credential, and the launcher's environment now carries WES_SERVICE_KEY alongside the pre-existing NGS360_AUTH_TOKEN and SB_AUTH_TOKEN. Mask values whose variable name contains TOKEN, KEY, SECRET, or PASSWORD, keeping the character count: "set but empty" and "set to a truncated value" are both real misconfigurations, and the dump exists to tell them apart. Everything else is logged unchanged. Adds tests/test_platform_factory.py -- the factory had no test of its own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The module reads six environment variables across connect(), get_current_task() and submit_task(), so the full contract was only discoverable by reading the whole file. Whoever writes the AWS Batch job definition -- or debugs a launcher that starts and cannot find itself -- needs it in one place. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A launcher running headless (e.g. as an AWS Batch job) is given an ngs360_
API key minted for the user who started it, in NGS360_AUTH_TOKEN. When
NGS360_AUTH_KEY_ID names that key, connect() registers an exit handler
that retires it through DELETE /api/v1/auth/api-keys/{id}, so the key
does not outlive the run. The key authenticates its own retirement.
The handler is registered before the /auth/me probe, so a launcher that
fails to connect still retires its key. A failed retirement is logged,
not raised; the key's expiry bounds a launcher that is killed first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
accumb3ns
force-pushed
the
feature/launcher-orchestration
branch
from
October 28, 2026 13:41
9ba6760 to
a4ac6f7
Compare
accumb3ns
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Bioinformatics launchers (e.g. the RNA-Seq launcher) will run directly as AWS Batch jobs, as first-class GA4GH WES runs that link every child run back to themselves through a
ParentRunIdtag (see NGS360/GA4GH-WES-API-Service#233). A launcher running that way has to identify its own run, and it needs a credential that outlives the job without anyone pasting a token in.What changed
get_current_task()returns aWESTaskbuilt fromWES_RUN_ID(SevenBridges usesTASK_IDthe same way) instead of always returningNone. This lets a launcher read its own inputs back throughget_task_input.submit_task()tags every submitted run withParentRunIdwhenWES_RUN_IDis set. WES can then work out launcher progress from stored rows, without the launcher having to expose a status endpoint.NGS360_AUTH_KEY_IDnames thengs360_API key inNGS360_AUTH_TOKEN,connect()registers an exit handler that retires the key (DELETE /api/v1/auth/api-keys/{id}), so a key minted for one launch doesn't outlive it./auth/meprobe, so a launcher that fails to connect still cleans up.PlatformFactory.detect_platform()redacts credential-shaped environment values (TOKEN/KEY/SECRET/PASSWORDin the name) before logging them when platform detection fails.Changed since the first revision
This branch was rebased onto
main, which now sends the bearer token on every NGS360 API call. In the process I dropped theWES_SERVICE_KEY/WES_ON_BEHALF_OFfallback:Instead, the launcher authenticates as the user who started it, with an API key minted for that one job. The minting side is NGS360/APIServer#484.
Not in this PR
get_tasks_by_namehas no pagination, so it silently returns at most the 10 newest matching runs.Testing
uv run pytest: 166 passed, 18 skipped. Every commit in the series passes on its own.🤖 Generated with Claude Code