Repository navigation
Add first-class GitHub Actions and Azure Pipelines integrations - #46
Open
Cristian Velez (kekivelez) wants to merge 43 commits into
Open
Cristian Velez (kekivelez) wants to merge 43 commits into
Cristian Velez (kekivelez) wants to merge 43 commits into
Conversation
…slice Sixty-first corrective pass. The pass-60 fix keyed escaped- ew-identifier provenance on a leading @ (masked[t.start] === '@'), but C# permits identifiers named ew without a @ prefix: a Unicode-escaped spelling (n\u0065w) or a spelling containing a removed Cf format character both normalize to ew yet are identifiers (a keyword can never carry a @, a Unicode escape, or a format char). Because normalization dropped provenance for those non-@ spellings, classifyNewTypeAt misread a reassigning nullable-return n\u0065w() local function as an object initializer, causing csharpComposedConditionalRequestHeader to wrongly keep tracking across the reassignment and return If-Match instead of null. - normalizeMaskedIdentifiers: replaced the @-only provenance check with a raw-slice comparison (t.name === 'new' && masked.slice(t.start, t.end) !== 'new' ? '@new' : t.name), robustly covering every non-literal spelling (verbatim @new, Unicode escape, Cf format char). - Corrected stale doc comments claiming the @ was the mandatory/only identifier spelling for ew. - Added pass-61 regression tests: Unicode-escaped (nullable- and void-return) and Cf-format-char local-function negatives, plus a retained literal new() object-initializer positive guard. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a genuinely-pending, abort-aware PendingCredential test fake to harness.ts whose getArmToken(scope, signal) returns a promise that never settles on its own and only rejects with AbortError when its signal aborts (optional settleAfter lets earlier acquisitions resolve normally). Add new regressions in run.test.ts and orchestrator.test.ts that let the deadline watcher advance virtual time and abort the never-settling credential, proving the watcher itself enforces the deadline rather than only a post-acquisition recheck. Relabel the prior slow-but-settling FakeCredential tests to clarify they cover a different path, retaining that coverage alongside the new pending-abort regressions. Test-only change; no src/ files modified. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…, strict getBool, template-labeled examples)
- Implement GithubInputReader/OutputSetter adapter over shared E2 core (adapter.ts, host.ts, index.ts)
- Add AzureCliCredential-based WIF credential provider (auth.ts) reusing azure/login session
- Harden GithubInputReader.getBool to strictly accept only true/false (case/space-insensitive), throwing CoreError('identifier', ...) on malformed input instead of silently defaulting to false
- Add contract, parity, WIF, and examples tests (adapter.test.ts, parity.test.ts, auth.test.ts, index.test.ts, examples.test.ts)
- Relabel per-mode example workflows as non-runnable templates pending the published preview release SHA; rewrite examples README accordingly
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Implement thin azure-pipelines-task-lib adapter over the E2 core (input/output/logging, SIGINT/SIGTERM to AbortSignal, task.complete mapping, setSecret) - Add ICredentialProvider backed by @azure/identity ClientAssertionCredential using ARM service-connection WIF - Extract OIDC token exchange into pure, injectable-transport module (oidc.ts) with a true absolute request deadline instead of a socket-inactivity timeout; single done() guard clears the timer and destroys the request on every settlement path - Add 8 deterministic oidc.ts tests using fake transport + fake timers - Add adapter contract tests, parity tests, credential tests, and extension-manifest clean-install tests - Add per-mode examples including deployment-gate - Ignore azure-pipelines-task-lib's generated .taskkey scratch file - Correct test-header wording to describe entry/credential tests as offline tests with injected dependencies rather than SDK-free Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- E5-T1: cross-platform parity suite (test/parity/**) - E5-T2: manifest/package/dist-integrity suite (test/package/**) - E5-T3: supply-chain hardening - fork-safe license-free gitleaks CLI secret scan with fingerprint-pinned baseline, dependency/license review, npm audit, RBAC role template + RV2 positive/negative gates, CODEOWNERS - E5-T4: docs for per-mode/per-platform usage, deployment gates, no-wait warning, two-identity least-privilege, concurrency guidance Resumed and completed interrupted correction loop: replaced licensed gitleaks-action with checksum-pinned CLI (fork-PR safe, no secrets required), added security/gitleaks-history.ignore baseline derived from real gitleaks run, extended TS CI path filters, aligned RBAC docs to workspace-resource assignment scope. Verified: typecheck, build, 702 tests (700 pass, 2 environment-gated skips), real gitleaks scans failing without and passing with baseline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Implements E6 (preview release & handoff) — the repo-side half of "run RV1-RV3,
then publish both marketplaces from one commit". Live RV execution and the
marketplace publications need a real preview environment and repo/ADO admin
rights; they are deliberately NOT simulated. What ships here is the enforcement
that makes them unskippable.
E6-T1 — reproducible release-validation evidence:
- test/release-validation/receipt.ts: receipt schema plus fail-closed RV1-RV3
evaluators that RE-EVALUATE every recorded observation against the
source-proven contract constants, so a receipt that merely claims passed is
rejected. RV1 requires one transcript per shipping adapter, each from its own
run (distinct, case-folded run IDs); execute/cancel Location suffixes must
address the observed run. RV2 evaluates every identity entry so a
contradictory duplicate cannot hide behind a passing one.
- scripts/lib/rv-receipt.mjs: deterministic stamp/verify CLI. The digest is
mandatory, the committed template is refused by both verbs, and freshness is
bounded on the observations as well as the envelope (post-dated stamps and
observations that post-date their own receipt are rejected).
E6-T2 — one commit, two marketplaces:
- Receipt gate in the non-secret �alidate job of release-action.yml and in the
unsigned �uild stage of OneBranch.Official.yml. Each proves receipt->release
ancestry AND byte-identical shipped trees, so a receipt cannot be reused
across a behaviour change. The ADO parameter is basename-only and the shallow
clone is deepened before the git comparisons, failing closed either way.
- docs/runbooks/release.md and rollback-and-deprecation.md.
E6-T3 — drift detection and operational handoff:
- .github/workflows/contract-drift.yml: scheduled, privilege-split. drift runs
repo code with no write permission;
eport holds issues: write and no
checkout. A protocol mismatch opens a SERVICE DEFECT — fixtures are never
regenerated and the pinned api-version is never bumped to make it green.
- docs/runbooks/{release-validation,contract-drift,README}.md, linked from
docs/ci-cd-integrations.md.
Runbooks are reconciled against what the workflows actually enforce (dispatch-only
release, exact vMAJOR.MINOR.PATCH tags, workflow-managed floating major tag), and
the limits of each control — the receipt trust boundary, the absent SLSA
attestation, and drift detection's blindness to unmirrored upstream change — are
documented rather than overclaimed.
Verification: npm run typecheck (0), npm test (753 tests, 751 pass, 0 fail,
2 pre-existing skips needing RUN_WIF_INTEGRATION=1), npm run build reproduces
dist byte-identically.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…te the drift defect on contract evidence Two reviewer findings on the committed E6 implementation (f1eb742..f4b09f3). (1) OneBranch shipped bytes the receipt never validated. The receipt gate compares the COMMITTED shipping paths (packages, action.yml, dist, azure-pipelines-extension) between the validated and build commits, but the pipeline REBUILDS dist/ and stages that generated runtime into the VSIX. A build INPUT outside those paths - scripts/build.mjs, root dependency metadata, a transitive bundler version - could therefore change the packaged bytes while the comparison still passed. scripts/lib/verify-built-runtime.mjs compares the rebuilt working-tree runtime against the runtime COMMITTED at a baseline commit: complete file set (added AND removed), git-normalized modes, and git blob hashes, rejecting symlinks/submodules and non-regular rebuilt entries, and refusing an empty rebuild or an empty baseline. The OneBranch build stage runs it after `npm run build` and BEFORE staging, packaging, signing or publishing. The receipt gate publishes the validated commit as `ReleaseValidatedCommit`; a publishing run with no 40-hex baseline FAILS rather than degrading to HEAD, and a non-publishing build compares against HEAD (the same build-drift gate the GitHub test/release-action workflows already enforce). (2) contract-drift claimed a service defect on evidence it did not have. Every drift-job failure - including `npm ci` and unrelated release-validation assertion failures - opened an issue asserting the source contract disagrees with the shipped fixtures. The drift job now publishes an EXPLICIT `contractMismatch` verdict set only by the three checks that can observe the contract (provenance re-derivation, the source-contract suite, the pinned api-version assertion); the release-validation suite is split into its own step and records no verdict. The report job branches on that verdict: a confirmed mismatch files the `contract-drift` service defect unchanged, anything else files a `contract-drift-workflow-failure` issue that states plainly it is not evidence of drift. The privilege split is preserved - `report` still holds the only write permission, takes no checkout, and runs no repo code. Runbooks updated to match (release.md, release-validation.md, contract-drift.md). Verification: npm run typecheck (0), npm test (764 tests, 762 pass, 0 fail, 2 pre-existing skips needing RUN_WIF_INTEGRATION=1), npm run build reproduces dist byte-identically. STILL BLOCKED, deliberately not simulated: live RV1-RV3 in a target preview environment and the two marketplace publications require repo-admin/ADO-admin rights and a real environment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…n errors The scheduled drift workflow escalates a protocol mismatch as a SERVICE DEFECT, but its shell wrappers could not tell a comparison mismatch from an execution error: any nonzero `node --test` exit (syntax error, unresolvable import, empty discovery, a test that died before any assertion) was recorded as `contract=mismatch`, and `git diff`/`grep` command errors were indistinguishable from `differences found`/`no match`. The three contract checks now live in scripts/lib/contract-drift.mjs with an explicit three-way verdict: 0 = checked and matching, 1 = checked and IDENTIFIED mismatch (the only path that records contract=mismatch), 2 = the check could not be evaluated (records nothing, so it is reported under the separate contract-drift-workflow-failure label). A contract-suite mismatch is derived from a failing test's own TAP `code` diagnostic key rather than the process exit status, git diff --exit-code status 1 is separated from every other status, and the grep scan is replaced by direct reads whose I/O errors are execution errors. The contract checks also run independently of one another and ahead of the repository-side release-validation suite, so one failure cannot skip another check and downgrade real drift to a workflow failure. test/release-validation/contract-drift-checks.test.ts adds behavioural coverage: each failure mode is provoked against a purpose-built tree and the resulting verdict asserted. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ocol assertions A CONTRACT MISMATCH is a claim about the SERVICE, escalated as a service defect. Mismatch-producing evaluation ran the whole packages/core/test/contract glob, so a genuine ERR_ASSERTION from a repository-only assertion (the packaged task manifest's minimumAgentVersion, the ADO signing configuration, the release rulesets) would have been reported as a protocol mismatch on no protocol evidence. - Classify every contract test file explicitly: `contract-suite` runs only the six source-protocol files and is the only mismatch-capable check; the new `repository-suite` runs the six repository-policy files and can report only green or unevaluable; the new `classify` check fails on an unclassified or stale file, so a new test is neither assumed to be protocol evidence nor silently skipped. `classify` runs independently of the suites so it cannot suppress a real protocol verdict. - Attribute failures per assertion inside the two protocol files that also carry repository-only assertions (contract.test.ts client conventions; provenance.test.ts `negative:` fixture-validator self-tests) via an audited REPOSITORY_ONLY_ASSERTIONS list; a rule that matches nothing in its file, collides with another protocol file, or names a non-protocol file fails `classify`. - Compare the pinned api-version against the module's ACTUAL exported API_VERSION value instead of an exact source-text substring: equivalent declarations no longer read as drift, a commented expected declaration no longer satisfies the check, and a missing/non-string export is an execution error rather than a service defect. - Wire the classify/repository-suite steps into the drift workflow and document the classification, the exemptions and the reproduction commands in the runbook. Behavioural coverage in test/release-validation/contract-drift-checks.test.ts grows from 18 to 35 cases. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…n file `contract-suite` flattened every source-protocol file's REPOSITORY_ONLY_ASSERTIONS rules into one list and matched failures by name alone, so a rule declared for one file could suppress a genuine protocol failure that happened to carry that name in a different file, downgrading real drift to a mere workflow failure. `classify` detects the collision, but it runs independently by design, so the suite still consumed the invalid exemption. Each declared source-protocol file is now run and classified on its OWN rules, and the per-file verdicts are folded by a new `aggregateContractVerdicts` (mismatch anywhere outranks an error elsewhere; details name the originating file). The collision regression is extended with a failing protocol assertion in the non-exempt file, which must emit contract=mismatch, plus the mirror case where the exempt assertion fails in the file that declared it and stays a repository regression. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…-bearing release commit Both release configs read the release-validation receipt from their own checkout of the commit being released, so the operational steps must select the later receipt-bearing release commit, not the validated core commit the receipt records. The runbooks said the opposite, which would have selected a commit with no receipt and blocked publication. The ancestry and byte-identical shipping-equality gates are unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…index steps The incident repoint of the floating v1 tag pointed at the validated core commit rather than the commit the last-good version tag resolves to, and the runbook index and roll-forward prose still conflated the two commits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… one phrasing The regression assertions accepted only the literal phrase 'release commit' and did not cover the runbook index or the RV build pinning. They now accept either spelling of the receipt-bearing commit, assert the incident repoint on the command line itself, and cover both remaining documents. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…triage contract mismatches before calling them service defects Bind the OneBranch publication to the GitHub release's own durable record: the exact version tag. A per-pipeline receipt gate accepts ANY receipt-bearing descendant of the validated core commit, so both marketplaces could pass their own gate on different commits. scripts/lib/release-commit.mjs resolves refs/tags/<tag> and rejects any other build commit; the releaseTag parameter now selects both the receipt path and that commit, so they cannot disagree. Report a drift verdict for what the checks can actually see. The drift checks observe only this repository's committed artifacts, so a repo-only editing error produces the same mismatch as a real service change. The issue now asks for triage and reserves the service-defect classification for authoritative-source or live-RV evidence, matching docs/runbooks/contract-drift.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…eady-open drift issues The previous pass withdrew the "confirmed service defect" verdict from new contract-drift reports, but an issue opened by the earlier generation kept its incorrect title and body: the recurrence path only appended a short "found it again" comment and returned, so the corrected triage explanation never reached the issue an operator actually reads. Every generated body now ends with a classification marker. An open issue that lacks it AND still contains the withdrawn wording verbatim is recognized as stale, and handled by whether a human has touched it. One still exactly as the workflow wrote it is migrated in place to the corrected title and body, because there is no triage to lose. One that has been edited, retitled, or filed by a person is left as filed and receives a one-time corrective comment instead, so recorded triage is never overwritten. Text appended to the generated run-URL line counts as human content, and the dedup marker is looked for across the whole comment history rather than its first page. The reporting job deliberately has no checkout and holds the only write permission, so the logic stays inline in the workflow. The new test therefore extracts the shipped inline script and evaluates it against fake github, context and core objects, and replays the real previous generation out of git history rather than against a copy of its own wording. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…require generated-report evidence The non-tautology guard searched git history for the withdrawn verdict, but the current workflow quotes that verdict verbatim in order to recognize a stale issue, so the search selected HEAD and the guard compared the template with itself. The search was also unrunnable: every job that runs this suite uses a default shallow checkout and has no history to search, so scheduled runs would have reported failures on healthy artifacts. The previous generation is now preserved as a byte-identical committed fixture and replayed from there. Git is used only to re-confirm the fixture against the pinned blob when full history happens to be present, and its absence is not a failure. Staleness also demanded only the withdrawn verdict, so a human discussion issue quoting that paragraph was told it had been filed with a classification the workflow had withdrawn - a statement untrue of someone else's issue. A stale issue must now be bot-authored and carry the whole structure of the old report. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… label alone The reporter deduplicated on the first open issue carrying the report label, without establishing it was a generated report. An unrelated human-filed (or unrelated-automation) issue that merely carries the label absorbed the run's recurrence comment, and the unconditional return that followed suppressed creation of the actual triage report -- so a real contract mismatch could go unreported behind an unrelated issue. Selection is now by positive provenance: scan the labeled issues for one this workflow authored that carries either the current classification marker or the whole structure of a recognized earlier generation. legacyEvidence is declared per report, so the unchanged workflow-failure wording is recognized too. Anything else keeps its label and is left alone: no recurrence comment, and no suppression of the report the run owes. Scanned rather than taken from the head of the list, because the issues API does not guarantee a report is listed first. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… page is reused The reporter selected the generated drift report from a single 100-issue page. The contract-drift label also accumulates human-filed and triage-applied issues, so the report a run must reuse can sit beyond the first page; missing it opened a duplicate on every scheduled run instead of reusing the current-generation report or migrating a legacy one. Walk the labeled issues with github.paginate, matching the treatment the comment history already received, and apply the provenance test across all pages. The fake listForRepo is now page-aware so a single-page reader cannot pass, and two regressions place a generated report after a full page of labeled human issues: a current-generation report must draw only the recurrence comment, and a legacy report must be migrated in place. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- R1: replace fail-fast dist placeholders with real esbuild-bundled, self-contained CJS adapters (github-action + azure-pipelines-task), keeping Node20/Node20_1 runtime compatibility; add sentinel-absence, size, and executable smoke-check tests for both bundles. - R2: generate build-provenance attestation (actions/attest-build-provenance) bound to the packaged release artifact digest in the publish job, scoped to job-level id-token/attestations permissions; document verification in docs/runbooks/release.md; add workflow-structure tests binding the attestation to the verified artifact and asserting least-privilege scope. - R3: require exact positive safe-integer timeout inputs in both adapters' getInt, rejecting partial/negative/zero/overflowing values instead of silently defaulting; add coverage for malformed, fractional, negative, zero, and overflow cases on both platforms. - R4: emit completed-at for every normally observed terminal run (Succeeded, Failed, Canceled), not only success; update parity/orchestrator tests. - R5: preserve ARM error message/details (bodyErrorMessage, armErrorMessage) through CoreError/NormalizedError/failure-reason redaction instead of discarding them; add coverage for message preservation and redaction. - R6: clarify that the private dev extension contributes AzureChaosStudioScenarioDev@1 (not the production task name) in the Azure Pipelines example README and release-validation runbook; add a doc-consistency test. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Terminal-run cancellation is a no-op (RV3), so a single run's transcript could never legitimately show both Succeeded and, separately, Canceled. Split the RV1 transcript schema into a successRun (driven to Succeeded) and a cancellationRun (a second, independently identified run whose own execute is driven to Canceled via cancel). The evaluator now requires all four recorded runs per receipt (two adapters x success/cancellation) to be distinct, and rejects a cancellation run that reuses the success run's ID. Updated the receipt schema/evaluator, template, runbook guidance, and all fixtures/tests (rv-receipt.test.ts, release-handoff.test.ts) to model the two independent journeys while preserving per-adapter identity binding. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- R2: bind the verified release commit into signed provenance via a second actions/attest step alongside the standard build-provenance attestation, so ancestor releases and retries are traceable to the packaged commit. - R1: eliminate the impossible RV1 cancellation transcript by separating execute acceptance, an in-flight non-terminal observation, and terminal Canceled into distinct phases; a terminal-run cancel is no longer required to be a no-op success. - R3: add recursive secret-key redaction (password/clientSecret/accessToken/ etc.) alongside existing token-pattern scrubbing, and redact adapter fallback exception messages and failure reasons on both platforms before they reach logs or outputs. - R4: require exactly 202 for POST action acceptance (validate/execute/ cancel), separate from the any-2xx GET/poll path, rejecting off-contract 200/201 acceptances that carry a valid Location. - R5: document the completion-timeout-seconds + 300s cleanup budget job timeout requirement and the validations/latest + executionPlanJson concurrency coupling; show explicit job timeouts in both platform examples. Adds/updates tests for all of the above; full suite (862 tests) and typecheck pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- R1 (HIGH): RV1 in-flight cancellation observation now requires the source-contract-defined HTTP 202 (not 200) while a run is nonterminal, matching run-nonterminal-202.json and validation-nonterminal-202.json. Updated receipt.ts's checkInFlight/InFlightObservation docs, the RV1 fixtures in rv-receipt.test.ts and release-handoff.test.ts, and the release-validation runbook. - R2 (HIGH): regenerated dist/github-action/index.js and dist/azure-pipelines-task/index.js from HEAD so the committed bundles include the R1/R3/R4 runtime fixes and match the clean-rebuild gate. - R3 (HIGH): the Azure Pipelines host now reads the job's OAuth token from the built-in SYSTEMVSSCONNECTION service endpoint (task-lib getEndpointAuthorization), validates its OAuth scheme, and masks the token, instead of requiring an explicit System.AccessToken variable mapping that a normal custom-task job does not provide. Added host.test.ts (child-process-isolated, since task-lib's vault is populated once per process from env) covering the SYSTEMVSSCONNECTION composition, missing endpoint, wrong scheme, and missing AccessToken parameter cases. - R4 (MEDIUM): ArmHttpClient.requestOnce now catches credential-provider failures separately from the transport call and normalizes them to the `auth` category (preserving `timeout` on a deadline race and existing `transport` behavior for an already-aborted signal), instead of letting them escape as raw exceptions that toNormalizedError would otherwise default to `transport`. Added a FailingCredential test helper and targeted http.test.ts coverage for auth categorization on GET/POST, the never-reaches-transport invariant, and preserved timeout/cancellation precedence. All changes are repository-local fixes to existing E1-E6 work; RV1-RV3 live execution and marketplace publication remain external gates. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- R1 (MEDIUM): release-action.yml now generates both provenance attestations (build-provenance + release-commit) AFTER staging and verifying the draft release's assets but BEFORE the draft is flipped to published. Publication is now a separate, dedicated step gated on both attestation steps having already succeeded, so an attestation failure can never leave a public release without its required provenance. Retry handling (already-published skip) and the post-publish immutability + final asset re-verification are preserved. - R2 (MEDIUM): pinned TfxInstaller@4 to an exact tfx-cli version (0.17.0) identically across .pipelines/OneBranch.PullRequest.yml and both TfxInstaller@4 uses in .pipelines/OneBranch.Official.yml, replacing the mutable 'v0.x' range so identical commits always package/publish with identical tooling. - R3 (MEDIUM): docs/ci-cd-integrations.md no longer claims validate-and-execute is safe from validations/latest replacement because its calls run back-to-back; it now explains the polling window hazard and requires the same coordinated concurrency controls for every mode, including combined validate-and-execute jobs. - R4 (MEDIUM): added 'Trusted trigger requirements' guidance to docs/ci-cd-integrations.md, examples/github/README.md, and examples/azure-pipelines/README.md, covering trusted branch/ environment restrictions, required-reviewer approvals, and the prohibition on exposing chaos-capable credentials to untrusted fork code or workflows executing unreviewed contributions. - R5 (MEDIUM): docs/runbooks/release.md now documents the GitHub Marketplace enrollment handoff (Developer Agreement acceptance, publish opt-in, category selection, listing verification) as a distinct manual admin step following the automated GitHub Release, separate from and prior to the Azure Pipelines extension release. Test coverage: added regression tests in release-handoff.test.ts proving (a) both attestations must succeed before either draft- publication path can flip the release to published, (b) tfx-cli installs use a single, exact, non-ranged pinned version across all pipeline stages, and (c) the release runbook documents the Marketplace listing handoff distinctly from automated release completion. Typecheck and the full test suite (874 tests, 872 pass, 2 intentional live-WIF skips) pass. Live RV1-RV3 execution, external protection provisioning, and marketplace publication remain acknowledged external gates, not repository-fixable findings. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Architecture: - Break the private-preview bootstrap circularity in OneBranch.Official.yml: the RV1-RV3 receipt gate and the receipt-bound runtime-equality baseline now apply only to the PRODUCTION manifest (vss-extension.json). Publishing the private, unlisted ChaosStudioWorkspacesDev manifest (public:false, Preview gallery flag, distinct task GUID/name) no longer requires a receipt, so the private build RV needs can be published first. Signing, environment approvals, and private visibility are unaffected; the production gate is unchanged and unconditional. Documented the initial bootstrap sequence in docs/runbooks/release-validation.md. Code: - Implement a real, deterministic offline self-test in both adapter entrypoints (packages/action-github/src/index.ts, packages/task-azure-pipelines/src/index.ts) behind CHAOS_STUDIO_SMOKE_CHECK=1, so scripts/smoke-action-bundle.mjs exercises actual wiring instead of an unimplemented contract. The Azure adapter now also sets process.exitCode=1 on a genuine task failure instead of exiting 0. Regenerated both dist bundles and fixed a TDZ ReferenceError in the smoke harness early-argument-validation path. - Fix the release-action.yml shell-scoping bug: require_immutable and verify_final_assets (and the asset digest map) were defined only in the earlier staging step shell and were undefined in the later publication step fresh shell. Extracted both into scripts/lib/release-asset-verify.sh, sourced by both steps; the asset digest map is persisted to pkg/asset-digests.tsv and reloaded in the publish step. - Add verify_required_attestations to that shared library and call it on the already-published retry path, so a retry no longer treats "the release exists" as proof both provenance attestations were generated; both are now verified against the artifact digest and release commit. - Reject non-literal-boolean receipt fields (tokenAcquired, secretless, duplicateCancelAccepted, cancelOnTerminalRunAccepted, cleanupFailurePreservesOriginalFailure) in test/release-validation/receipt.ts instead of coercing via truthiness, with actionable failure messages naming the malformed field and value. Documentation: - docs/ci-cd-integrations.md: distinguish the internal completion-timeout cleanup budget (300s) from platform-enforced cancellation grace (GitHub: SIGINT, then SIGTERM at 7.5s, then process-tree kill at 10s total; not extended by job timeout), describe cancellation as best-effort, and add steps to confirm or manually cancel a surviving run. Tests: - Packaged smoke harness coverage (new test/release-validation/smoke-action-bundle.test.ts): runs the actual scripts/smoke-action-bundle.mjs subprocess against both real committed bundles and synthetic bundles covering non-zero exit, thrown errors, the placeholder sentinel, and missing arguments. - Unit coverage for runSmokeSelfTest and main() CHAOS_STUDIO_SMOKE_CHECK path in both adapters index.test.ts. - Malformed-JSON receipt-type coverage in rv-receipt.test.ts for both RV2 and RV3 fields. - Lifecycle test proving the Dev manifest can publish without an RV receipt while the production manifest gates are never exempted (release-handoff.test.ts). Full suite: 887 tests (885 pass, 2 live-WIF skips, up from 872/2), typecheck and build clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- R1: materialize scripts/lib/release-asset-verify.sh in the isolated publish job by extracting it BY CONTENT (git cat-file) from the trusted default-branch tip, before either step sources it. The job still runs no actions/checkout and no release-commit code. - R2: constrain both required attestations to the release-action signer workflow (--signer-workflow), and fail closed on a missing or malformed releaseCommit predicate (require a well-formed 40-hex sha) rather than skipping the check when data is absent/unshaped. - R3: mark scripts/verify-release-protections.sh and scripts/verify-ado-signing-protections.sh executable (100755) so the commands CONTRIBUTING.md documents work in a clean POSIX checkout. - R4: update docs/runbooks/release.md to show --signer-workflow on the gh attestation verify examples and distinguish the standard build-provenance check from the custom release-commit binding check. - Tests: add test/release-validation/release-asset-verify.test.ts (exercises the real verify_required_attestations helper against missing, malformed, mismatched, and unsigned predicates — all fail closed) and test/release-validation/publish-handoff.test.ts (pins the workflow materialization step and simulates the publish job's clean filesystem handoff end to end). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- R1 (HIGH): add an opt-in, redacted RV1 protocol-metadata capture hook
(ArmHttpClient.onObservation / ProtocolObservation / formatProtocolObservation
in packages/core/src/http.ts, createObservingRun in orchestrator.ts) that both
private adapters wire via CHAOS_STUDIO_RV_CAPTURE=1, printing one redacted
`RV-OBSERVATION {...}` line per ARM call. Rewrote docs/runbooks/release-validation.md
RV1/RV2 sections to document the real mechanism and an executable per-operation
RV2 negative-test matrix (replacing "with request logging on" and "rerun the
journey"). Added capture-path regression tests in packages/core/test/core/http.test.ts.
- R2 (HIGH): extend redaction.ts with generic per-secret-key-name key=value scrub
rules so free-text ARM/exception diagnostics get the same protection as
structured objects; added regression tests.
- R3 (MEDIUM): require --source-ref (trusted default-branch ref) on both
gh attestation verify calls in release-asset-verify.sh, failing closed when
DEFAULT_BRANCH is unset; mirrored in release.md and covered by new tests.
- R4 (MEDIUM): evaluate ALL verified attestation entries for a well-formed,
matching release-commit predicate instead of only the first array entry;
mirrored in release.md and covered by new order-independent/multi-entry tests.
- R5 (MEDIUM): inject the OIDC transport in host.test.ts instead of relying on a
real network call to vstoken.dev.azure.com failing as proof of correctness;
threaded an optional request-transport override through readArmServiceConnection/
fetchAzureDevOpsOidcToken in host.ts.
Verified: npx tsc -b --noEmit passes; npm test passes (907 pass, 2 pre-existing skips, 0 fail).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e evidence, pypi retry gate)
R1 (HIGH): Rebuild and commit fresh dist/github-action/index.js and
dist/azure-pipelines-task/index.js so shipped bundles match HEAD, including
RV capture wiring and redaction behavior. Restores byte-equality with the
clean-rebuild gate.
R2 (HIGH): The documented RV1/RV2 receipt procedure could not actually
produce required business-state/wire-shape evidence, and RV2 had no
independent single-operation invocation mechanism.
- Extend ProtocolObservation (packages/core/src/http.ts) with
businessState/startTime/endTime/statusField/startTimeField/endTimeField/
errorChannelsPresent, derived generically from the observed body (not
pinned contract constants) so real wire-shape drift surfaces as
undefined rather than being masked.
- Add scripts/lib/rv2-invoke.mjs, an operator CLI for independently
invoking a single ARM operation (validate/validationRead/execute/
runRead/runCancel) using the same secretless AzureCliCredential/
ArmHttpClient/URL-builder path the adapters use, printing an
RV-OBSERVATION line and an RV2-INVOKE-RESULT summary.
- Update docs/runbooks/release-validation.md to document populating
receipts from captured observations and using rv2-invoke.mjs as the
executable RV2 invocation mechanism.
- Add capture-to-receipt integration test
(test/release-validation/capture-to-receipt.test.ts) proving an
Rv1Transcript built ENTIRELY from captured ProtocolObservations passes
evaluateRv1, plus a negative case proving real wire-shape drift is
surfaced, not rubber-stamped.
- Add unit coverage in packages/core/test/core/http.test.ts for the new
observation fields, including drift and empty-body cases.
R3 (MEDIUM): publish-pypi relied on github-release's environment-protection
preflight, so a failed-job-only retry could reuse a stale pass if the pypi
environment's protections were weakened between attempts. Add a preflight
step to publish-pypi that independently re-verifies required reviewers,
prevent_self_review, can_admins_bypass, and the single default-branch
deployment policy on every attempt, failing closed if the verification
token or protections are unavailable. Add
test/release-validation/publish-pypi-retry-gate.test.ts covering the new
gate.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ch-policy name in retry preflight - publish-pypi's preflight now authenticates with a new PYPI_VERIFY_TOKEN environment secret on the pypi environment instead of ACTION_RELEASE_TOKEN, which lives only on release/mcp-release and is never shared across environments (would always be empty and fail the pypi preflight). - Add DEFAULT_BRANCH to publish-pypi's env and require the pypi deployment policy to be BOTH type=branch AND named exactly DEFAULT_BRANCH, closing a wildcard/non-default-branch acceptance gap. - Update CONTRIBUTING.md provisioning instructions and scripts/verify-release-protections.sh to check for and document PYPI_VERIFY_TOKEN on the pypi environment. - Add test coverage for the new credential wiring, exact policy-name enforcement, and doc/verifier alignment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…te OneBranch PR on runtime equality) R1: dist/github-action/index.js and dist/azure-pipelines-task/index.js were stale relative to a deterministic rebuild and omitted readObservedWireShape and its observation fields (businessState, statusField, errorChannelsPresent), so committed adapters could not emit the documented RV1 transcript and existing runtime-equality gates rejected the build. Regenerated both bundles via 'npm run build' and confirmed scripts/lib/verify-built-runtime.mjs now passes against HEAD. R2: .pipelines/OneBranch.PullRequest.yml rebuilt and staged the runtime without comparing it to the committed dist, so it could package different bytes than what GitHub CI / Official validation would accept for the same revision. Added a verify-built-runtime.mjs HEAD gate after the build and before staging, plus a release-handoff.test.ts assertion that this gate precedes packaging. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e-check containment, rollback freeze, RV sharing step) - R1 (core.ts:orchestrator.ts): readMode now rejects undefined/empty mode as a missing-required-input identifier error instead of silently applying DEFAULT_MODE. The documented validate-and-execute default is applied only by platform metadata (action.yml/task.json) when the input is genuinely omitted; an adapter-normalized empty string (e.g. a blank workflow/pipeline expression) now fails closed before any credential or ARM call, per FR16. Added core + adapter-parity tests proving zero credential/transport calls for undefined and empty mode, while existing tests confirm explicit modes are unaffected. - R2 (action-github/src/index.ts, task-azure-pipelines/src/index.ts): the CHAOS_STUDIO_SMOKE_CHECK offline smoke shortcut is now explicitly rejected when the process is running inside a real platform execution context (GITHUB_ACTIONS=true / TF_BUILD=True), so it can never be inherited into a real job/task and silently succeed without doing any real work. The release smoke harness's scrubbed child environment is unaffected. Added entrypoint tests for both the offline-pass and real-context-reject paths. - R3 (docs/runbooks/rollback-and-deprecation.md): the incident-only v1 repoint procedure now requires cancelling every pending/in-progress publication of the bad version before repointing, prohibits ever re-running/re-dispatching the withdrawn version, and permits resumption only via a newer approved patch — closing the gap where a pending or rerun publication could re-advance the floating major back to the withdrawn version after a manual repoint. - R4 (docs/runbooks/release-validation.md): corrected the first-private- preview sequence's inaccurate claim that publishing produces an extension already shared with RV organizations. Publishing a private extension only creates it in the publisher account; a new mandatory manual step now requires explicitly sharing ChaosStudioWorkspacesDev with each approved RV organization and confirming access before installation. Regenerated dist/github-action/index.js and dist/azure-pipelines-task/index.js from the corrected source. Full suite (930 passing, 2 pre-existing skips) and typecheck verified green. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- R1: exclude properties.resources from RV1 errorChannelsPresent so a complete authoritative run response passes capture-to-receipt validation; resources arrays are genuine evidence, not error channels. - R2: preserve redacted ARM diagnostic context (error code, actionable message, correlation/request IDs) when logging a best-effort cancel cleanup failure, without masking the original forward failure. - R3: accept ARM-valid Unicode resource-group names (e.g. rg-café) by widening the validation pattern to Unicode letters/marks/digits, and percent-decode Location path segments before identifier comparison and resource-ID construction so encoded pathname segments compare correctly against raw input names. - Add targeted tests for all three fixes, including both adapter paths for the Unicode resource-group case, and rebuild dist bundles. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…n-timeout scope, E1-INVENTORY historical framing
- examples/github: map chaos-studio step outputs (run-id, run-state,
run-resource-id) through step env: vars instead of interpolating
${{ steps.*.outputs.* }} directly into run: Bash blocks. No-wait mode
can surface an arbitrary unknown service status, so direct interpolation
is not reliably inert even inside double quotes.
- docs/ci-cd-integrations.md: correct the completion-timeout-seconds
description - it is an end-to-end deadline shared across validation,
execute-request acceptance, and run-completion polling, not scoped only
to waiting for run completion. The separate 300s cleanup budget is
unchanged.
- E1-INVENTORY.md: label the change-set reconciliation section as
historical E1 review bookkeeping and bind its reproduction commands to
the E1 commit range, since the paths are now committed rather than
untracked at current HEAD.
- packages/action-github/test/examples.test.ts: add a regression test
requiring every example's run: steps to reference chaos-studio outputs
via env vars rather than direct ${{ steps.*.outputs.* }} interpolation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Detect the installed Windows Bash path convention instead of assuming WSL, and keep the publication simulation compatible with Git for Windows without modifying user configuration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 91ab299b-ebcd-4f33-8dd9-495cb246e9f2
Keep failed release validation fail-closed while distinguishing client, repository, environment, and confirmed service causes before filing a service defect. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 91ab299b-ebcd-4f33-8dd9-495cb246e9f2
Make unreleased refs unmistakable, align dev task naming, secure the PR validation template, document precise cancellation semantics, and define input/output presence behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 91ab299b-ebcd-4f33-8dd9-495cb246e9f2
Document Azure macro absence semantics, platform-specific cancellation behavior, and a tested local build and Dev VSIX inspection path with explicit hosted-validation limits. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 91ab299b-ebcd-4f33-8dd9-495cb246e9f2
Use block scalars for colon-bearing Azure Pipelines scripts and add guards against reintroducing invalid plain script scalars. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 91ab299b-ebcd-4f33-8dd9-495cb246e9f2
Avoid literal unclosed GitHub expression openers inside run-block comments and guard release workflows against the same parser regression. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 91ab299b-ebcd-4f33-8dd9-495cb246e9f2
Add the authoritative Node24 task handler, retain Node20_1 for Azure DevOps Server compatibility, and set the minimum agent to the Node20 handler floor. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 91ab299b-ebcd-4f33-8dd9-495cb246e9f2
Document task.complete as the authoritative registered-task result and guard normal adapter failures from acquiring GitHub-style process exit semantics. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 91ab299b-ebcd-4f33-8dd9-495cb246e9f2
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 91ab299b-ebcd-4f33-8dd9-495cb246e9f2
Cristian Velez (kekivelez)
requested review from
abhiajm,
Brent Rusinow (brrusino) and
Nikhil Kaul (nikhilkaul1234)
as code owners
October 12, 2026 17:41
added 2 commits
October 15, 2026 16:24
Preserve the hardened release and TypeScript workflows while adopting current action annotations and chaos-study documentation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 91ab299b-ebcd-4f33-8dd9-495cb246e9f2
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 91ab299b-ebcd-4f33-8dd9-495cb246e9f2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds first-class CI/CD integrations for Chaos Studio workspace scenarios: a root GitHub Action and an Azure Pipelines task backed by one deterministic TypeScript core. Both surfaces support
validate-and-execute,validate-only, andexecute-only, including wait/no-wait execution, bounded polling, cancellation cleanup, stable outputs, WIF authentication, examples, least-privilege guidance, release validation, and governed publishing/rollback tooling.Type of change
Checklist
Invoke-Pester ./skillsandpytest mcp/tests)plugin.jsonversionbumped if user-visiblemcp/pyproject.tomlversionbumped if the Python package changedCHANGELOG.mdentry added under Unreleasedfeat(chaos-impact): …)Validation
Task@0resolution remains delegated to the standard ADO publishing platform contract.Validation evidence
The matrix summarizes the tested scope and keeps the remaining platform/release gates explicit.
Test topology
GitHub Actions
Hosted WIF validation reached
Succeededand exposed the documented validation outputs.Wait mode reported terminal completion. No-wait reported immediate acceptance and a separate exact-run observer established terminal success.
The controlled timeout case retained the original timeout while cleanup canceled the accepted run and restored the target.
Healthy GitHub log markers
For execution, an accepted/nonterminal result is not presented as terminal success. Healthy no-wait evidence shows the immediate state first, then a separately observed terminal state and target restoration. Healthy timeout evidence shows the original error after successful cleanup; cleanup does not turn the operation green.
Azure Pipelines adapter
The exact packaged adapter bundle executed on Microsoft-hosted ADO agents with real WIF and Chaos calls. This interim wrapper exercises adapter logic and native
##vsooutput processing; it does not claim task-catalogTask@0resolution or a task-owned timeline.Execute-only consumed six safe output values downstream, omitted
validation-state, and preserved the prepared validation fingerprint.No-wait evidence separates immediate acceptance from later exact-run completion.
The timeout case intentionally left the task/build red, preserved the original timeout, reached service state
Canceled, and restored the target. Azure task-lib failure is authoritative throughtask.complete; a child process exit code of zero is not a successful task result.Healthy ADO log markers and triage guidance
Reviewers should distinguish:
task.complete result=Failed) from process exit semantics;Correlation and request IDs are retained for service-side triage. Raw credentials, signed URLs, resource IDs, and private ADO links are excluded from the public evidence.
Healthy log markers used during validation
Reviewers can use the attached excerpts as a triage reference. Healthy runs show:
Security and supply chain
AdoExtensionSignandAdoExtensionVerifywith a signed SHA-256 handoff.PublishAzureDevOpsExtension@5uses an AzureRM WIF publishing connection and pre-signedupdateTasksVersion: falsepackages.node_modules.Release gates
This PR does not publish an extension or create the public GitHub
v1tag. Production release still requires the protected signing KeyCode/environment, protected variable-group values, WIF publisher role onAzureChaosStudio, a live signed Official receipt with Marketplace validation, a current RV receipt, and exact shared release tag/commit promotion.Related issues
Implements the planned first-class GitHub Actions and Azure Pipelines CI/CD surface for Chaos Studio workspace scenarios.