Skip to content

Use compiler JSON metadata in the setup runtime - #75997

Open
pelikhan wants to merge 15 commits into
mainfrom
pelikhan/remove-queue-yaml-helper
Open

pelikhan wants to merge 15 commits into
mainfrom
pelikhan/remove-queue-yaml-helper

Conversation

@pelikhan

@pelikhan pelikhan commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Remove work_queue_yaml.cjs, the replacement parser dependency, and the rejected native metadata helper and its provisioning/build wiring.
  • Keep the Actions runtime in actions/setup/. JavaScript reads bounded JSON metadata/manifest headers from .lock.yml files through the shared lock-header reader; it does not parse the workflow body or install tools to interpret author syntax.
  • Enrich the existing compiler manifest with versioned, digest-bound artifact metadata and scoped model-pin projections. The root compiler handles authored frontmatter; engine-generated copies use JSON frontmatter without rewriting author files.
  • Migrate work-queue verification, agent/skill extraction, Pi configuration, model auditing, runtime imports, interpolation, and literal/provenance restoration to the compiler JSON contract.
  • Load executing metadata at the immutable workflow revision, merge matching caller/source records with the existing scoped GitHub client, and transport the catalog through aw_info.json. Missing or stale compiler metadata fails explicitly; sources must provide precompiled metadata, not a runtime parsing fallback.
  • Add Actions-wide boundary guards and isolated deployment/embedding checks. The developer-only fixture compiler remains under scripts/, outside the deployed Actions runtime.
  • Merge main, preserve concurrent branch updates, regenerate workflow locks, and clean the workspace cache.

Validation

  • make agent-report-progress passed: build, Go/custom/JavaScript lint, schema freshness, impacted Go tests, workflow drift checks, 2,416 setup JavaScript tests, and 721 ESLint-factory tests.
  • Recompiled all 365 workflows successfully.
  • Refreshed and verified compiler string-API golden fixtures.
  • Focused regressions cover immutable execution revisions, caller/private-source metadata, metadata bounds, JSON-only reads, Pi delegation, deployment dependency closure, and prompt-literal safety.

No Go compiler, native metadata executable, or parsing package is provisioned by the setup runtime.

pelikhan and others added 2 commits November 8, 2026 14:02
Use the pinned yaml package across runtime consumers and deploy its locked dependency during setup.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use compiler JSON lock headers for worker provisioning and a separate metadata helper for Markdown frontmatter operations. Remove runtime npm parser installation and prevent parser imports and new Go implementations in setup.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pelikhan pelikhan changed the title Remove vendored work queue YAML parser Remove setup JavaScript YAML parsing and add runtime boundary guards Nov 8, 2026
Deny native helper and subprocess imports while verifying both lock JSON headers and worker provisioning against a deliberately malformed workflow body.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pelikhan
pelikhan marked this pull request as ready for review November 9, 2026 00:19
Copilot AI auto review requested due to automatic review settings November 9, 2026 00:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved defects break action builds and setup on default runners, while metadata parsing can omit declared tool restrictions and corrupt protected placeholders.

10 open issues
What changed in this PR

This PR moves Markdown metadata parsing out of gh-aw’s setup JavaScript and limits work-queue verification to bounded compiler JSON headers.

Changes:

  • Introduces a separate Go metadata helper with a JSON-only JavaScript bridge.
  • Replaces worker workflow-body parsing with compiler-manifest validation.
  • Adds runtime boundary guards and updates provisioning, build integration, and regression tests.
File Description
specs/​work-queue/​verify_native.py Removes the deleted parser from source fingerprints.
specs/​work-queue/​verify_native_test.py Updates fingerprint expectations.
scripts/​agent-report-progress.sh Adds validation for the separate helper module.
Makefile Adds helper build, vet, and test targets.
eslint-factory/​src/​eslint-config-severity.test.ts Removes the obsolete parser-ignore assertion.
eslint-factory/​eslint.config.cjs Removes the vendored-parser lint exclusion.
actions/​setup/​setup.sh Provisions the helper and removes parser deployment entries.
actions/​setup/​session_parsers.go Updates embedded JavaScript dependencies.
actions/​setup/​README.md Documents the metadata processing boundary.
actions/​setup/​js/​workflow_tool_policy_alignment.test.cjs Migrates metadata and manifest readers.
actions/​setup/​js/​workflow_pull_request_migration.test.cjs Uses the metadata bridge in workflow checks.
actions/​setup/​js/​work_queue_yaml.test.cjs Deletes legacy parser tests.
actions/​setup/​js/​work_queue_stress_workflow.test.cjs Migrates workflow parsing to the helper.
actions/​setup/​js/​work_queue_provisioning.cjs Verifies workers through bounded manifest headers.
actions/​setup/​js/​work_queue_provisioning_checks.cjs Tests manifest validation and parser-free provisioning.
actions/​setup/​js/​work_queue_dispatch.test.cjs Adds worker manifests to dispatch fixtures.
actions/​setup/​js/​vitest.config.mjs Registers helper compilation before JavaScript tests.
actions/​setup/​js/​smoke_work_queue_workflows.test.cjs Migrates smoke workflow metadata checks.
actions/​setup/​js/​smoke_claude_sub_agents.test.cjs Reads manifests through the bounded header helper.
actions/​setup/​js/​skill_inventory.cjs Delegates skill metadata reading to the helper.
actions/​setup/​js/​setup_sh_file_lists.test.cjs Tests isolated helper provisioning and deployment lists.
actions/​setup/​js/​setup_runtime_boundary.test.cjs Guards setup against parser and Go implementation additions.
actions/​setup/​js/​scripts/​work_queue_yaml/​package.json Deletes parser-bundle build dependencies.
actions/​setup/​js/​scripts/​work_queue_yaml/​entry.mjs Deletes the parser bundling entry point.
actions/​setup/​js/​scripts/​work_queue_yaml/​build.mjs Deletes parser bundling tooling.
actions/​setup/​js/​prompt_provenance_regressions.test.cjs Migrates provenance assertions to the helper.
actions/​setup/​js/​prompt_literal_values.test.cjs Migrates literal-restoration assertions.
actions/​setup/​js/​prompt_literal_values.cjs Delegates source-preserving scalar restoration.
actions/​setup/​js/​pi_subagent_extension.test.cjs Removes the SDK frontmatter parser stub.
actions/​setup/​js/​pi_subagent_config.test.cjs Uses real Markdown metadata fixtures.
actions/​setup/​js/​pi_subagent_config.cjs Reads Pi agent metadata through the bridge.
actions/​setup/​js/​pi_models_json.test.cjs Replaces parser mocks with Markdown fixtures.
actions/​setup/​js/​model_pin_check.cjs Delegates frontmatter document parsing.
actions/​setup/​js/​markdown_metadata.cjs Adds the bounded JSON process bridge.
actions/​setup/​js/​haiku_printer_workflow.test.cjs Migrates workflow parsing assertions.
actions/​setup/​js/​fuzz_template_substitution_harness.test.cjs Migrates scalar round-trip assertions.
actions/​setup/​js/​extract_inline_sub_agents.test.cjs Migrates agent frontmatter assertions.
actions/​setup/​js/​extract_inline_sub_agents.cjs Delegates agent naming and metadata extraction.
actions/​setup/​js/​engine_subagent_models.test.cjs Migrates model metadata assertions.
actions/​setup/​js/​engine_subagent_models.cjs Uses helper-provided metadata and scalar offsets.
actions/​setup/​js/​daily_squid_image_scan.test.cjs Uses bounded manifest-header reads.
actions/​setup/​js/​daily_report_portfolio.test.cjs Migrates compiled workflow checks.
actions/​setup/​js/​daily_mcp_concurrency_workflow.test.cjs Migrates workflow frontmatter checks.
actions/​setup/​js/​copilot_sdk_session.test.cjs Migrates source and compiled workflow checks.
actions/​setup/​js/​copilot_harness.cjs Delegates agent discovery and model scalar extraction.
actions/​setup/​js/​cache_memory_workflows.test.cjs Migrates workflow metadata assertions.
actions/​markdown-metadata/​vitest_setup.mjs Builds the helper for JavaScript tests.
actions/​markdown-metadata/​setup.sh Builds the deployed helper executable.
actions/​markdown-metadata/​main.go Implements bounded parsing and source-preserving edits.
actions/​markdown-metadata/​main_test.go Tests offsets, validation, naming, and restoration.
actions/​markdown-metadata/​go.sum Records pinned dependency checksums.
actions/​markdown-metadata/​go.mod Defines the separate Go module.
.gitignore Excludes generated helper executables.

Review effort: Max (auto)

Comment thread actions/markdown-metadata/go.mod Resolved
Comment thread actions/markdown-metadata/main.go Resolved
Comment thread actions/markdown-metadata/main.go Resolved
Comment thread actions/markdown-metadata/main.go Resolved
Comment thread actions/setup/js/model_pin_check.cjs Resolved
Comment thread actions/setup/setup.sh Resolved
Comment thread Makefile Resolved
Comment thread actions/markdown-metadata/main.go Resolved
Comment thread actions/markdown-metadata/setup.sh Resolved
@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot

Copilot assignment requested for the following outstanding work:

  1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
  2. Fix failing check actions-build (FAILURE): https://github.com/github/gh-aw/actions/runs/42539551036/job/128911307418.
  3. Fix failing check actions-build (FAILURE): https://github.com/github/gh-aw/actions/runs/42531644233/job/128887011134.
  4. Fix failing check JS Tests (shard 5/8) (FAILURE): https://github.com/github/gh-aw/actions/runs/42539551133/job/128911308144.
  5. Fix failing check JS Tests (shard 5/8) (FAILURE): https://github.com/github/gh-aw/actions/runs/42531644235/job/128887010333.
  6. Fix failing check Local Git and concurrent workers (FAILURE): https://github.com/github/gh-aw/actions/runs/42531644255/job/128886924550.
  7. Fix failing check pre_activation (FAILURE): https://github.com/github/gh-aw/actions/runs/42539551267/job/128911203765.
  8. Fix failing check pre_activation (FAILURE): https://github.com/github/gh-aw/actions/runs/42539551200/job/128911203707.
  9. Fix failing check pre_activation (FAILURE): https://github.com/github/gh-aw/actions/runs/42539551033/job/128911203345.
  10. Fix failing check pre_activation (FAILURE): https://github.com/github/gh-aw/actions/runs/42539551037/job/128911203331.
  11. Fix failing check pre_activation (FAILURE): https://github.com/github/gh-aw/actions/runs/42539551055/job/128911202959.
  12. Fix failing check pre_activation (FAILURE): https://github.com/github/gh-aw/actions/runs/42539551007/job/128911202454.

Once you start work, post this work summary as a PR comment without the leading Copilot mention, including the Sous-chef head, work, and state lines below, to record that the assignment is running. Apply the necessary fixes on this PR's existing branch, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved; stop when only human review or CI remains. Do not merge or trigger CI.

Sous-chef requested-head: 515fcb7
Sous-chef requested-work: 0ba78f688aad93971538194802da12a38cec064ef823d357719f32e19c3725a1 4154465ffeb3a1a232db436c8c0ba02d6499231eb8cf1e49c83c6e3ee5bf6ca5 78cff96338665a216aed174395db53962090e0167d19046c48b8353c0696ba8f e2b1dcdd50909d11ea908683e207665058785302844e162780d5fdb2120a01e0
Sous-chef requested-state: e84b22beac4aff607d300c913ec0322bada25c64146a0a8304743612d843108e

Generated by 👨🍳 PR Sous Chef · pi · haiku45 · ~7 AIC · ⌖ ~15 AIC · ⊞ 1.2K · ◷
Comment /souschef to run again

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Hey @pelikhan 👋 — thanks for this architectural refactor! The changes are well-organized and thoroughly tested. A few items to confirm before merging:

  • New Go dependency — The PR adds github.com/goccy/go-yaml v1.19.2 to actions/markdown-metadata/go.mod. The provisioning trade-off section explains the rationale (Go 1.23+ requirement, pinned module cache/proxy access, failures stop setup). Confirm the team is aligned on this new runtime requirement and any CI/deployment implications.
  • Large removal — The 6,198-line deletion of vendored YAML parsing and build tooling is a significant cleanup. The validation section shows comprehensive test coverage (859 setup JavaScript, 721 ESLint-factory tests), which is excellent, but flag this for maintainer review to ensure no edge cases are missed.
  • Boundary guards — The setup boundary tests rejecting parser imports and APIs are well-designed. Confirm these constraints are sufficient for the project's security/stability goals.

The PR is well-documented and the changes are cohesive. If the team has signed off on the dependency and deployment model, this looks ready to move forward.

Generated by ✅ Contribution Check · copilot · routed: haiku55 xhigh · ~21 AIC · ⌖ ~2 AIC · ⊞ 5.4K · ◷

Copilot AI commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

@copilot

Copilot assignment requested for the following outstanding work:...

Assignment started. I’ll merge the latest main, inspect and address the listed CI failures and outstanding review feedback, then run focused validation before publishing changes.

Sous-chef requested-head: 515fcb7
Sous-chef requested-work: 0ba78f688aad93971538194802da12a38cec064ef823d357719f32e19c3725a1 4154465ffeb3a1a232db436c8c0ba02d6499231eb8cf1e49c83c6e3ee5bf6ca5 78cff96338665a216aed174395db53962090e0167d19046c48b8353c0696ba8f e2b1dcdd50909d11ea908683e207665058785302844e162780d5fdb2120a01e0
Sous-chef requested-state: e84b22beac4aff607d300c913ec0322bada25c64146a0a8304743612d843108e

Copilot AI and others added 5 commits November 9, 2026 01:08
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
…-yaml-helper

Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pelikhan pelikhan changed the title Remove setup JavaScript YAML parsing and add runtime boundary guards Use compiler JSON metadata in the setup runtime Nov 9, 2026
Keep worker fixtures aligned with JSON-only route verification, isolate synthetic harness workspaces from CI checkout agents, and initialize developer-only metadata tooling before timed runtime assertions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Comment thread actions/setup/js/markdown_metadata.cjs Resolved
pelikhan and others added 3 commits November 8, 2026 17:56
Treat Markdown separators as prose, retain inline artifact boundaries, and provision compiler-owned metadata for caller-attribution, analytics, and AW policy fixtures without weakening runtime checks.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Require compiler provenance for JSON metadata, preserve trusted derived records, handle Markdown fences safely, and normalize Windows boundary paths. Regenerate workflow locks after merging origin/main.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep strict runtime provenance enabled while supplying compiler records for staged model-resolution test agents. Preserve all existing model availability assertions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
pelikhan and others added 2 commits November 8, 2026 19:20
Integrate the new skip-if-decision workflows and default-branch health sampling updates, regenerating compiled locks from merged sources.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve structured session tool results, merge the default Gemini run-info fixes, and regenerate compiler outputs while retaining the JSON-only setup runtime.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pelikhan

pelikhan commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

Regarding the original comment

The listed failures ran against older revisions; their removed-helper/bootstrap and stale-fixture causes are fixed, and latest main has now been merged with regenerated locks and combined local validation. Final-head CI confirmation remains separate; no workflows were dispatched or rerun.

@pelikhan

pelikhan commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

Regarding the original comment

The proposed runtime Go dependency and native helper have been removed: authored syntax is handled by the root compiler, while actions/setup consumes JSON headers without installing parser tooling. Boundary guards and deployment checks protect that separation, but maintainer review remains appropriate and team sign-off is not assumed.

@pelikhan

pelikhan commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

Regarding the original comment

The assignment kickoff is already recorded, so no duplicate kickoff is needed. The historical failures have been investigated, latest main integrated, and the combined changes validated locally for CI and human review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants