Merge pull request #660 from beyondnetcode/fix/gt-628-engine-disclosure #2320
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Documentation Validation | |
| concurrency: | |
| group: ${{ github.workflow }}-${{ github.ref }} | |
| cancel-in-progress: true | |
| on: | |
| workflow_dispatch: | |
| pull_request: | |
| push: | |
| branches: [main, develop] | |
| permissions: | |
| contents: read | |
| pages: write | |
| id-token: write | |
| jobs: | |
| # GT-476: the semantic tracking guard runs on every PR and on push to | |
| # main/develop so the gap board can't silently drift. Kept as a focused, | |
| # dependency-free job (git history only) so it stays fast and does not drag | |
| # the full doc-publish pipeline below into every pull request. | |
| tracking-guard: | |
| name: Validate semantic tracking (guard) | |
| runs-on: ubuntu-latest | |
| steps: | |
| - name: Checkout | |
| uses: actions/checkout@v7 | |
| with: | |
| fetch-depth: 0 | |
| - name: Setup Node.js | |
| uses: actions/setup-node@v7 | |
| with: | |
| node-version: "20" | |
| - name: Validate semantic tracking | |
| run: node .harness/scripts/ci/08-validate-tracking.mjs | |
| validate: | |
| name: Validate documentation | |
| # GT-563: this job used to carry `if: github.event_name == 'workflow_dispatch'`, | |
| # so on every push and pull_request it was SKIPPED -- and a skipped job reports | |
| # neutral, which GitHub rolls up as a green "Documentation Validation" check. The | |
| # workflow therefore reported SUCCESS on commits whose docs did not validate. A | |
| # check that cannot fail is not a check; it is a false signal, and it is worse than | |
| # a red one because nobody goes looking for it. | |
| # | |
| # The gate is removed: validation now runs on push and pull_request, and it BLOCKS. | |
| # It is deliberately not marked continue-on-error -- a non-blocking gate would | |
| # reproduce the exact defect this fixes, just with more convincing logs. | |
| # | |
| # The old comment justified the gate as keeping "the full documentation-publish | |
| # pipeline" out of PRs. That rationale did not hold: this workflow has no publish | |
| # step. The job is npm ci plus a handful of node validators. | |
| runs-on: ubuntu-latest | |
| steps: | |
| - name: Checkout | |
| uses: actions/checkout@v7 | |
| with: | |
| fetch-depth: 0 | |
| - name: Setup Node.js | |
| uses: actions/setup-node@v7 | |
| with: | |
| node-version: "20" | |
| cache: 'npm' | |
| cache-dependency-path: package-lock.json | |
| - name: Install Dependencies | |
| run: npm ci | |
| - name: Validate docs | |
| run: node .harness/scripts/ci/01-validate-docs.mjs | |
| # Here rather than beside their subjects, and the reason is the defect itself. | |
| # `17-validate-knowledge-intake.mjs` runs in knowledge-intake.yml, but that | |
| # workflow is `paths:`-filtered to `KI-*`/`SRC-*` files and the intake schema — | |
| # so a pull request that changes the GUARD does not trigger it, and a self-test | |
| # placed next to the guard would be unreachable in exactly the case it is for. | |
| # The three `knowledge-okf-*` guards run in no workflow at all. All four are | |
| # hermetic, so the required job is where they produce a signal. | |
| - name: Self-tests for the knowledge intake and OKF guards | |
| run: | | |
| node --test .harness/scripts/ci/17-validate-knowledge-intake.test.mjs | |
| node --test .harness/scripts/knowledge-okf-project.test.mjs | |
| node --test .harness/scripts/knowledge-okf-standard-watch.test.mjs | |
| node --test .harness/scripts/knowledge-okf-precommit-guard.test.mjs | |
| - name: Check bilingual parity | |
| run: node .harness/scripts/ci/04-check-bilingual-parity.mjs | |
| # GT-620's negative fixtures for the language heuristic the step above depends | |
| # on — including the two cases where it must DECLINE to judge. They ran in no | |
| # workflow, so the heuristic that closed GT-620 was itself unguarded. | |
| - name: Self-test for the bilingual language heuristic | |
| run: node --test .harness/scripts/ci/suites/bilingual-suite.test.mjs | |
| # GT-702. The step above asks whether the ES file EXISTS, matches heading counts | |
| # and is Spanish — all properties of the files as they stand, none of which can | |
| # see that the two DISAGREE. On 2026-08-16 the EN catalog was corrected to 266 | |
| # violation literals while its ES twin kept asserting 251, and parity stayed | |
| # green over the contradiction for the whole day. | |
| # | |
| # Runs against this branch's own range, so historical one-sided commits do not | |
| # fail somebody else's PR. Depends on the `fetch-depth: 0` checkout above: with a | |
| # shallow clone there is no range to read, and the guard's `assertScanned` call | |
| # makes that a failure rather than a silent pass. | |
| - name: Check bilingual pairs moved together | |
| run: node .harness/scripts/ci/66-validate-bilingual-sync.mjs --since origin/${{ github.base_ref || 'main' }} | |
| - name: Self-test for the bilingual sync guard | |
| run: node .harness/scripts/ci/66-validate-bilingual-sync.mjs --self-test | |
| - name: Validate semantic tracking | |
| run: node .harness/scripts/ci/08-validate-tracking.mjs | |
| # Two files, one subject: both import `08-validate-tracking.mjs` directly. The | |
| # 17 cases in the first are what GT-629 added when the guard learned to refuse an | |
| # open row with no criteria at all; neither file was referenced anywhere. | |
| - name: Self-tests for the semantic tracking guard | |
| run: | | |
| node --test .harness/scripts/ci/08-validate-tracking.test.mjs | |
| node --test .harness/scripts/validate-tracking.test.mjs | |
| - name: Reconcile maturity evidence | |
| run: node .harness/scripts/ci/09-reconcile-maturity.mjs --check | |
| # The inventory summaries are the source every doc quotes when it says how many | |
| # ADRs, rulesets and schemas the corpus carries -- and until now nothing checked | |
| # them. `07-generate-inventories.mjs` appears in exactly one other place in the | |
| # repository: its own unit test in ci-cd.yml. The generator ran in no workflow, | |
| # and in the local hook it hung off `infraChanged` (compose files, helm, the | |
| # workflow directory) -- never off `src/rulesets/` or the ADR directory, which | |
| # are the only two things that can change the numbers. | |
| # | |
| # The result was a published tally that had been stale since 2026-08-16: 141/181 | |
| # against a measured 142/182, quoted onward by the README as "412 rules". This | |
| # step runs the generator in --check mode, so a ruleset or ADR added without | |
| # regenerating the summaries fails here instead of aging into the front page. | |
| - name: Check inventory summaries match the corpus | |
| run: node .harness/scripts/ci/07-generate-inventories.mjs --check | |
| - name: Self-test for the inventory generator | |
| run: node --test .harness/scripts/ci/07-generate-inventories.test.mjs | |
| # 21 cases over `09-reconcile-maturity.mjs`, including the GT-576 rule that a | |
| # capability may not be marked Validated on an ADR citation alone and the GT-596 | |
| # ISO/IEC 33020 threshold rule. Both were closed with "ships with a negative | |
| # self-test" as the evidence; the self-test ran nowhere. | |
| - name: Self-test for the maturity reconciler | |
| run: node --test .harness/scripts/reconcile-maturity.test.mjs | |
| # GT-627: the generated ADR-conformance corpus fell SEVEN rulesets behind | |
| # its own generator, and six of the seven were security standards | |
| # (ADR-0119..0124: SSRF prevention, input validation, shell-execution | |
| # safety, timing-safe comparison, credential management, API hardening) — | |
| # accepted decisions with no conformance ruleset at all. Nothing detected | |
| # it for as long as it lasted, because the generator was only ever run by | |
| # hand. It has always supported `--check`; nobody ran it. | |
| # | |
| # This lives in a REQUIRED check on purpose. The same drift in the skill | |
| # registry (GT-424) went unnoticed for weeks behind a guard that existed | |
| # and was not enforced. | |
| - name: ADR ruleset corpus matches its generator | |
| run: node .harness/scripts/generate-adr-rulesets.mjs --check | |
| - name: Self-tests for the ADR ruleset generator | |
| run: node --test .harness/scripts/generate-adr-rulesets.test.mjs | |
| # GT-598: `native-evaluability-snapshot.json` says of itself that it is a | |
| # capture, and until now nothing captured it — it was written by hand, and | |
| # it drifted. It still declared `documentation-only: 129` after Core moved | |
| # to 136 (the same seven ADR rulesets the step above exists for), and its | |
| # guard could not see that, because it compared the snapshot against six | |
| # numbers hardcoded in the test: the same six the snapshot contained. | |
| # | |
| # Drift here is not contained. `build-iso-5055-mapping.mjs` stamps | |
| # nativeEvaluability onto all 388 rows of the mapping FROM this file, so a | |
| # stale class is laundered into the larger artifact and overstates the | |
| # handler backlog. The order of these two steps is the order of the chain. | |
| - name: Native evaluability snapshot is a faithful capture of Core's triage | |
| run: node src/rulesets/standards/capture-native-evaluability-snapshot.mjs --check | |
| - name: ISO/IEC 5055 mapping guard | |
| run: node --test src/rulesets/standards/iso-5055-mapping.test.mjs | |
| # GT-583: the `capability operation schemas` link in the chain below boots | |
| # the mcp-server DI graph through ts-node to ask ToolRegistryService to | |
| # describe itself. mcp-server resolves `@beyondnet/evolith-core-domain` | |
| # through the workspace symlink, whose `exports` map points at `dist/` — | |
| # so with only `npm ci` behind it the guard dies on MODULE_NOT_FOUND for | |
| # `dist/capabilities/capability-operations.js` and reports the link STALE. | |
| # It passed in `test-mcp-server` and on every developer machine because | |
| # both had a built `dist/` (gitignored) sitting there already. `npm run | |
| # build` further down this job is too late — the guard runs here. | |
| - name: Build workspace dependencies for the runtime-derived guards | |
| run: | | |
| npm run build --workspace @beyondnet/evolith-contracts | |
| npm run build --workspace @beyondnet/evolith-core-domain | |
| npm run build --workspace @beyondnet/evolith-agent-runtime | |
| npm run build --workspace @beyondnet/evolith-sdk | |
| npm run build --workspace @beyondnet/evolith-core | |
| npm run build --workspace @beyondnet/evolith-infra-providers | |
| # GT-630: the derived artifacts have a dependency ORDER — the executive | |
| # summary is built FROM the maturity reconciliation, which is built from | |
| # the board. Generate the summary first and it captures a value the | |
| # reconciler then moves; each artifact's own --check still passes at that | |
| # moment, so the suite goes green locally and this job goes red. That cost | |
| # three red required checks on 2026-07-28 and the order was written down | |
| # nowhere. It is now data, and this step walks it. | |
| - name: Derived artifacts are current, in dependency order | |
| run: node .harness/scripts/ci/46-validate-derived-artifact-order.mjs --verbose | |
| - name: Self-test for the derived-artifact order guard | |
| run: node --test .harness/scripts/ci/46-validate-derived-artifact-order.test.mjs | |
| # GT-632: `40-validate-path-literals` scans string LITERALS, so it is blind | |
| # to a path that is BUILT — `path.join(ctx.corePath, 'sdk', 'cli', …)`. | |
| # Twelve of those survived the `src/` move: one resolved the compiled ABAC | |
| # policy at the pre-refactor path, and because the evaluator denies | |
| # fail-closed when it cannot load, EVERY MCP tool was refused in production | |
| # on a clean checkout; three more probed files that had moved and reported | |
| # false NEGATIVES, which nothing notices because a false negative looks | |
| # exactly like a clean run. The specs agreed with the code and neither | |
| # agreed with the repository, so no existing suite could catch it. | |
| - name: Built (not written) paths resolve to something that exists | |
| run: node .harness/scripts/ci/47-validate-joined-paths.mjs --verbose | |
| - name: Self-test for the joined-path guard | |
| run: node --test .harness/scripts/ci/47-validate-joined-paths.test.mjs | |
| # GT-587. The OpenTelemetry GenAI conventions are Development-status, and the | |
| # Core emits their attribute names as plain LITERALS (it must not import an | |
| # observability SDK — rule HXA-05). Literals do not move when upstream moves, so | |
| # a rename there turns Evolith's "standard" telemetry back into a private | |
| # vocabulary with nothing going red. This is the check that goes red. | |
| - name: Validate the pinned OpenTelemetry semconv revision | |
| run: node .harness/scripts/ci/51-validate-semconv-pin.mjs --verbose | |
| - name: Self-test for the semconv pin guard | |
| run: node --test .harness/scripts/ci/51-validate-semconv-pin.test.mjs | |
| # GT-597. The gate itself runs inside `openssf-scorecard.yml`, where the | |
| # Scorecard JSON exists; only its logic is tested here. That logic is the | |
| # difference between a weekly score that is published and a weekly score | |
| # that is CHECKED, so it is tested on its failure surface: a dropped check, | |
| # a vanished check, an inconclusive -1, an unseeded baseline, and a SARIF | |
| # file handed to it by mistake must each be red. | |
| - name: Self-test for the Scorecard regression gate | |
| run: node --test .harness/scripts/ci/52-validate-scorecard-regression.test.mjs | |
| - name: Validate machine contracts | |
| run: node .harness/scripts/ci/10-validate-contract-conformance.mjs | |
| - name: Validate product documentation synchronization | |
| run: node .harness/scripts/ci/11-validate-product-docs.mjs | |
| - name: Check surface parity matrix | |
| run: node .harness/scripts/ci/24-check-surface-parity.mjs | |
| - name: Validate topology rule coverage | |
| run: node .harness/scripts/ci/26-validate-topology-rule-coverage.mjs | |
| # The guard above has run in this required check since it was written; its own | |
| # self-test ran in no job at all, and had been red on both `develop` and `main` | |
| # since `98a20dca` without anyone seeing it. That commit moved `rulesets/` under | |
| # `src/` and updated the path this test's fixture WRITES TO, but not the relative | |
| # path written INSIDE it — so the fixture's satellite reference stopped resolving | |
| # and the coverage assertion it exists to make was never reached. | |
| # | |
| # A validator whose own tests nobody runs is the defect this board keeps | |
| # recording, so it goes next to its guard, in the same required check, exactly | |
| # like every other self-test in this job. | |
| - name: Self-test for the topology rule coverage guard | |
| run: node --test .harness/scripts/generate-rule-coverage.test.mjs | |
| - name: Validate harness path literals | |
| run: node .harness/scripts/ci/39-validate-harness-paths.mjs | |
| # Re-armed: disabled in 42cf18a8's mass "disable until fixed" sweep and never | |
| # restored. Its matrix path was dead (ENOENT); both it and the matrix's own | |
| # stale module paths are fixed, so it now validates 3 surfaces for real. | |
| - name: Validate surface compatibility matrix | |
| run: node .harness/scripts/ci/20-validate-surface-compatibility.mjs | |
| # All six cases were red — and had never once run. `setup()` wrote the fixture | |
| # matrix to `reference/core/control-center/`, one directory above the `audits/` | |
| # path the guard reads, into a directory it also never created: every case died | |
| # in setup with ENOENT. Four of the six are negative cases asserting the guard's | |
| # specific failure text, so re-arming the step above (see the note on it) was | |
| # done with its self-test in this state. | |
| - name: Self-test for the surface compatibility guard | |
| run: node --test .harness/scripts/ci/20-validate-surface-compatibility.test.mjs | |
| # The governance suite (16 validators) previously existed ONLY behind | |
| # `ci-runner.mjs`, whose sole caller was `.husky/pre-commit` — and that hook | |
| # exits before reaching it whenever there is no TTY (every agent, script and | |
| # CI invocation). So 16 validators were inert: they could fail for weeks and | |
| # tell nobody. Running them here makes them independent of anyone's local | |
| # git config, TTY or shell. | |
| - name: Run governance validator suite | |
| run: node .harness/scripts/ci-runner.mjs governance | |
| # Suite exploratoria cross-superficie: ejerce CLI, MCP y REST de VERDAD | |
| # (48 operaciones, 66 invocaciones) y compara sus respuestas. | |
| # | |
| # Es hermetica: su cwd es el satelite temporal que ella construye, asi que | |
| # no scaffoldea en la raiz del repo ni depende de lo que dejaran corridas | |
| # anteriores. Y los how-to derivados ya no embeben el VEREDICTO de la | |
| # evaluacion, que variaba entre maquinas y hacia infalseable el chequeo | |
| # anti-drift: primero se resto campo a campo (`rulesChecked`, `issues`, las | |
| # listas de violaciones, `confidence`) -- una denylist que solo podia crecer, | |
| # porque nombraba la volatilidad por la que ya nos habian mordido -- y ahora | |
| # el generador proyecta el sobre a su FORMA e imprime literal solo lo que | |
| # esta en su allowlist. Un campo nuevo ya no puede romper este paso. | |
| - name: Build workspace packages | |
| run: npm run build | |
| - name: Populate CLI bundled ruleset corpus | |
| run: npm run copy-rulesets --workspace @beyondnet/evolith-cli | |
| - name: Cross-surface exploration suite | |
| run: npm run test:exploration | |
| # GT-563: the steps below REPORT, they do not validate. This job runs no tests and | |
| # produces no coverage data, and COVERAGE_REPORT.md is untracked, so on a PR the | |
| # dashboard --check can only ever report "COVERAGE_REPORT.md is missing" and the | |
| # upload can only ever attach nothing. They stay on workflow_dispatch, where a | |
| # coverage run can precede them and they measure something real. | |
| # | |
| # This is a deliberately narrow gate: every VALIDATOR above runs on push and | |
| # pull_request and blocks. Only report generation is manual. | |
| - name: Check coverage dashboard is up to date | |
| if: github.event_name == 'workflow_dispatch' | |
| run: node .harness/scripts/coverage-dashboard.mjs --check | |
| # Deliberately NOT carrying the `workflow_dispatch` condition above it. That gate | |
| # exists because generating the report needs a coverage run; these two need | |
| # nothing but a tmp directory, and inheriting the condition would wire them into | |
| # a job they almost never run in — which is the defect being closed, not a fix | |
| # for it. | |
| # | |
| # Five of the seven coverage-dashboard cases were red. `coverage-dashboard.mjs` | |
| # wrote its report with no `mkdirSync` for the parent, so off the real tree it | |
| # printed a correct 100% report and then died on an unhandled ENOENT. Only the | |
| # positive cases failed; the negative ones exit before the write and passed, | |
| # which is what made it look like a scoring bug rather than a crash. | |
| - name: Self-tests for the coverage dashboard | |
| run: | | |
| node --test .harness/scripts/coverage-dashboard.test.mjs | |
| node --test .harness/scripts/lib/generated-doc-exclusions.test.mjs | |
| - name: Upload coverage report | |
| uses: actions/upload-artifact@v7 | |
| if: github.event_name == 'workflow_dispatch' | |
| with: | |
| name: coverage-report | |
| path: COVERAGE_REPORT.md | |
| retention-days: 30 | |
| - name: Report bilingual coverage | |
| if: github.event_name == 'workflow_dispatch' | |
| run: node .harness/scripts/bilingual-coverage.mjs |