Skip to content

Publish what the reference implementation ships: re-vendor 0.7.3-0.7.5, add the 0.7.6 preview, gate drift on every PR - #50

Merged
fas89 merged 6 commits into
open-data-protocol:mainfrom
fas89:spec/publish-076-revendor
Oct 5, 2026
Merged

fas89 merged 6 commits into
open-data-protocol:mainfrom
fas89:spec/publish-076-revendor

Conversation

@fas89

@fas89 fas89 commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator

What this changes

The spec site publishes schema files that differ from the ones the reference implementation ships under the same $id. This PR makes them the same. It also replaces the drift check, which had stopped running, with one that runs on every pull request.

Published schemas that were wrong

  • fluid-schema-0.7.5.json was a pre-GA snapshot. It matched forge-cli v0.10.0, taken while 0.7.5 was still a preview. GA 0.7.5 adds the pgvector platform and format, binding.vectorConfig, the Iceberg catalog location fields and the Redshift Serverless / Kinesis location fields. So a pgvector contract that fluid validate accepts as "schema v0.7.5" failed the schema served at that URL.
  • 0.7.3, 0.7.4 and 0.7.5 described Lake Formation admins as additive. It is authoritative. PutDataLakeSettings replaces the account's admin list, so following the published text can remove administrators, including the principal running the apply. forge-cli corrected the description in v0.16.3. For 0.7.3 and 0.7.4 this description is the only difference.
  • fluid-schema-0.7.6.json was not published. It is a preview in forge-cli, and its $id points at this site, where it returned 404.

Changes

  • Schemas. Re-vendors 0.7.3, 0.7.4 and 0.7.5, and vendors 0.7.6 as a preview. Each file is byte-for-byte the copy in forge-cli v0.18.1 (commit 93e78d4), with no hand edits; 0.7.2–0.7.6 are identical to v0.17.0. 0.7.1 is deliberately not re-vendored, because it diverged historically; the docs PR records that.
  • generate-docs.py. It renders every synced version, not only the highest one. "Latest stable" is now separate from "highest version", so the 0.7.6 preview is never linked as latest. --check also accounts for every file under schema/ and specs/. A file that is neither generated output nor listed with its sha256 in the new frozen map fails the check, and so does a frozen file whose bytes changed.
  • Renderer lock. The renderer is hash-locked (scripts/requirements-docs.lock, installed with --require-hashes). An unpinned json-schema-for-humans had been enough to fail the docs check with no schema change.
  • Corpus.
    • New tests/0.7.6/ cases, following tests/README.md "Adding a version".
    • Cases for the GA 0.7.5 additions. Re-vendoring had dropped mutation coverage to 39.4%, below the 40% floor; it is now 41.0% for 0.7.5 and 42.0% for 0.7.6.
  • Drift check. schema-sync.yml had been disabled_inactivity since 31 August, after failing every scheduled run on 0.7.5. It also compared against forge-cli main, not a release. The new drift job in conformance.yml:
    • runs on every pull request;
    • compares against the release pinned in scripts/schema-versions.json (upstream.ref / upstream.commit), so the result is deterministic and a re-vendor shows up in the PR diff;
    • is joined by an advisory drift-latest job (continue-on-error) that reports when forge-cli has a newer release, so a new upstream release cannot turn every PR here red.
  • Hardening from the security review.
    • The GitHub token is sent with add_unredirected_header.
    • Resolved tags are validated before use in URLs.
    • deploy-docs.yml grants pages: write / id-token: write to the deploy job only.
    • .gitattributes keeps schema/, specs/ and schema-diffs/ byte-exact on every checkout.
  • Process docs.
    • CONTRIBUTING.md says the drift job fails a hand-edited schema PR; whether that blocks a merge is a branch-protection setting.
    • It also gives the pinned re-vendor steps.
    • GOVERNANCE.md records the stable / preview status model.

Tested

Run locally from this branch, with jsonschema[format] 4.26, the locked renderer and data-product-forge 0.18.1:

conformance/run.py              fluid conformance: 1061 cases, 1061 passed, 0 failed, 0 unexpectedly passed, 0 known failures
tests/meta_test.py              meta-test: 50/50 checks passed
scripts/check-compat.py         every release keeps the promise -- no document valid under a version stops validating under its successor
mutation_coverage --min 40      0.7.5 covered 432 (41.0%), 0.7.6 covered 477 (42.0%)
conformance/check_reference.py  1061 cases, 1057 agree, 0 disagree, 4 optional behaviour(s) not implemented
generate-docs.py --check        specs/ is current for 0.7.2..0.7.6; every other file under schema/ and specs/ is frozen and unchanged
generate-schema-diffs.py --check  schema-diffs/ is current (12 diff files)
check-schema-drift.py           every synced schema matches the reference release   (live, pinned v0.18.1)
npm run docs:build              success

Showing the new checks can fail. In a temp copy:

  • adding specs/0.7.6/extra.html fails the check;
  • appending one byte to specs/0.7.1/fluid-spec.html fails it;
  • swapping add_unredirected_header for add_header fails the meta-test.

These are now permanent cases in tests/meta_test.py.

Live check. The built site was served under /fluid/ and opened in a browser:

  • schema/fluid-schema-0.7.6.json resolves at its $id path, and specs/0.7.6/fluid-spec.html renders.
  • The served 0.7.5 contains pgvector and the authoritative admins text.
  • No console errors.

Security review. A dedicated review found no HIGH issues. Its MEDIUM and LOW findings that belong to this PR are fixed here. gitleaks reports no leaks in the new commits.

Prior art

  • One immutable, self-resolving $id per released version or public draft, with mutable aliases never used as $id. This follows the versioned-schema identity practice discussed in mcpdesc-specification#47 and JSON Schema's own release practice. The status model labels 0.7.6 as a preview until it is promoted.
  • Hash-locked tooling: pip-tools --generate-hashes.

For the maintainers

These are not changed here; they are settings or decisions for you.

  • Required checks. Branch protection requires only gates can fail, corpus and backward compatibility. The new schemas match the reference release, generated files are current and docs links carry the site base checks report but do not block. Add them if you want them to.
  • Link Check workflow. It is disabled_inactivity. Its per-PR base-prefix check now runs in conformance.yml; the weekly external-link job is still off.
  • Third-party script in the generated HTML. specs/<v>/fluid-spec.html loads a Font Awesome script from use.fontawesome.com without SRI. That is unchanged from main; the renderer's js_offline template would self-host it.
  • Action pinning. Actions use floating major tags. Pinning to SHAs with Dependabot would harden this.
  • Upstream items for forge-cli. Each schema's root examples[0] fails its own schema, and the column type pattern starts with (?i), which is not ECMA-262. Both are tracked for forge-cli, not edited here.

Commits are DCO signed-off.

fas89 added 6 commits October 5, 2026 02:38
…5, add 0.7.6 preview, gate drift per PR

Schemas (byte for byte from forge-cli v0.18.1, identical to v0.17.0):
- 0.7.3, 0.7.4: Lake Formation `admins` described as authoritative, not
  additive (the only difference from the published copies).
- 0.7.5: the published copy was forge-cli v0.10.0's pre-GA snapshot. GA adds
  the pgvector platform, pgvector_table format, binding.vectorConfig, and the
  Iceberg catalog-profile, Redshift Serverless and Kinesis location fields.
- 0.7.6: vendored as a PREVIEW so its $id resolves. 0.7.5 stays latest stable.
- 0.7.1 is deliberately not re-vendored.

Corpus: tests/0.7.5/binding.json pins the GA 0.7.5 additions (re-vendoring
alone took mutation coverage to 39.4%, below the 40% gate). tests/0.7.6 ports
every 0.7.5 group and adds packaging, consumers, consumes-pinning
(dependentRequired, which a Draft 7 validator ignores), binding-security,
lifecycle-expire-bucketpolicy and masking-aggparams. Coverage: 0.7.5 41.0%,
0.7.6 42.0%. check_reference.py with data-product-forge 0.18.1: 0 disagree.

Tooling and CI:
- scripts/schema-versions.json records synced versions, latest stable and
  previews; latest stable is no longer "highest version number".
- generate-docs.py renders every synced version (--all/--version), has
  --check and --print-latest-stable, runs json-schema-for-humans in-process
  with exact pins (scripts/requirements-docs.txt) and no footer timestamp.
  specs/0.7.2-0.7.6 regenerated.
- generate-schema-diffs.py gains --check; diff-0.7.5-to-0.7.6 added,
  diff-0.7.4-to-0.7.5 regenerated, both with notes.
- scripts/check-schema-drift.py compares byte for byte against forge-cli's
  latest release (resolved at run time), and checks bundled versions and the
  preview set. It replaces schema-sync.yml, which compared against main and
  was disabled for inactivity; the check now runs in conformance.yml on every
  pull request, with the specs/schema-diffs currency check and the docs
  base-prefix link check (moved from the disabled link-check.yml).
- check-compat.py defaults to the promised range (0.7.1 on), so a bare run
  passes; --all-history keeps the pre-promise pairs. meta_test.py proves the
  drift check and the default range can fail.
- CONTRIBUTING.md, tests/README.md, GENERATOR_DOCS.md: "checked by", manual
  re-vendor steps, Draft 7 question answered (forge-cli 0.15.0, #582).

Signed-off-by: Speculator55005 <50082482+fas89@users.noreply.github.com>
… naming, failure start date

Signed-off-by: Speculator55005 <50082482+fas89@users.noreply.github.com>
…note, list two known schema ambiguities

Signed-off-by: Speculator55005 <50082482+fas89@users.noreply.github.com>
…t implements in the version-status section

Signed-off-by: Speculator55005 <50082482+fas89@users.noreply.github.com>
… hash-lock the docs renderer

The drift check compared the vendored schemas with forge-cli's latest
release, resolved at run time, so a new upstream release turned every
pull request red until someone re-vendored. It now compares with the
release recorded in scripts/schema-versions.json (upstream.ref and
upstream.commit) and fetches by the commit. Moving to a newer release is
a change to the pin, reviewed in a pull request. `--latest` is the
separate question "has forge-cli released since?"; the conformance
workflow runs it as an advisory job whose step cannot fail the run.
CONTRIBUTING.md now says what is enforced (the drift job fails the pull
request; it does not claim branch protection rejects it) and gives the
re-vendor steps for the pin.

check-schema-drift.py also sends the token as an unredirected header, so
a redirect away from api.github.com cannot carry it, and validates every
ref and commit before it reaches a URL or a printed command.

generate-docs.py --check now fails on any file under specs/ that is not
generated output of a synced version, and on any file under schema/ that
is not a synced schema, unless it is listed with its sha256 in a new
"frozen" map in schema-versions.json. The map covers the non-synced
schemas (0.0.1 to 0.7.1) and their HTML pages, and is verified offline
(`--check-files` runs just that part). Before, those files could change,
or a stray file could be added, with every gate green. meta_test.py
proves the new failures.

deploy-docs.yml grants contents: read at the top, pages: read to the
build job, and pages: write plus id-token: write to the deploy job only.
The docs Python dependencies are hash-locked in
scripts/requirements-docs.lock (pip-compile --generate-hashes) and
installed with --require-hashes in both workflows that install them;
generate-docs.py --check fails if the lock and requirements-docs.txt
disagree.

Signed-off-by: Speculator55005 <50082482+fas89@users.noreply.github.com>
The drift gate and the frozen sha256 map compare raw bytes, so an
end-of-line conversion on a Windows checkout would fail them with no
content change.

Signed-off-by: Speculator55005 <50082482+fas89@users.noreply.github.com>
@fas89
fas89 merged commit 24a1eef into open-data-protocol:main Oct 5, 2026
7 checks passed
@fas89
fas89 deleted the spec/publish-076-revendor branch October 5, 2026 09:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant