Repository navigation
Publish what the reference implementation ships: re-vendor 0.7.3-0.7.5, add the 0.7.6 preview, gate drift on every PR - #50
Merged
Conversation
…5, add 0.7.6 preview, gate drift per PR Schemas (byte for byte from forge-cli v0.18.1, identical to v0.17.0): - 0.7.3, 0.7.4: Lake Formation `admins` described as authoritative, not additive (the only difference from the published copies). - 0.7.5: the published copy was forge-cli v0.10.0's pre-GA snapshot. GA adds the pgvector platform, pgvector_table format, binding.vectorConfig, and the Iceberg catalog-profile, Redshift Serverless and Kinesis location fields. - 0.7.6: vendored as a PREVIEW so its $id resolves. 0.7.5 stays latest stable. - 0.7.1 is deliberately not re-vendored. Corpus: tests/0.7.5/binding.json pins the GA 0.7.5 additions (re-vendoring alone took mutation coverage to 39.4%, below the 40% gate). tests/0.7.6 ports every 0.7.5 group and adds packaging, consumers, consumes-pinning (dependentRequired, which a Draft 7 validator ignores), binding-security, lifecycle-expire-bucketpolicy and masking-aggparams. Coverage: 0.7.5 41.0%, 0.7.6 42.0%. check_reference.py with data-product-forge 0.18.1: 0 disagree. Tooling and CI: - scripts/schema-versions.json records synced versions, latest stable and previews; latest stable is no longer "highest version number". - generate-docs.py renders every synced version (--all/--version), has --check and --print-latest-stable, runs json-schema-for-humans in-process with exact pins (scripts/requirements-docs.txt) and no footer timestamp. specs/0.7.2-0.7.6 regenerated. - generate-schema-diffs.py gains --check; diff-0.7.5-to-0.7.6 added, diff-0.7.4-to-0.7.5 regenerated, both with notes. - scripts/check-schema-drift.py compares byte for byte against forge-cli's latest release (resolved at run time), and checks bundled versions and the preview set. It replaces schema-sync.yml, which compared against main and was disabled for inactivity; the check now runs in conformance.yml on every pull request, with the specs/schema-diffs currency check and the docs base-prefix link check (moved from the disabled link-check.yml). - check-compat.py defaults to the promised range (0.7.1 on), so a bare run passes; --all-history keeps the pre-promise pairs. meta_test.py proves the drift check and the default range can fail. - CONTRIBUTING.md, tests/README.md, GENERATOR_DOCS.md: "checked by", manual re-vendor steps, Draft 7 question answered (forge-cli 0.15.0, #582). Signed-off-by: Speculator55005 <50082482+fas89@users.noreply.github.com>
… naming, failure start date Signed-off-by: Speculator55005 <50082482+fas89@users.noreply.github.com>
…note, list two known schema ambiguities Signed-off-by: Speculator55005 <50082482+fas89@users.noreply.github.com>
…t implements in the version-status section Signed-off-by: Speculator55005 <50082482+fas89@users.noreply.github.com>
… hash-lock the docs renderer The drift check compared the vendored schemas with forge-cli's latest release, resolved at run time, so a new upstream release turned every pull request red until someone re-vendored. It now compares with the release recorded in scripts/schema-versions.json (upstream.ref and upstream.commit) and fetches by the commit. Moving to a newer release is a change to the pin, reviewed in a pull request. `--latest` is the separate question "has forge-cli released since?"; the conformance workflow runs it as an advisory job whose step cannot fail the run. CONTRIBUTING.md now says what is enforced (the drift job fails the pull request; it does not claim branch protection rejects it) and gives the re-vendor steps for the pin. check-schema-drift.py also sends the token as an unredirected header, so a redirect away from api.github.com cannot carry it, and validates every ref and commit before it reaches a URL or a printed command. generate-docs.py --check now fails on any file under specs/ that is not generated output of a synced version, and on any file under schema/ that is not a synced schema, unless it is listed with its sha256 in a new "frozen" map in schema-versions.json. The map covers the non-synced schemas (0.0.1 to 0.7.1) and their HTML pages, and is verified offline (`--check-files` runs just that part). Before, those files could change, or a stray file could be added, with every gate green. meta_test.py proves the new failures. deploy-docs.yml grants contents: read at the top, pages: read to the build job, and pages: write plus id-token: write to the deploy job only. The docs Python dependencies are hash-locked in scripts/requirements-docs.lock (pip-compile --generate-hashes) and installed with --require-hashes in both workflows that install them; generate-docs.py --check fails if the lock and requirements-docs.txt disagree. Signed-off-by: Speculator55005 <50082482+fas89@users.noreply.github.com>
The drift gate and the frozen sha256 map compare raw bytes, so an end-of-line conversion on a Windows checkout would fail them with no content change. Signed-off-by: Speculator55005 <50082482+fas89@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
The spec site publishes schema files that differ from the ones the reference implementation ships under the same
$id. This PR makes them the same. It also replaces the drift check, which had stopped running, with one that runs on every pull request.Published schemas that were wrong
fluid-schema-0.7.5.jsonwas a pre-GA snapshot. It matched forge-cli v0.10.0, taken while 0.7.5 was still a preview. GA 0.7.5 adds the pgvector platform and format,binding.vectorConfig, the Iceberg catalog location fields and the Redshift Serverless / Kinesis location fields. So a pgvector contract thatfluid validateaccepts as "schema v0.7.5" failed the schema served at that URL.adminsas additive. It is authoritative.PutDataLakeSettingsreplaces the account's admin list, so following the published text can remove administrators, including the principal running the apply. forge-cli corrected the description in v0.16.3. For 0.7.3 and 0.7.4 this description is the only difference.fluid-schema-0.7.6.jsonwas not published. It is a preview in forge-cli, and its$idpoints at this site, where it returned 404.Changes
v0.18.1(commit93e78d4), with no hand edits; 0.7.2–0.7.6 are identical to v0.17.0. 0.7.1 is deliberately not re-vendored, because it diverged historically; the docs PR records that.generate-docs.py. It renders every synced version, not only the highest one. "Latest stable" is now separate from "highest version", so the 0.7.6 preview is never linked as latest.--checkalso accounts for every file underschema/andspecs/. A file that is neither generated output nor listed with its sha256 in the newfrozenmap fails the check, and so does a frozen file whose bytes changed.scripts/requirements-docs.lock, installed with--require-hashes). An unpinned json-schema-for-humans had been enough to fail the docs check with no schema change.tests/0.7.6/cases, followingtests/README.md"Adding a version".schema-sync.ymlhad beendisabled_inactivitysince 31 August, after failing every scheduled run on 0.7.5. It also compared against forge-climain, not a release. The newdriftjob inconformance.yml:scripts/schema-versions.json(upstream.ref/upstream.commit), so the result is deterministic and a re-vendor shows up in the PR diff;drift-latestjob (continue-on-error) that reports when forge-cli has a newer release, so a new upstream release cannot turn every PR here red.add_unredirected_header.deploy-docs.ymlgrantspages: write/id-token: writeto the deploy job only..gitattributeskeepsschema/,specs/andschema-diffs/byte-exact on every checkout.CONTRIBUTING.mdsays the drift job fails a hand-edited schema PR; whether that blocks a merge is a branch-protection setting.GOVERNANCE.mdrecords the stable / preview status model.Tested
Run locally from this branch, with jsonschema[format] 4.26, the locked renderer and data-product-forge 0.18.1:
Showing the new checks can fail. In a temp copy:
specs/0.7.6/extra.htmlfails the check;specs/0.7.1/fluid-spec.htmlfails it;add_unredirected_headerforadd_headerfails the meta-test.These are now permanent cases in
tests/meta_test.py.Live check. The built site was served under
/fluid/and opened in a browser:schema/fluid-schema-0.7.6.jsonresolves at its$idpath, andspecs/0.7.6/fluid-spec.htmlrenders.pgvectorand the authoritativeadminstext.Security review. A dedicated review found no HIGH issues. Its MEDIUM and LOW findings that belong to this PR are fixed here. gitleaks reports no leaks in the new commits.
Prior art
$idper released version or public draft, with mutable aliases never used as$id. This follows the versioned-schema identity practice discussed in mcpdesc-specification#47 and JSON Schema's own release practice. The status model labels 0.7.6 as a preview until it is promoted.--generate-hashes.For the maintainers
These are not changed here; they are settings or decisions for you.
gates can fail,corpusandbackward compatibility. The newschemas match the reference release,generated files are currentanddocs links carry the site basechecks report but do not block. Add them if you want them to.disabled_inactivity. Its per-PR base-prefix check now runs inconformance.yml; the weekly external-link job is still off.specs/<v>/fluid-spec.htmlloads a Font Awesome script fromuse.fontawesome.comwithout SRI. That is unchanged from main; the renderer'sjs_offlinetemplate would self-host it.examples[0]fails its own schema, and the column type pattern starts with(?i), which is not ECMA-262. Both are tracked for forge-cli, not edited here.Commits are DCO signed-off.