Skip to content
This repository was archived by the owner on Sep 30, 2026. It is now read-only.

Document the v3 entities beyond collections, and how to catch silent drops - #24

Open
jonico wants to merge 1 commit into
Postman-Devrel:mainfrom
jonico:docs/v3-git-sync-corrections
Open

jonico wants to merge 1 commit into
Postman-Devrel:mainfrom
jonico:docs/v3-git-sync-corrections

Conversation

@jonico

@jonico jonico commented Sep 22, 2026 •

Copy link
Copy Markdown

Why

collection-schema-v3 covers postman/collections/ and postman/environments/. Everything else an agent writes on disk is undocumented: datasets, local mock servers, workspace globals, specs, documents, SDKs and .postman/resources.yaml. None of those use $kind — a dataset uses type: dataset, a mock uses version: 1, an environment uses name + values — so an agent generalising from the collection format produces a file that is silently wrong.

The failures are quiet rather than loud:

  1. Unrecognised keys are dropped, not rejected. A request containing metod:, event: or dataMode: runs with no method override and no scripts, and reports green with 0 assertions.
  2. postman collection lint does not run the unrecognised-field rules at all — it returns errorCount: 0, warningCount: 0 for all of the above, and -f warning does not change that. postman workspace lint reports each one as FMT206 (FMT208 in definition.yaml), and environment lint / globals lint report their own as ENV007.

So the guidance tells agents to finish with workspace lint, not collection lint.

What this changes

File Change
skills/collection-schema-v3/SKILL.md Addendum below the existing content: every non-collection entity, the filename suffixes the CLI enforces, per-kind settings blocks, the type traps, and how to verify
skills/collection-schema-v3/reference/other_entities.md New. Complete field lists per entity and per request kind — auth, bodies, scripts, all eight request kinds, messages, examples, datasets, mocks, environments, globals, specs, documents, SDKs, the workspace manifest
skills/collection-schema-v3/reference/other_protocols.md One correction, below
skill description Widened so the skill also triggers for datasets, mocks, globals, specs and documents — it currently names only collections and environments, so it does not fire for any of them

The existing SKILL.md body is unchanged. Everything is additive except the correction.

The one correction to existing text

reference/other_protocols.md documents type on userPrompts / systemPrompts items:

- `userPrompts` — array of `{id, value, timestamp, active, type: "text"}`.

The defined field is contentType. Both spellings lint clean, so nothing warns you — type is unrecognised, so it is accepted, stripped, and the content type is lost.

Measured

153 one-shot authoring prompts across every entity type and request kind, plus 4 covering script language, exotic LLM settings and identifier format. One subagent per batch, no linting, no iteration, no reading any existing collection. Scored by the CLI's own validators (collection|environment|globals|spec lint, workspace lint, dataset get) plus semantic checks for the defects lint strips. Guidance SHA-256 verified unchanged at both ends of the run.

Guidance Always-on tokens Trials Failures Rate
Current skill alone ~2,000 43 20 46.5%
This PR ~6,350 157 0 0%

0 failures in 157 bounds the true rate under 1.91% at 95% confidence (rule of three). The on-demand other_entities.md is a further ~4,370 tokens, paid only when read.

The token cost is deliberate. A half-size variant containing every one of the same rules scored 3.0%, because rules compressed into dense multi-clause sentences got missed. The extra tokens buy prominence, not content, which is why the text uses callouts and one rule per paragraph.

Design notes, corpus, oracle and results: https://gist.github.com/jonico/be778cafdec3966c2ee84543734c31db
Line-by-line comparison against the current text: https://gist.github.com/jonico/d1d105b0f8a1609399697b7c7ba0ae89

Notes on the content

  • No executable and no schema dump. Every field fact is written out, so the text works in a sandbox with no tooling.
  • Nothing about unreleased capability. The text documents only what the shipped CLI accepts.
  • Every claim was verified against CLI 1.56.1, not read off a schema.
  • .postman/resources.yaml is deliberately not specified. postman init writes it and workspace push / mock push maintain it, so the guidance says not to hand-author it and gives only what is needed to read or repair one.

Also documented, undocumented until now

  • FMT209 is symmetric: a url carrying ?a=1 with no queryParams warns, and a queryParams block whose url has no query string warns. It is a warning, so a clean errorCount hides it.
  • postman spec lint reports governance findings at ERROR severity — a valid spec with no security block fails with Security field is not defined, issue type Governance. A non-zero error count there does not mean the spec is malformed.
  • postman collection run needs no login; pass --no-report-events so the CLI does not attempt the post-run report upload, which is the only part that reaches the cloud.

Relationship to postmanlabs/postman-plugin

These skills arrived here via #23 (sync/skills-hooks-from-postman-plugin), and skills/collection-schema-v3/ is byte-identical to the upstream copy. The same change is filed upstream as postmanlabs/postman-plugin#22; whichever lands first, the next sync reconciles them.

This PR also folds in the bootstrap/SKILL.md fix filed upstream as postmanlabs/postman-plugin#23: step 3.1 read .postman/resources.yaml for localResources, a key that exists nowhere, and branched on the bindings rather than on workspace.id. Since postman init writes only the workspace stanza, an initialised repo that has not pushed yet has no cloudResources and was being sent back through init.

This repo has no manifest.json, so that part of the upstream change does not apply. All seven Plugin Checks jobs pass.

@jonico

jonico commented Sep 23, 2026 •

Copy link
Copy Markdown
Author

Carries the same changes as postmanlabs/postman-plugin#22 and #23. Each verified against CLI 1.56.1:

  • script language permits exactly text/javascript; language: javascript is a hard FMT015
  • the LLM settings object has thirteen keys — streamTools, repetitionPenalty and contextWindow are now listed
  • postman collection run needs no login. The run is local; without --no-report-events the CLI attempts a post-run report upload, and only that reaches the cloud. Verified: local collection against a local mock, exit 0, 1/1/1 requests/test-scripts/assertions, no session. Consistent with this repo's "local commands need no login" rule.
  • dataset and mock id placeholders are now valid v4 values
  • $kind is the only required field except on mcp-request, which also requires transport
  • bootstrap step 3.1 routes on workspace.id rather than on the bindings, since postman init writes only the workspace stanza

This repo has no manifest.json, so that part of the upstream change does not apply. The bootstrap edit is applied in place rather than copied from upstream, whose heading levels would fail this repo's markdownlint. All seven Plugin Checks jobs pass.

…drops

The synced collection-schema-v3 skill covers postman/collections/ and
postman/environments/. Everything else an agent writes on disk — datasets,
local mock servers, workspace globals, specs, documents, SDKs and
.postman/resources.yaml — is undocumented, and none of those use $kind. Each
has its own discriminator (type: dataset, version: 1, name + values), so an
agent generalising from the collection format produces a file that is
silently wrong.

Unrecognised keys are dropped rather than rejected, and `postman collection
lint` does not run the unrecognised-field rules at all: a request containing
`metod:`, `event:` or `dataMode:` returns errorCount 0 and warningCount 0,
then runs with no method override and no scripts while reporting green.
`postman workspace lint` reports each as FMT206 (FMT208 in definition.yaml),
and environment/globals lint report their own as ENV007.

Adds:
- a SKILL.md addendum covering every non-collection entity, the filename
  suffixes the CLI enforces, per-kind settings blocks, the type traps, and
  how to verify
- reference/other_entities.md with complete field lists per entity and per
  request kind
- a widened skill description so the skill also triggers for datasets,
  mocks, globals, specs and documents

Corrects:
- reference/other_protocols.md documented `type` on userPrompts and
  systemPrompts items; the field is `contentType`, and `type` lints clean
  before being stripped
- bootstrap step 3.1 read resources.yaml for `localResources`, a key that
  exists nowhere, and branched on the bindings rather than on `workspace.id`;
  since `postman init` writes only the `workspace` stanza, an initialised
  repo that has not pushed yet has no `cloudResources` and was being sent
  back through init

Measured over 157 one-shot authoring prompts scored by the CLI's own
validators: the current skill alone fails 46.5% of the exotic subset, this
text fails 0 of 157.

Co-Authored-By: Claude <noreply@anthropic.com>
@jonico
jonico force-pushed the docs/v3-git-sync-corrections branch from 46b056e to 9e837df Compare September 23, 2026 10:45
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant