Skip to content

Add experimental-API enforcement tooling for TS, Python, Go, and Rust#1719

Open
stephentoub wants to merge 2 commits into
mainfrom
stephentoub/super-bassoon
Open

Add experimental-API enforcement tooling for TS, Python, Go, and Rust#1719
stephentoub wants to merge 2 commits into
mainfrom
stephentoub/super-bassoon

Conversation

@stephentoub

Copy link
Copy Markdown
Collaborator

Why

The C# and Java SDKs mark unstable APIs with [Experimental] / @CopilotExperimental, and the toolchain produces a real diagnostic that a consumer must explicitly suppress to opt in. The TypeScript, Python, Go, and Rust SDKs only had doc comments, so nothing actually stopped a user from depending on an experimental API by accident. This brings all four languages up to parity with a mechanism that the compiler or linter legitimately flags.

Approach

Each language gets the idiomatic equivalent of [Experimental], and codegen emits the marker/gate onto generated experimental methods so the surface stays in sync with the schema:

  • TypeScript - new standalone @github/eslint-plugin-copilot-sdk package with a type-aware no-experimental-api rule that flags references to any symbol whose JSDoc carries @experimental. Opt in per call site with // eslint-disable-next-line @github/copilot-sdk/no-experimental-api.
  • Python - an @experimental decorator that raises ExperimentalWarning under a configurable warn / error / ignore policy (also driven by the COPILOT_EXPERIMENTAL env var), plus allow_experimental() and set_experimental_policy(). Codegen applies it to generated experimental methods.
  • Go - a go/analysis analyzer (go/copilotexperimental/, nested module) that flags references to Experimental:-documented symbols via exported analysis Facts. Suppress with a same-line //nolint:copilotexperimental.
  • Rust - a new experimental Cargo feature. Codegen gates experimental RPC methods by conditional visibility: pub when the feature is enabled, and a pub(crate) fallback (identical body) when it is not. External consumers hit a hard E0624 compile error unless they opt in with features = ["experimental"], while the SDK's own internal callers keep compiling.

Notes for reviewers

  • The two large generated diffs (rust/src/generated/rpc.rs, python/copilot/generated/rpc.py) are produced by the codegen changes in scripts/codegen/*.ts, not hand-edited.
  • Rust CI (.github/workflows/rust-sdk-tests.yml) now passes test-support,experimental to the clippy and test steps. The SDK's own tests and the manual_tool_resume example exercise experimental RPCs directly, so they need the feature enabled, just as the C#/Java test projects suppress the experimental diagnostic. cargo doc --all-features already covers it, and the manual_tool_resume example is declared with required-features = ["experimental"].
  • Verified: Rust lib compiles with the feature both on and off, the gate produces the intended E0624 for an external consumer, and the regenerated file passes rustfmt. Go, Python, and the TS plugin test suites all pass.

Bring TypeScript, Python, Go, and Rust in line with the C# ([Experimental])
and Java (@CopilotExperimental) SDKs, where consuming an experimental API
produces a real diagnostic the user must explicitly suppress. Previously these
four languages only had doc comments.

- TypeScript: new @github/eslint-plugin-copilot-sdk package with a type-aware
  no-experimental-api rule keyed off the @experimental JSDoc tag.
- Python: @experimental decorator emitting ExperimentalWarning with a
  configurable warn/error/ignore policy (COPILOT_EXPERIMENTAL); emitted by
  codegen onto generated experimental methods.
- Go: go/analysis analyzer (go/copilotexperimental) that flags references to
  Experimental:-documented symbols; suppress with //nolint:copilotexperimental.
- Rust: experimental Cargo feature; codegen gates experimental RPC methods via
  conditional visibility (pub with the feature, pub(crate) fallback without) so
  external callers get a hard compile error while internal SDK code keeps
  working.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@stephentoub stephentoub requested a review from a team as a code owner June 18, 2026 13:47
Copilot AI review requested due to automatic review settings June 18, 2026 13:47


@overload
def experimental(obj: _T) -> _T: ...


@overload
def experimental(*, since: str | None = ...) -> Callable[[_T], _T]: ...

import pytest

import copilot

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds enforceable “experimental API” opt-in mechanisms for the TypeScript, Python, Go, and Rust SDKs (bringing them closer to the C# / Java [Experimental] / @CopilotExperimental behavior) by introducing language-idiomatic enforcement and updating codegen so experimental markers stay in sync with the schema.

Changes:

  • Rust: Gate experimental generated RPC methods behind a new experimental Cargo feature via conditional visibility, and enable that feature in Rust CI for lint/tests.
  • Python: Add a runtime @experimental decorator + policy controls and have codegen decorate experimental generated RPC methods; add unit tests and public re-exports.
  • TypeScript & Go: Introduce a new ESLint plugin rule for TS consumers and a go/analysis vettool analyzer for Go consumers; document usage.
Show a summary per file
File Description
scripts/codegen/rust.ts Emits feature-gated conditional visibility for experimental generated Rust RPC methods.
scripts/codegen/python.ts Imports and applies @experimental to generated Python RPC methods when marked experimental.
rust/README.md Documents the experimental feature and how to opt in to experimental Rust APIs.
rust/Cargo.toml Adds experimental feature and marks example(s) requiring it.
python/test_experimental.py Adds unit tests validating runtime experimental gating behavior.
python/copilot/generated/rpc.py Regenerated RPC client with @experimental decorators on experimental methods.
python/copilot/experimental.py Implements Python runtime experimental decorator, warning type, and policy controls.
python/copilot/init.py Re-exports experimental gating utilities in the public copilot package API.
nodejs/README.md Documents experimental API enforcement via the companion ESLint plugin rule.
nodejs/eslint-plugin/test/no-experimental-api.test.js Adds tests for the ESLint rule behavior against fixture consumers.
nodejs/eslint-plugin/test/fixtures/tsconfig.json TS project config used by the rule’s type-aware tests.
nodejs/eslint-plugin/test/fixtures/sdk.ts Fixture SDK surface with @experimental JSDoc tags.
nodejs/eslint-plugin/test/fixtures/re-export.ts Fixture re-export used by consumer aliasing test scenarios.
nodejs/eslint-plugin/test/fixtures/consumer-suppressed.ts Fixture demonstrating per-use suppression.
nodejs/eslint-plugin/test/fixtures/consumer-stable.ts Fixture verifying stable API usage is not flagged.
nodejs/eslint-plugin/test/fixtures/consumer-experimental.ts Fixture verifying experimental usages are flagged.
nodejs/eslint-plugin/test/fixtures/consumer-aliased.ts Fixture verifying aliased experimental references are flagged.
nodejs/eslint-plugin/rules/no-experimental-api.js Implements the type-aware ESLint rule that flags @experimental symbol references.
nodejs/eslint-plugin/README.md Documents installation/configuration and suppression for the ESLint plugin.
nodejs/eslint-plugin/package.json Defines the new standalone ESLint plugin package metadata/deps.
nodejs/eslint-plugin/package-lock.json Locks dependencies for the standalone ESLint plugin package.
nodejs/eslint-plugin/index.js Exposes plugin entry point, rule registration, and recommended configs.
nodejs/eslint-plugin/eslint.config.js Flat-config used for plugin’s own tests/fixtures.
nodejs/eslint-plugin/.gitignore Ignores local node_modules for the plugin package.
go/README.md Documents running the new copilotexperimental analyzer via go vet -vettool.
go/copilotexperimental/testdata/src/sdk/sdk.go Analyzer testdata: “SDK” declarations marked experimental via doc markers.
go/copilotexperimental/testdata/src/consumer/consumer.go Analyzer testdata: consumer references + suppression directive.
go/copilotexperimental/README.md Documents analyzer install/run and suppression behavior.
go/copilotexperimental/go.sum Dependency checksums for the analyzer’s nested module.
go/copilotexperimental/go.mod Nested module definition for the analyzer.
go/copilotexperimental/experimental.go Implements the analyzer using exported analysis Facts + suppression scanning.
go/copilotexperimental/experimental_test.go Runs analysistest suite for the analyzer.
go/copilotexperimental/cmd/copilotexperimental/main.go Provides a go vet-compatible singlechecker entry point.
.github/workflows/rust-sdk-tests.yml Enables experimental feature during Rust clippy and test runs.

Copilot's findings

Files not reviewed (1)
  • nodejs/eslint-plugin/package-lock.json: Generated file
  • Files reviewed: 31/35 changed files
  • Comments generated: 2

Comment on lines +1 to +5
module github.com/github/copilot-sdk/go/copilotexperimental

go 1.24

require golang.org/x/tools v0.28.0
Comment on lines +5 to +9
const DOCS_URL = "https://github.com/github/copilot-sdk/tree/main/nodejs/eslint-plugin";
const EXPERIMENTAL_TAG = "experimental";

const createRule = ESLintUtils.RuleCreator((name) => `${DOCS_URL}#${name}`);

@github-actions

This comment has been minimized.

The standalone @github/eslint-plugin-copilot-sdk package ships its own node:test suite (run via its package's npm test). The SDK root 'vitest run' was globbing eslint-plugin/test/no-experimental-api.test.js and failing to collect it (No test suite found), breaking Node.js SDK Tests on all platforms. Exclude eslint-plugin/** from the root vitest config so the plugin's tests run only under their own runner.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

Cross-SDK Consistency Review ✅

This PR does a strong job achieving feature parity across all six SDKs for experimental API enforcement. Here's how the six implementations compare:

SDK Mechanism Enforcement level Opt-in
Java CopilotExperimental annotation + processor Compile-time error AllowCopilotExperimental or -Acopilot.experimental.allowed=true
.NET [Experimental("CSDK0001")] attribute Compiler diagnostic #pragma warning disable / [SuppressMessage]
TypeScript (new) eslint-plugin-copilot-sdk ESLint rule Lint-time error // eslint-disable-next-line comment
Python (new) experimental decorator → ExperimentalWarning Runtime warning (escalatable) with allow_experimental(): or set_experimental_policy("ignore")
Go (new) go/analysis analyzer (go/copilotexperimental) Analysis-time error //nolint:copilotexperimental
Rust (new) experimental Cargo feature → conditional visibility Compile-time error features = ["experimental"]

All four new implementations are idiomatic for their language ecosystem and consistent with the existing Java and .NET patterns.

Minor note: Python's since parameter

Python's experimental decorator supports an optional since="1.2" parameter that includes a version in the warning message. None of the other SDKs surface this metadata at the call site (Java's CopilotExperimental annotation has no since() attribute, TypeScript JSDoc doesn't use a parameterized experimental tag, etc.). The generated code doesn't use it — only bare @experimental is emitted — so this is purely an enhancement available for hand-written usage. Not a consistency problem, but if surfacing the version in diagnostics were useful for Java consumers too, CopilotExperimental could grow a since() attribute in a follow-up.

Note on enforcement strength

The enforcement levels differ by design and are all appropriate:

  • Hard compile errors: Java, .NET, Rust (compile step enforces it automatically)
  • Tooling-required: TypeScript (ESLint), Go (go/analysis) — consumers must integrate the tool into their build/CI
  • Runtime warnings: Python — appropriate since Python has no compile step; escalatable to errors via set_experimental_policy("error") or COPILOT_EXPERIMENTAL=error

No changes required.

Generated by SDK Consistency Review Agent for issue #1719 · sonnet46 2.3M ·

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants