[devin-ai-integration]

Overview

👉 TL;DR: When a connection check tests only regular streams, it no longer looks up or sets up the connector's dynamically generated streams. As a result it makes no extra discovery requests, and problems with those generated streams appear at discover and sync time instead of failing the check. Specifically, when every name is static, ConcurrentDeclarativeSource._streams_by_name no longer evaluates self.dynamic_streams and builds only the selected static configs, with all_configs=static_configs. Subclasses that override streams() and checks with dynamic_streams_check_configs still go through streams().

This PR is part of a stacked set:

• fix(low-code): build only the checked static streams during check #1247
• (This PR:) fix(low-code): skip dynamic streams when check reads only static streams #1248

Summary

PR 2 of the plan in the issue (D2). When every name CheckStream reads is a static stream, ConcurrentDeclarativeSource._streams_by_name no longer evaluates self.dynamic_streams: components resolvers are not built, resolver HTTP requests are not sent, and dynamic streams are not built.

 selected_configs = [c for c in static_configs if c.get("name") in selected_names]
-    + self.dynamic_streams
-return self._create_streams(selected_configs, static_configs + dynamic_configs)
+return self._create_streams(selected_configs, static_configs)

Resolves https://github.com/airbytehq/airbyte-internal-issues/issues/17934:

• https://github.com/airbytehq/airbyte-internal-issues/issues/17934

Close on merge: no — internally filed, but the issue also asks for a benchmark row for each release; close once that row is added. (Second of two stacked PRs; #1247 is Part of.)

Rebased onto #1247's review fix: the _uses_default_streams() guard stays first, so a subclass that overrides streams() (e.g. source-google-ads) still goes through its override during check and still resolves dynamic streams there.

Behaviour change (D2, decided)

Duplicate dynamic stream names, a failing resolver request, or a dynamic stream that cannot be built no longer fail check when the checker reads only static streams. discover and read still raise them. Per the issue, the affected manifests are source-bing-ads, source-google-search-console, source-hubspot, source-linkedin-ads and source-pinterest.

Unchanged: a checked name that is a dynamic stream, __airbyte_check_stream_names naming a non-static stream, and checkers with dynamic_streams_check_configs still go through streams().

use_cache risk (accepted in the issue): parent names now come from static configs only, so a static stream read as a parent only by a dynamic stream no longer gets use_cache during check. The request sent is the same.

Releaser: paste into the release draft

check skips dynamic streams when it reads only static streams (#1248). When every stream CheckStream reads is a static stream, check no longer resolves or builds dynamic streams, so it sends no resolver requests. A failing resolver request, duplicate dynamic stream names, or a dynamic stream that can't be built no longer fail check; discover and read still raise them. Connectors that override streams() and checks that name a dynamic stream or use dynamic_streams_check_configs are unchanged.

If #1247 and #1248 ship in the same release, combine the two notes and drop #1247's "Dynamic streams are still resolved and built during check until #1248." sentence.

Test Coverage

New tests in unit_tests/sources/declarative/checks/test_check_stream.py:

• a check naming only static streams sends no HttpComponentsResolver request (HttpMocker);
• duplicate dynamic stream names: check SUCCEEDED, discover raises Dynamic streams list contains a duplicate name;
• resolver HTTP 500: check SUCCEEDED, discover raises;
• a dynamic name and dynamic_streams_check_configs still evaluate the resolver; static-only checks evaluate it zero times;
• a subclass overriding streams() still runs its override once and still sends the resolver request (fails without the override guard);
• fix(low-code): build only the checked static streams during check #1247's "unbuildable dynamic stream fails check" test is flipped: static-only check succeeds, discover raises.

The first three fail on the PR 1 code. unit_tests/sources/declarative, unit_tests/connector_builder and entrypoint tests pass locally (unit_tests/sources/declarative unit_tests/connector_builder: 2511 passed, 1 skipped); ruff and mypy clean.

Declarative-First Evaluation

N/A — CDK framework change, not a connector manifest change.

Link to Devin session: https://app.devin.ai/sessions/328d83d551ec44219895b69f3d2f9bd3
Open in Devin Desktop: https://app.devin.ai/desktop/session/328d83d551ec44219895b69f3d2f9bd3?variant=devin

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1792688475-check-skip-dynamic-streams#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1792688475-check-skip-dynamic-streams

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

5 902 tests  +4   5 871 ✅ +4   6m 14s ⏱️ + 1m 2s
    1 suites ±0      31 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 2a09438. ± Comparison against base commit c6002a1.

♻️ This comment has been updated with latest results.

[github-actions]

PyTest Results (Full)

5 905 tests  +4   5 874 ✅ +4   10m 58s ⏱️ +18s
    1 suites ±0      31 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 2a09438. ± Comparison against base commit c6002a1.

♻️ This comment has been updated with latest results.

[Patrick Nilan (pnilan)]

🤖 This comment was generated by an AI Agent.

Carry-over from the #1247 review: do this after #1247 is updated

#1247 has a review with one blocker and several follow-ups. This PR is stacked on #1247's branch, and both PRs change streams_by_name. Please follow these steps in order:

1. Wait until the fix(low-code): build only the checked static streams during check #1247 review has been addressed.
2. Rebase this branch onto the updated fix(low-code): build only the checked static streams during check #1247 branch. Expect a conflict in streams_by_name.
3. Apply the carry-over below, then re-run CI.

Carry-over

• Keep the streams() override guard first.
  • fix(low-code): build only the checked static streams during check #1247 is adding _uses_default_streams() to streams_by_name and first_streams. Subclasses that override streams() then keep going through it; source-google-ads depends on that.
  • When you resolve the conflict, keep that check as the first line of streams_by_name, before anything that skips dynamic streams.
  • Add a test here. Use a subclass that overrides streams(), a manifest with dynamic streams, and a check that names only static streams. check must call the override, so dynamic streams are still resolved through streams() for that subclass.
• Keep the setup order from fix(low-code): build only the checked static streams during check #1247. fix(low-code): build only the checked static streams during check #1247's finding 9 moves _prepare_for_stream_build() after the static-name check. If that lands, keep the same order here.
• Follow any rename. If fix(low-code): build only the checked static streams during check #1247 renames streams_by_name and first_streams to private names, use the new names here. Keep this PR's docstring, which is correct for D2, and add the None return for an overridden streams().
• Flip fix(low-code): build only the checked static streams during check #1247's PR-1 test on purpose. fix(low-code): build only the checked static streams during check #1247 adds a test that a dynamic stream that can't be built still fails check. In this PR the expected result becomes "check SUCCEEDED, discover raises". Update that test instead of deleting it.
• Re-run the stream-group tests from fix(low-code): build only the checked static streams during check #1247.
  • This PR passes only static configs as all_configs. With fix(low-code): build only the checked static streams during check #1247's stream-group fix, those configs also become the set of top-level names the ancestor walk uses.
  • Confirm that the grandparent-conflict test still fails check here. A grandparent conflict is a checked stream sharing a group with its parent's parent.
• Release note.
  • This PR has the same problem as fix(low-code): build only the checked static streams during check #1247: the release drafter reads only PR titles, so the note in the body never reaches the release on its own.
  • Add a "Releaser: paste into the release draft" section that holds the D2 note.
  • If fix(low-code): build only the checked static streams during check #1247 and fix(low-code): skip dynamic streams when check reads only static streams #1248 ship in the same release, combine the two notes into one. Then drop fix(low-code): build only the checked static streams during check #1247's sentence "Dynamic streams are still resolved and built during check until fix(low-code): skip dynamic streams when check reads only static streams #1248."
• Closing reference. fix(low-code): build only the checked static streams during check #1247 is switching to "Part of #17934", so the issue stays open until this PR lands. Add Resolves #17934 here only if the issue should close when this PR merges. The issue also asks for a benchmark row for each release.

[devin-ai-integration]

Re: the carry-over comment: I've applied the list after rebasing onto #1247's review fix (1101804). The rebased D2 commit is 84880e8a, with the follow-up in 14a37570503b47dbdf553a29ba7155a19b339de6.

• _streams_by_name now runs in this order: override guard, static configs (falls back on error), name check, _prepare_for_stream_build(), then a build of static streams only. The D2 docstring is kept and also documents None when streams() is overridden.
• New test: a source overrides streams() and has a dynamic stream, and the check names only a static stream. The override runs once and the resolver request is sent. The test fails without the guard.
• I flipped fix(low-code): build only the checked static streams during check #1247's unbuildable-dynamic-stream test on purpose: a static-only check succeeds and discover raises.
• The A→B→C stream-group tests and the setup-order tests pass. The API-budget test for _streams_by_name now expects no dynamic-config evaluation.
• PR body: I added the D2 release note and a note on combining it with fix(low-code): build only the checked static streams during check #1247's note. The link is now Resolves #17934 with "Close on merge: no" until the benchmark row is added.

Written by Devin

[Patrick Nilan (pnilan)]

/prerelease

Prerelease Job Info

This job triggers the publish workflow with default arguments to create a prerelease.

Prerelease job started... Check job output.

✅ Prerelease workflow triggered successfully.

View the publish workflow run: https://github.com/airbytehq/airbyte-python-cdk/actions/runs/39879313492