[devin-ai-integration]

Overview

👉 TL;DR: Lets the sync-token incremental cursor replay a stored token that is a complete URL (Microsoft Graph delta links), and adds build-time checks so first-request-only token shapes (Box events) fail with a clear message instead of a request collision on page 2. Specifically, SyncTokenCursor.token_option now accepts RequestPath, SimpleRetriever gains a first_request_path hook with an origin check, the factory validates token/paginator collisions and inject_on_first_request, and SyncTokenCursor caps stored token state at 4 MB per stream.

Resolves https://github.com/airbytehq/airbyte-internal-issues/issues/18023:

• https://github.com/airbytehq/airbyte-internal-issues/issues/18023

Close on merge: yes — internally filed, no customer to confirm with.

This PR is part of a stacked set:

• feat(declarative): add SyncTokenCursor for server-issued sync tokens (1/2: capture and commit) #1234
• feat(declarative): add RESET_SYNC_TOKEN recovery for SyncTokenCursor (2/2) #1235
• (This PR)

Changes

• declarative_component_schema.yaml / model: SyncTokenCursor.token_option is anyOf: [RequestOption, RequestPath] (Union[RequestOption, RequestPath]).
• SyncTokenRequestOptionsProvider: returns {} from all request-option methods for RequestPath; new get_request_path(stream_slice) returns the stored token, else rendered initial_token, else None.
• SimpleRetriever.first_request_path: on the first request of a pass (no paginator path, no page token) the token URL replaces the requester path; paginator RequestPath links still win on later pages. Absolute token URLs whose scheme or host differ from the requester base raise a config_error before any request is sent.
• ModelToComponentFactory:
  • binds first_request_path to the inner SyncTokenRequestOptionsProvider (unwrapping PerPartitionRequestOptionsProvider) only for SyncTokenCursor + RequestPath; every other stream gets None.
  • rejects a RequestOption token that sets the same field as the paginator's page_token_option/page_size_option unless ignore_stream_slicer_parameters_on_paginated_requests: true.
  • rejects PageIncrement/OffsetIncrement with inject_on_first_request: true when the token goes in the path or on the first request only.
• SyncTokenCursor: max_state_bytes (default 4 MB) tracked incrementally from partition key + token sizes; stored partitions keep updating, new partitions past the limit are read in full, one WARN per stream.

Review Spotlight

Reviewers with limited time, please review first:

• simple_retriever.py _paginator_path / _check_token_host: first-request precedence and the origin check
• model_to_component_factory.py validations: collision and first-request checks

Reproduction

On the base branch (4ced6454):

• SyncTokenRequestOptionsProvider(token_option=RequestPath(...)).get_request_params(...) → AttributeError: 'RequestPath' object has no attribute 'inject_into'.
• Box shape (token and paginator both stream_position as request parameters) → page 2 fails with ValueError: Request body collision, duplicate keys detected at key path: stream_position.
• The new tests: 31 failed / 7 passed before the fix.

Risks

• Reset (RESET_SYNC_TOKEN) with a RequestPath token re-reads from initial_token if set, otherwise the requester path — same as the RequestOption patterns. A manifest whose initial_token means "sync from now" would skip history on reset; adopters should leave initial_token unset for Graph.
• The origin check covers the stored token URL only, not paginator next links.
• The two new factory checks only apply to SyncTokenCursor streams, which are unreleased (this stack).

Declarative-First Evaluation

This is a CDK change that adds built-in declarative capability; no connector custom component is written. The adopters (Graph, Box) are manifest-only and could not express this with existing components: DefaultPaginator with RequestPath only applies after a page token exists (not the first request), RequestOption cannot carry a full URL, and no existing check catches the token/paginator key collision at build time.

Test Coverage

New tests in test_sync_token_cursor_integration.py (exact URLs/params via HttpMocker), test_sync_token_cursor.py, and test_sync_token_request_options_provider.py: T-URL, T-URL-PARAMS, T-URL-INIT, T-URL-RESET, T-URL-PAR, T-ORIGIN, T-POS, T-COLLIDE, T-FIRSTREQ, T-BYTES, T-INERT, plus provider unit tests.

• Focused declarative suites: 1173 passed. Full unit_tests/: 5820 passed, 22 skipped, 2 failed — test_parse_start_date[date_only_ab_datetime_parse_fallback] and [with_timezone_offset_converted_to_utc], which fail the same way on the base commit 4ced6454.
• ruff check, ruff format --check, mypy on changed files: clean.

Versioning

No breaking change: existing RequestOption manifests and the stored state format are unchanged, and RequestPath is an added union member. The CDK version comes from the release process via the conventional feat title; no version file is edited.

Follow-ups

• Docs: docs/platform/connector-development/config-based/understanding-the-yaml-file/incremental-syncs.md in airbytehq/airbyte has no SyncTokenCursor section yet (PRs 1234/1235 did not add one), so the RequestPath / first-request guidance from the issue should land with the stack's docs PR.

Link to Devin session: https://app.devin.ai/sessions/ef74cd18b819430c89b904a53a3ed78e
Open in Devin Desktop: https://app.devin.ai/desktop/session/ef74cd18b819430c89b904a53a3ed78e?variant=devin

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1792471568-sync-token-request-path#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1792471568-sync-token-request-path

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Full)

5 844 tests  +29   5 813 ✅ +29   10m 50s ⏱️ - 3m 0s
    1 suites ± 0      31 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit e09fd86. ± Comparison against base commit 4ced645.

[github-actions]

PyTest Results (Fast)

5 841 tests  +29   5 810 ✅ +29   13m 2s ⏱️ +3s
    1 suites ± 0      31 💤 ± 0 
    1 files   ± 0       0 ❌ ± 0 

Results for commit e09fd86. ± Comparison against base commit 4ced645.

[Patrick Nilan (pnilan)]

/prerelease

Prerelease Job Info

This job triggers the publish workflow with default arguments to create a prerelease.

Prerelease job started... Check job output.

✅ Prerelease workflow triggered successfully.

View the publish workflow run: https://github.com/airbytehq/airbyte-python-cdk/actions/runs/39463931382