Skip to content

fix(cdk): name the stream in the rate-limit-exhausted error message - #1242

Draft
devin-ai-integration[bot] wants to merge 1 commit into
mainfrom
devin/1792528193-rate-limit-error-message-stream-name
Draft

devin-ai-integration[bot] wants to merge 1 commit into
mainfrom
devin/1792528193-rate-limit-error-message-stream-name

Conversation

@devin-ai-integration

Copy link
Copy Markdown
Contributor

Summary

Resolves https://github.com/airbytehq/oncall/issues/13799:

Close on merge: no — treated as customer-filed (the issue references a specific customer connection/job); support confirms before closing.

When HttpClient._send_with_retry exhausts retries on HTTP 429, the user-facing message was just "API rate limit exceeded." with no context. It now names the requester/stream:

- message="API rate limit exceeded.",
+ message=f"API rate limit exceeded for stream '{self._name}' after all retries were exhausted.",

internal_message, failure_type=transient_error and stream_descriptor are unchanged. Follows the existing in-file pattern (f"Stream '{self._name}' page rejected...") and the error-message guidelines (deterministic per stream, no remediation hint for transient errors, <200 chars).

Note on async retrievers: AsyncRetriever sub-requesters are named by the factory as e.g. "job extract_url - <stream>", so for the reported case (source-amazon-seller-partner, download_target_requester) the message reads API rate limit exceeded for stream 'job extract_url - GET_V2_SETTLEMENT_REPORT_DATA_FLAT_FILE' after all retries were exhausted. — this names both the real stream and the job phase that was rate limited, and matches the stream descriptor already emitted. Stripping the prefix to a bare stream name would require plumbing a separate stream name through HttpRequester/HttpClient; left out to keep this change minimal.

Related: open PR #949 also touches rate-limit messaging (default 429 mapping + a RateLimitBackoffException branch suggesting fewer workers). This PR only changes the existing 429-exhausted message string and doesn't conflict semantically; whichever lands second should keep the stream name in the message.

Rollout: manifest-only connectors (incl. source-amazon-seller-partner) pick this up only after a source-declarative-manifest base image bump — not done here. The connector-side missing 429 handler is tracked separately in https://github.com/airbytehq/oncall/issues/13798.

Test Coverage

  • Updated the two existing match="API rate limit exceeded." asserts in test_http.py.
  • New test_rate_limit_exhaustion_includes_stream_context (test_http.py): asserts exact message, transient_error, and stream descriptor.
  • New test_rate_limit_exhaustion_message_includes_stream_name (test_http_client.py): HttpClient(name="job extract_url - my_stream") exhausting 429 retries.
  • Verified the new/updated tests fail on unfixed code (assert 'API rate limit exceeded.' == ...) and pass with the fix; unit_tests/sources/streams/http/: 535 passed. Ruff lint/format and MyPy clean.

Declarative-First Evaluation

Not applicable — this is a CDK core change, not a connector manifest change.

Breaking change

Not breaking: only the human-readable message text changes; no schema, spec, state, or failure-type changes. Version is managed by the CDK release process (no manual bump).

Link to Devin session: https://app.devin.ai/sessions/08fec716181a4970a327e02d3623902c
Open in Devin Desktop: https://app.devin.ai/desktop/session/08fec716181a4970a327e02d3623902c?variant=devin

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@github-actions

Copy link
Copy Markdown

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1792528193-rate-limit-error-message-stream-name#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1792528193-rate-limit-error-message-stream-name

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

  • /autofix - Fixes most formatting and linting issues
  • /poetry-lock - Updates poetry.lock file
  • /test - Runs connector tests with the updated CDK
  • /prerelease - Triggers a prerelease publish with default arguments
  • /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
  • /poe <command> - Runs any poe command in the CDK environment
📚 Show Repo Guidance

Helpful Resources

📝 Edit this welcome message.

@github-actions

Copy link
Copy Markdown

PyTest Results (Fast)

5 840 tests  +2   5 809 ✅ +2   11m 52s ⏱️ + 3m 43s
    1 suites ±0      31 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 8fcfb69. ± Comparison against base commit 8b721cd.

@github-actions

Copy link
Copy Markdown

PyTest Results (Full)

5 843 tests  +2   5 812 ✅ +2   13m 27s ⏱️ - 3m 14s
    1 suites ±0      31 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 8fcfb69. ± Comparison against base commit 8b721cd.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants