[devin-ai-integration]

Overview

👉 TL;DR: When a connector waits before retrying a failed request, the CDK logged "Rate limit exceeded" no matter why the request failed — including plain server errors — which led a customer and support to think a GitHub quota was exhausted. The log line now says why it is waiting and only mentions a rate limit for HTTP 429, with the wait rounded to one decimal. user_defined_backoff_handler.sleep_on_ratelimit in rate_limiting.py now logs e.g. Backing off for 2911.8 seconds after HTTP 500 response (attempt 1), then retrying. instead of UserDefinedBackoffException: Rate limit exceeded (HTTP 500). Retrying in 2911.8324723243713 seconds.

Resolves https://github.com/airbytehq/oncall/issues/13838:

• https://github.com/airbytehq/oncall/issues/13838

Close on merge: yes — internally filed, no customer to confirm with.

Related to https://github.com/airbytehq/oncall/issues/13837:

• https://github.com/airbytehq/oncall/issues/13837

Changes

• airbyte_cdk/sources/streams/http/rate_limiting.py: sleep_on_ratelimit picks the log line by cause:
  • HTTP 429 → Rate limit exceeded (HTTP 429). Backing off for {wait} seconds (attempt {n}), then retrying.
  • other HTTP status → Backing off for {wait} seconds after HTTP {status} response (attempt {n}), then retrying.
  • no response (request exception) → Backing off for {wait} seconds after a request error (attempt {n}), then retrying.
  • {wait} is formatted :.1f; the actual time.sleep(retry_after + 1) value is unchanged.
• New unit_tests/sources/streams/http/test_rate_limiting.py asserting the exact log line for 500, 429, and a ReadTimeout, and that the sleep value is not rounded.

Not changed / related PRs

• No retry, wait-time, or FailureType behavior changes; log text only.
• http_client.py exhausted-retries messages are intentionally untouched to avoid conflicts with open PRs that edit those lines: fix(cdk): improve rate limiting error messages to suggest reducing num_workers #949 (drops "Please see logs" and rewords rate-limit messages), fix(cdk): name the stream in the rate-limit-exhausted error message #1242 (names the stream in the 429-exhausted message), fix(http): classify body-coded rate-limit exhaustion as transient_error #1087 (body-coded rate-limit classification). Prior related change: fix: suppress misleading backoff library log in user_defined_backoff_handler #981 (suppressed the misleading backoff library log in this same handler).
• The source-github behavior that makes a single 500 wait ~50 minutes (X-RateLimit-Reset applied to all retryable responses) is out of scope and tracked in airbytehq/oncall#13837.

Test Coverage

New parametrized test fails on main (logs UserDefinedBackoffException: Rate limit exceeded (HTTP 500). Retrying in 2911.8324723243713 seconds.) and passes with the fix. unit_tests/sources/streams/http/ passes locally (536 passed); ruff and mypy clean.

Pre-fix test output

Show/Hide Content

INFO  UserDefinedBackoffException: Rate limit exceeded (HTTP 500). Retrying in 2911.8324723243713 seconds. (rate_limiting.py:112)
FAILED
E  At index 0 diff: 'UserDefinedBackoffException: Rate limit exceeded (HTTP 500). Retrying in 2911.8324723243713 seconds.' != 'Backing off for 2911.8 seconds after HTTP 500 response (attempt 1), then retrying.'
INFO  UserDefinedBackoffException: Rate limit exceeded (HTTP 429). Retrying in 60 seconds. (rate_limiting.py:112)
FAILED
INFO  UserDefinedBackoffException: Rate limit exceeded. Retrying in 5 seconds. (rate_limiting.py:116)
FAILED

Link to Devin session: https://app.devin.ai/sessions/e2baef250f524c608deead0a3eca669e
Open in Devin Desktop: https://app.devin.ai/desktop/session/e2baef250f524c608deead0a3eca669e?variant=devin

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1792771896-neutral-user-backoff-log#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1792771896-neutral-user-backoff-log

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

• /autofix - Fixes most formatting and linting issues
• /poetry-lock - Updates poetry.lock file
• /test - Runs connector tests with the updated CDK
• /prerelease - Triggers a prerelease publish with default arguments
• /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
• /poe <command> - Runs any poe command in the CDK environment

📚 Show Repo Guidance

Helpful Resources

• CDK API Reference

📝 Edit this welcome message.

[github-actions]

PyTest Results (Fast)

5 905 tests  +3   5 874 ✅ +3   6m 10s ⏱️ -11s
    1 suites ±0      31 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit c094142. ± Comparison against base commit 3e85723.

[github-actions]

PyTest Results (Full)

5 908 tests  +3   5 877 ✅ +3   11m 8s ⏱️ +5s
    1 suites ±0      31 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit c094142. ± Comparison against base commit 3e85723.