Repository navigation
feat: enable Datadog memory profiling for connector builds - #74308
devin-ai-integration[bot] wants to merge 25 commits into
Conversation
- Change entrypoint to use ddtrace-run wrapper - Add DD_PROFILING_ENABLED, DD_PROFILING_TIMELINE_ENABLED env vars - Disable sqlite3 tracing to avoid known issues - Set airbyte-ci install mode to source to pick up pipeline changes Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
Note 📝 PR Converted to Draft More info...Thank you for creating this PR. As a policy to protect our engineers' time, Airbyte requires all PRs to be created first in draft status. Your PR has been automatically converted to draft status in respect for this policy. As soon as your PR is ready for formal review, you can proceed to convert the PR to "ready for review" status by clicking the "Ready for review" button at the bottom of the PR page. To skip draft status in future PRs, please include |
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksPR Slash CommandsAirbyte Maintainers (that's you!) can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful Resources
|
Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
|
|
/publish-connectors-prerelease |
|
/publish-connectors-prerelease
|
The publish workflow uses docker-images/Dockerfile.manifest-only-connector (not build_customization.py) to build manifest-only connector images. Updated the Dockerfile to use ddtrace-run wrapper and set DD env vars so Datadog profiling is active at runtime. Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
|
/publish-connectors-prerelease
|
…3.13 profiling fix) Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
|
/publish-connectors-prerelease
|
…profiling Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
…p sample size Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
|
/publish-connectors-prerelease
|
Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
… fragmentation Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
|
/publish-connectors-prerelease
|
…tter memory management Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
… TTL Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
The CI test builds connector images using the Dockerfile.manifest-only-connector with the base image from metadata.yaml. If the base image doesn't have ddtrace installed, the ddtrace-run entrypoint fails with exit code 127 (command not found). Fix: Use a wrapper script that checks for ddtrace-run at build time and falls back to plain python if not available. Also revert build_customization.py to use plain python entrypoint since it's used for CI test builds. Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
Update base image to 7.10.1.post12.dev22961776456 which has ddtrace v3 installed (confirmed via docker inspection). Previous prerelease (post19.dev22961350734) did not have ddtrace installed. Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
|
/publish-connectors-prerelease
|
Updates base image to 7.10.1.post13.dev22973317414 which includes periodic cache purging to prevent unbounded memory growth from requests_cache lazy expiration. Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
|
/publish-connectors-prerelease
|
Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
|
/publish-connectors-prerelease |
|
Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
|
…ng for testing Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
…E support Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
|
Co-Authored-By: gl_anatolii.yatsuk <gl_anatolii.yatsuk@airbyte.io>
|
/publish-connectors-prerelease |
|
|
Closing as stale. This PR has been open and unreviewed for 3-5 months with no linked issue or ticket. Master has moved substantially since the branch was cut, so it would need re-verification rather than a straight rebase. If the underlying need still stands, please reopen or raise a fresh PR against current master — the work here is preserved in the branch and this closure is reversible. |
What
Enables Datadog APM and memory profiling for manifest-only connector builds so we can investigate memory issues on connectors like Twilio. Follows the same pattern established in #48609 and #61344.
Also adds
AIRBYTE_USE_IN_MEMORY_CACHE=trueto force the CDK'srequests_cacheto use in-memory SQLite instead of file-based SQLite. This eliminates OS page cache growth from SQLite file I/O — confirmed via sandbox testing that RSS stays at ~300 MB whilekubernetes.memory.usagepreviously grew to 2+ GB due to kernel page cache.Companion CDK PR: airbytehq/airbyte-python-cdk#932 (adds
ddtracev3 to thesource-declarative-manifestbase image, respectsAIRBYTE_USE_IN_MEMORY_CACHEenv var, adds 1-hour TTL cache expiration, and periodically purges expired cache entries).How
docker-images/Dockerfile.manifest-only-connector: Adds a conditional entrypoint wrapper script that usesddtrace-run python main.pyif ddtrace is installed in the base image, otherwise falls back to plainpython main.py. This allows the same Dockerfile to work with both ddtrace-enabled (prerelease) and standard base images. Sets Datadog profiling env vars (DD_PROFILING_ENABLED,DD_PROFILING_TIMELINE_ENABLED,DD_TRACE_SQLITE3_ENABLED=false,DD_PROFILING_HEAP_SAMPLE_SIZE=262144) andAIRBYTE_USE_IN_MEMORY_CACHE=true.build_customization.py: Adds the same DD env vars andAIRBYTE_USE_IN_MEMORY_CACHE=truefor theairbyte-cipipeline path. Uses plainpythonentrypoint (noddtrace-run) since airbyte-ci builds may use standard base images without ddtrace.install-airbyte-ci/action.yml: Switches CI install mode frombinarytosourceso CI picks up the pipeline changes from this branch.source-twilio/metadata.yaml: PinsbaseImageto CDK prerelease7.10.1.post13.dev22973317414which includes ddtrace v3,AIRBYTE_USE_IN_MEMORY_CACHEsupport, 1-hour cache TTL, and periodic cache purging.Updates since last revision
7.10.1.post13.dev22973317414: This prerelease adds periodic active purging of expired cache entries in the HTTP client.requests_cacheuses lazy expiration — expired entries stay in the SQLite database until re-accessed, not automatically deleted. For connectors making thousands of unique paginated API calls, this caused expired entries to accumulate indefinitely in the in-memory SQLite database, growing RSS to ~2 GB and triggering OOM kills even with the 1-hour TTL. The fix callssession.cache.delete(expired=True)every 100 requests to actively reclaim memory.kubernetes.memory.usage(no page cache gap), but RSS itself grew to ~2 GB due to the lazy expiration issue above. The periodic cache purging should keep the in-memory cache bounded to approximately the last hour of non-expired data.Previous updates
ddtrace-runentrypoint with a shell wrapper script (entrypoint.sh) that checkscommand -v ddtrace-runat build time. Fixes CI test failures (exit code 127) when building against base images without ddtrace.DD_PROFILING_STACK_ENABLED=false: ddtrace v3 has full Python 3.13 stack profiling support.expire_after=3600inCachedLimiterSession.kubernetes.memory.rssstays at ~186 MiB avg / 300 MB max. The 2 GBkubernetes.memory.usagewas RSS + OS page cache from SQLite file I/O. The ~1.7 GB gap was reclaimable kernel page cache, not a process memory leak.AIRBYTE_USE_IN_MEMORY_CACHE=true: Forces in-memory SQLite, eliminating file I/O and OS page cache growth.Review guide
docker-images/Dockerfile.manifest-only-connector— key changes: conditional entrypoint wrapper script, DD env vars (stack profiling ENABLED with v3), heap sample size, andAIRBYTE_USE_IN_MEMORY_CACHE=trueairbyte-ci/connectors/pipelines/pipelines/airbyte_ci/connectors/build_image/steps/build_customization.py— DD env vars andAIRBYTE_USE_IN_MEMORY_CACHE=true, plainpythonentrypoint.github/actions/install-airbyte-ci/action.yml— CI install mode changeairbyte-integrations/connectors/source-twilio/metadata.yaml— base image pinned to CDK prerelease7.10.1.post13.dev22973317414docker-images/Dockerfile.manifest-only-connectorandbuild_customization.pyaffect all manifest-only connectors, not just Twilio.DD_PROFILING_ENABLED=trueandAIRBYTE_USE_IN_MEMORY_CACHE=truewill apply to every manifest-only connector.DD_PROFILING_ENABLED=trueis hardcoded in the Dockerfile: Profiling will be active for every manifest-only connector built from this Dockerfile, not gated by LaunchDarkly. Verify this is the intended behavior vs. relying on the platform'sconnectors.apm-enabledfeature flag.7.10.1.post13.dev22973317414). The prerelease pin must be reverted to a stable release before merging to master.install-mode=source: This changes CI behavior globally — all CI runs will installairbyte-cifrom source instead of using the pre-built binary, which is slower. This should be reverted before merging to master.DD_PROFILING_HEAP_SAMPLE_SIZE=262144(256 KB): More aggressive than the default (~1 MB). Provides better visibility but increases profiling overhead slightly.User Impact
When source-twilio is built from this branch and deployed with the updated prerelease base image:
Expected result:
kubernetes.memory.usageand RSS should both remain near ~300-500 MB instead of growing to 2+ GB, confirming that:kubernetes.memory.usagegrowth to 2+ GB was OS page cache from file-based SQLite (now eliminated with in-memory cache)requests_cache(now mitigated with periodic purging)Can this PR be safely reverted and rolled back?