Repository navigation
fix(source-amazon-seller-partner): stream-decode large Brand Analytics reports to prevent OOM - #79669
Conversation
|
Note 📝 PR Converted to Draft More info...Thank you for creating this PR. As a policy to protect our engineers' time, Airbyte requires all PRs to be created first in draft status. Your PR has been automatically converted to draft status in respect for this policy. As soon as your PR is ready for formal review, you can proceed to convert the PR to "ready for review" status by clicking the "Ready for review" button at the bottom of the PR page. To skip draft status in future PRs, please include |
…s reports to prevent OOM
The Brand Analytics report streams buffered the entire gzip-decompressed JSON document
(multi-GB) into memory, causing OOM kills (exit 137) on large sellers. Switch their
download_decoder to the streaming GzipDecoder{JsonItemsDecoder}, which yields records
one at a time via ijson.
Amazon serves these report documents gzip-compressed but labels them
'Content-Encoding: identity' by default, so the header-based GzipDecoder would skip
decompression. Request enableContentEncodingUrlHeader=true on getReportDocument so Amazon
sets an honest Content-Encoding (gzip when compressed, identity when not).
Validated end-to-end on real reports: 3.2GB search-terms report peak memory ~10GB -> ~150MB
with identical records; repeat-purchase stream reads 475 records cleanly.
Uses prerelease CDK 7.21.2.post1.dev27346905256 (JsonItemsDecoder, airbytehq/airbyte-python-cdk#1049);
baseImage to be finalized to the official SDM release once #1049 merges.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
03b98d7 to
7a90d37
Compare
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. 💡 Show Tips and TricksPR Slash CommandsAirbyte Maintainers (that's you!) can execute the following slash commands on your PR:
📚 Show Repo GuidanceHelpful Resources
|
|
…Decoder; mock honest Content-Encoding Bump the unit-test airbyte-cdk to the prerelease that provides JsonItemsDecoder (7.21.2.post1.dev27346905256) so the manifest validates, and set Content-Encoding: gzip on the compressed report-download mock to mirror enableContentEncodingUrlHeader so the header-based GzipDecoder decompresses it. 317 report-based stream tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
/publish-connectors-prerelease
|
…arge JSON report streams Migrate all 12 large JSON report streams (Brand Analytics, Sales and Traffic, Vendor) to the streaming JsonItemsDecoder on CDK 7.23.1. 7.23.1 includes the ijson use_float fix, so float fields (e.g. Search Terms clickShare/conversionShare) serialize correctly instead of failing as Decimal. Validated against a real 3.45 GB Search Terms report: the released 5.7.7 OOM-kills at a 2, 4, and even 10 GiB cap, while this build streams all 8,533,151 records under a 2 GiB cap (peak ~150 MiB), exit 0, no Decimal errors. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
/publish-connectors-prerelease
|
|
Deploy preview for airbyte-docs ready!
Deployed with vercel-action |
…in vendor forecasting compressed-report mock The vendor forecasting (FRESH/RETAIL) compressed-report test builds its mocked response manually and was dropping the response headers, so the served gzip body had no Content-Encoding: gzip. After migrating these streams to the header-based GzipDecoder + JsonItemsDecoder, the decoder no longer decompressed the body and ijson received raw gzip bytes (IncompleteJSONError). Pass the headers through so the mock mirrors Amazon's enableContentEncodingUrlHeader=true behavior. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
All 12 JSON report streams now use the native streaming JsonItemsDecoder, so the custom GzipJsonDecoder is no longer referenced in the manifest. Remove the dead class. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…n-seller-partner/streaming-brand-analytics-decoder # Conflicts: # docs/integrations/sources/amazon-seller-partner.md
What
Resolves the OOM (exit 137) on large JSON report streams in
source-amazon-seller-partner(oncall #12143).These report streams used a custom
GzipJsonDecoderthat buffered the entire gzip-decompressed JSON document (multi-GB) plus its parsed object tree in memory — ~10 GB for a 3.45 GB report, OOM-killing even 8–10 GB containers.How
CustomDecoder(GzipJsonDecoder)to the streamingGzipDecoder{JsonItemsDecoder}(CDK airbytehq/airbyte-python-cdk#1049), which yields array elements one at a time viaijson— bounded memory regardless of report size. Covered streams: the 5 Brand Analytics reports (Search Terms, Market Basket, Repeat Purchase, Alternate Purchase, Item Comparison), the 3 Sales and Traffic reports (by ASIN, by date, by month), and the 4 Vendor reports (Sales, Inventory, Forecasting FRESH/RETAIL).Content-Encoding: identityby default, so the header-basedGzipDecoderwould skip decompression. RequestenableContentEncodingUrlHeader=trueongetReportDocument(scoped to these streams) so Amazon sets an honestContent-Encoding(gzipwhen compressed,identitywhen not). CSV/XML report downloads are untouched.baseImageto CDK 7.23.1, which includes bothJsonItemsDecoderand theijson use_float=Truefix (#1052) so non-integer fields (e.g. Search TermsclickShare/conversionShare) serialize as floats instead of failing asDecimal.GzipJsonDecoder(no longer referenced by any stream). CSV/XML decoders remain.Validation (real account at cloud memory limits)
GET_BRAND_ANALYTICS_SEARCH_TERMS_REPORT, 3.45 GB uncompressed / 8,533,151 records:5.7.7(oldGzipJsonDecoder)JsonItemsDecoder, CDK 7.23.1)Same report, same 2 GiB cap: the old decoder OOM-kills even at 10 GiB, while this build streams all 8.5M records under 2 GiB with no
Decimalserialization errors. Validated with the official preview image built from this branch.Both the compressed (
Content-Encoding: gzip) and uncompressed (identity) download paths are covered bytest_given_compressed_report_when_read_then_return_recordsandtest_given_report_when_read_then_return_recordsrespectively.Note for reviewers
Two of the Brand Analytics streams —
GET_BRAND_ANALYTICS_ALTERNATE_PURCHASE_REPORTandGET_BRAND_ANALYTICS_ITEM_COMPARISON_REPORT— now return FATAL"This report is now deprecated."from Amazon. They're migrated for consistency but are effectively dead upstream; removing them can be handled separately.Note: the OOM and Amazon-side report
FATAL(generation failures) are distinct failure modes. This PR removes the OOM whenever Amazon returns aDONEreport; it does not affect cases where Amazon fails to generate the report at all.🤖 Generated with Claude Code