fix(source-amazon-seller-partner): use streaming JSON decoder for large Brand Analytics reports by devin-ai-integration[bot] · Pull Request #77709 · airbytehq/airbyte

devin-ai-integration · 2026-05-02T12:32:07Z

Resolves https://github.com/airbytehq/oncall/issues/12143:

https://github.com/airbytehq/oncall/issues/12143

What

The five Brand Analytics streams download a single gzipped JSON document whose dataByDepartmentAndSearchTerm / dataByAsin array can be multiple gigabytes when decompressed. The legacy GzipJsonDecoder reads the full HTTP body into memory, decompresses it, decodes it as a string and finally calls json.loads() on it — peak memory usage is roughly 4–5x the decompressed size because the gzipped body, decompressed bytes, decoded string and parsed Python object tree all coexist briefly. On the customer-reported sync the report was approximately 3.2 GB decompressed and the connector container was OOM-killed (exit code 137) on an 8 GB cap.

This change introduces a streaming decoder that yields one record at a time without ever materializing the full document and switches the five Brand Analytics streams over to it. All other streams are unaffected.

How

New GzipJsonStreamingItemsDecoder (in source-amazon-seller-partner/components.py):

Sets is_stream_response() = True so the CDK reads the body off the socket instead of buffering it into response.content.
Decompresses gzip incrementally via gzip.GzipFile(fileobj=...) instead of gzip.decompress(response.content).
Parses JSON incrementally via ijson.items(stream, "<items_field_path>.item"), which yields each element of the configured array path one at a time.
Detects the gzip magic number (1f 8b) at the head of the body and falls back to plain-JSON parsing when the body is not gzipped (used by the test setup; the real S3-pre-signed URL response is always gzipped).

Manifest changes (one block per stream, applied to all five Brand Analytics streams):

download_decoder:
  type: CustomDecoder
  class_name: source_declarative_manifest.components.GzipJsonStreamingItemsDecoder
  items_field_path: dataByDepartmentAndSearchTerm   # or dataByAsin
download_extractor:
  type: DpathExtractor
  field_path: []

The extractor's field_path is now empty because the decoder already yields individual records.

Declarative-First Evaluation

Per the /ai-fix Declarative-First Evaluation Gate I evaluated whether this could be solved with built-in CDK / declarative features:

RecordFilter, AddFields, RemoveFields, DatetimeBasedCursor, DefaultPaginator, SubstreamPartitionRouter, HttpRequester error handlers, transformations, $ref overrides — none of these affect how the response body is read or parsed.
JsonDecoder, GzipJsonDecoder, IterableDecoder, JsonlDecoder, XmlDecoder — all of these either buffer the full body into memory or assume a line-delimited format that the Brand Analytics report does not have (it is a single JSON document with a long array nested inside a wrapping object).

A streaming gzip + streaming JSON parser is the only thing that can keep peak memory bounded for a multi-GB single-document report, and no declarative component currently provides that. A custom Python component is therefore required.

Test Coverage

New unit_tests/test_streaming_decoder.py covers:

Yielding each element of the configured array path for both dataByDepartmentAndSearchTerm and dataByAsin (the two paths used by the five streams).
Empty-array payloads (zero records).
Uncompressed-JSON fallback (gzip magic absent).
is_stream_response() returning True.
Lazy yielding: after next(generator) returns the first record, less than the full response body has been read from the underlying stream — confirms records are produced as they are parsed, not after the whole document is buffered.
Memory comparison: peak tracemalloc allocations for the streaming decoder are at least 5x lower than the legacy GzipJsonDecoder on the same 20 000-record gzipped payload, which is the property that fixes the OOM. (Without the streaming decoder this test fails because the legacy decoder materializes the full document.)

All 508 existing connector unit tests still pass locally.

Review guide

airbyte-integrations/connectors/source-amazon-seller-partner/components.py — new GzipJsonStreamingItemsDecoder and _PrefixedStream helper.
airbyte-integrations/connectors/source-amazon-seller-partner/manifest.yaml — five Brand Analytics streams switched to the new decoder.
airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/test_streaming_decoder.py — new tests.
airbyte-integrations/connectors/source-amazon-seller-partner/metadata.yaml — version bump 5.7.4 → 5.7.5.
docs/integrations/sources/amazon-seller-partner.md — changelog entry.

User Impact

The five Brand Analytics streams (GET_BRAND_ANALYTICS_SEARCH_TERMS_REPORT, GET_BRAND_ANALYTICS_MARKET_BASKET_REPORT, GET_BRAND_ANALYTICS_REPEAT_PURCHASE_REPORT, GET_BRAND_ANALYTICS_ALTERNATE_PURCHASE_REPORT, GET_BRAND_ANALYTICS_ITEM_COMPARISON_REPORT) no longer OOM on multi-GB reports; peak memory is bounded by the size of a single record plus ijson's small parse buffers rather than the full document.
All other streams keep using the existing GzipJsonDecoder and are unchanged.
No schema, primary key, cursor, spec, or state changes — this is a non-breaking PATCH bump (5.7.4 → 5.7.5).

Breaking change evaluation

Walked the Breaking Change Evaluation Gate checklist:

Schema (datatype / removed / renamed field, PK, cursor) — unchanged.
Spec / config — unchanged.
Stream removal — none.
State format — unchanged.

This is a non-breaking PATCH bump.

Dependencies

This PR depends on the ijson runtime dependency being available in the source-declarative-manifest base image. That is being added in airbytehq/airbyte-python-cdk#1011. Once that PR is merged and a new SDM image is published, this PR's metadata.yaml baseImage should be bumped to that version before merging. The PR is kept in draft until then.

Can this PR be safely reverted and rolled back?

YES 💚

Reverting drops back to the legacy GzipJsonDecoder for the five streams; users would again hit the OOM on very large reports but otherwise behavior is identical.

Link to Devin session: https://app.devin.ai/sessions/e31a7df6ebe54ce4a68e0eecc7117555

…ge Brand Analytics reports The five Brand Analytics streams (search terms, market basket, repeat purchase, alternate purchase, item comparison) download single-document gzipped JSON reports that can be multiple gigabytes when decompressed. The legacy GzipJsonDecoder loads the entire response into memory and then parses the full document at once, which causes the connector container to be OOM-killed (exit code 137) on multi-GB reports. This change adds a GzipJsonStreamingItemsDecoder that: - Streams the HTTP body off the socket (is_stream_response=True) - Decompresses gzip incrementally - Yields each element of the configured array path one at a time via ijson, without ever materializing the full document or its parsed object tree The five Brand Analytics streams are switched over to the new decoder; all other streams keep using the legacy GzipJsonDecoder. Resolves airbytehq/oncall#12143

devin-ai-integration · 2026-05-02T12:32:09Z

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

Disable automatic comment and CI monitoring

github-actions · 2026-05-02T12:32:30Z

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

🛠️ Quick Fixes
- /format-fix - Fixes most formatting issues.
- /bump-version - Bumps connector versions, scraping changelog description from the PR title.
  - Bump types: patch (default), minor, major, major_rc, rc, promote.
  - The rc type is a smart default: applies minor_rc if stable, or bumps the RC number if already RC.
  - The promote type strips the RC suffix to finalize a release.
  - Example: /bump-version type=rc or /bump-version type=minor
- /bump-progressive-rollout-version - Alias for /bump-version type=rc. Bumps with an RC suffix and enables progressive rollout.
❇️ AI Testing and Review (internal link: AI-SDLC Docs):
- /ai-prove-fix - Runs prerelease readiness checks, including testing against customer connections.
- /ai-canary-prerelease - Rolls out prerelease to 5-10 connections for canary testing.
- /ai-review - AI-powered PR review for connector safety and quality gates.
📝 AI Documentation:
- /ai-docs-review - AI-powered documentation review for PRs with connector changes.
- /ai-create-docs-pr - Creates a documentation PR for connector changes, stacked on the current PR.
🚀 Connector Releases:
- /publish-connectors-prerelease - Publishes pre-release connector builds (tagged as {version}-preview.{git-sha}) for all modified connectors in the PR.
☕️ JVM connectors:
- /update-connector-cdk-version connector=<CONNECTOR_NAME> - Updates the specified connector to the latest CDK version.
  Example: /update-connector-cdk-version connector=destination-bigquery
🐍 Python connectors:
- /poe connector source-example lock - Run the Poe lock task on the source-example connector, committing the results back to the branch.
- /poe source example lock - Alias for /poe connector source-example lock.
- /poe source example use-cdk-branch my/branch - Pin the source-example CDK reference to the branch name specified.
- /poe source example use-cdk-latest - Update the source-example CDK dependency to the latest available version.
⚙️ Admin commands:
- /force-merge reason="<REASON>" - Force merges the PR using admin privileges, bypassing CI checks. Requires a reason.
  Example: /force-merge reason="CI is flaky, tests pass locally"

📚 Show Repo Guidance

Helpful Resources

Breaking Changes Guide - Breaking changes, migration guides, and upgrade deadlines
Developing Connectors Locally
Managing Connector Secrets
On-Demand Regression Tests
#connector-ci-issues
#connector-publish-updates
#connector-build-statuses

📝 Edit this welcome message.

github-actions · 2026-05-02T12:38:12Z

Deploy preview for airbyte-docs ready!

Project:	`airbyte-docs`
Status:	✅ Deploy successful!
Preview URL:	https://airbyte-docs-irdnk2nt3-airbyte-growth.vercel.app
Latest Commit:	`dd1455d`

Deployed with vercel-action

github-actions · 2026-05-02T15:57:48Z

`source-amazon-seller-partner` Connector Test Results

522 tests 508 ✅ 3h 14m 24s ⏱️
2 suites 4 💤
2 files 10 ❌

For more details on these failures, see this check.

Results for commit dd1455d.

devin-ai-integration · 2026-05-02T15:59:11Z

CI status note (PR is intentionally still draft):

Connector unit tests passed (508 pass / 4 skip; new test_streaming_decoder.py included).
Connector standard / integration tests failed with ModuleNotFoundError: No module named 'ijson' when invoking docker run airbyte/source-amazon-seller-partner:dev-latest spec. This is expected — the dev-latest image is built FROM source-declarative-manifest:7.17.2, which does not yet contain ijson.

Unblocking sequence:

Merge build: add ijson as a runtime dependency airbyte-python-cdk#1011 (adds ijson as a runtime dep).
Wait for the next source-declarative-manifest base image to be published with that CDK version.
Bump baseImage in this PR's metadata.yaml to that new SDM tag.
Re-run CI; integration tests should then pass.
Mark this PR ready for review.

Devin session

Octavia Squidington III (octavia-squidington-iii) added the connectors/source/amazon-seller-partner label May 2, 2026

Alfredo Garcia (agarctfi) closed this May 14, 2026

devin-ai-integration Bot mentioned this pull request May 14, 2026

feat: add JsonItemsDecoder for streaming large JSON responses airbytehq/airbyte-python-cdk#1026

Open

4 tasks

Navigation Menu

Repository navigation

fix(source-amazon-seller-partner): use streaming JSON decoder for large Brand Analytics reports - #77709

fix(source-amazon-seller-partner): use streaming JSON decoder for large Brand Analytics reports#77709
devin-ai-integration[bot] wants to merge 1 commit into
masterfrom
devin/1777723282-amazon-seller-partner-streaming-decoder

devin-ai-integration Bot commented May 2, 2026

Uh oh!

devin-ai-integration Bot commented May 2, 2026

Uh oh!

github-actions Bot commented May 2, 2026

PR Slash Commands

Helpful Resources

Uh oh!

github-actions Bot commented May 2, 2026

Uh oh!

github-actions Bot commented May 2, 2026

Uh oh!

devin-ai-integration Bot commented May 2, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

devin-ai-integration Bot commented May 2, 2026

What

How

Declarative-First Evaluation

Test Coverage

Review guide

User Impact

Breaking change evaluation

Dependencies

Can this PR be safely reverted and rolled back?

Uh oh!

devin-ai-integration Bot commented May 2, 2026

🤖 Devin AI Engineer

Uh oh!

github-actions Bot commented May 2, 2026

👋 Greetings, Airbyte Team Member!

PR Slash Commands

Helpful Resources

Uh oh!

github-actions Bot commented May 2, 2026

Uh oh!

github-actions Bot commented May 2, 2026

source-amazon-seller-partner Connector Test Results

Uh oh!

devin-ai-integration Bot commented May 2, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

`source-amazon-seller-partner` Connector Test Results