Skip to content

feat(source-amazon-seller-partner): Test JsonItemsDecoder for brand analytics - #78360

Draft
Alfredo Garcia (agarctfi) wants to merge 5 commits into
masterfrom
devin/1779460421-amazon-seller-partner-json-items-cdk
Draft

feat(source-amazon-seller-partner): Test JsonItemsDecoder for brand analytics#78360
Alfredo Garcia (agarctfi) wants to merge 5 commits into
masterfrom
devin/1779460421-amazon-seller-partner-json-items-cdk

Conversation

@agarctfi

@agarctfi Alfredo Garcia (agarctfi) commented May 22, 2026 •
edited by devin-ai-integration Bot
Loading

Copy link
Copy Markdown
Contributor

What

Alfredo Garcia asked for a draft PR to test the JsonItemsDecoder changes from airbytehq/airbyte-python-cdk#1026 in the Amazon Seller Partner connector. This PR updates the Brand Analytics JSON report downloads for the OOM investigation tracked in https://github.com/airbytehq/oncall/issues/12143.

How

The connector now builds from the SDM pre-release generated by the completed CDK publish workflow for airbytehq/airbyte-python-cdk#1026:

docker.io/airbyte/source-declarative-manifest:7.21.1.post9.dev27167986557@sha256:a5b8673bd083193321a9ed06aafdfe0e8fafa6a5848a8dea3ba95b27ec11b7dd

The unit-test environment is also pinned to airbyte-cdk==7.21.1.post9.dev27167986557 so connector validation runs against the same JsonItemsDecoder-capable CDK pre-release.

The five Brand Analytics report streams replace the custom source_declarative_manifest.components.GzipJsonDecoder with a declarative GzipDecoder wrapping JsonItemsDecoder, and extraction now reads the streamed array items directly from field_path: [].

The branch was updated against latest master; the Amazon Seller Partner conflict kept the JsonItemsDecoder manifest changes and the CDK/SDM pre-release pin.

Validation on this branch:

  • docker buildx imagetools inspect docker.io/airbyte/source-declarative-manifest:7.21.1.post9.dev27167986557 → digest matches sha256:a5b8673bd083193321a9ed06aafdfe0e8fafa6a5848a8dea3ba95b27ec11b7dd
  • poetry check from source-amazon-seller-partner/unit_tests → passed
  • poetry run pytest test_source.py::test_streams_count -q → 1 passed after merge
  • poetry run pytest integration/test_report_based_streams.py::TestFullRefresh::test_given_report_when_read_then_return_records --maxfail=1 -q → 22 passed after merge
  • poetry run pytest -q from source-amazon-seller-partner/unit_tests → 554 passed after merge
  • git diff --check scoped to the touched Amazon Seller Partner files → passed

Review guide

  1. airbyte-integrations/connectors/source-amazon-seller-partner/metadata.yaml
  2. airbyte-integrations/connectors/source-amazon-seller-partner/manifest.yaml
  3. airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/pyproject.toml
  4. airbyte-integrations/connectors/source-amazon-seller-partner/unit_tests/poetry.lock

User Impact

If the CDK image accepts this manifest shape, Brand Analytics JSON report records should be streamed from the nested arrays instead of loading the full decompressed JSON payload into memory before extraction. This targets OOM risk on large Brand Analytics reports without changing the emitted fields.

Can this PR be safely reverted and rolled back?

  • YES 💚
  • NO ❌

Link to Devin session: https://app.devin.ai/sessions/15c02c695f434bbbab5ca5ec5d099405
Requested by: Alfredo Garcia (@agarctfi)

@devin-ai-integration

devin-ai-integration Bot commented May 22, 2026

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@github-actions

github-actions Bot commented May 22, 2026

Copy link
Copy Markdown
Contributor

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

  • 🛠️ Quick Fixes
    • /format-fix - Fixes most formatting issues.
    • /bump-version - Bumps connector versions, scraping changelog description from the PR title.
      • Bump types: patch (default), minor, major, major_rc, rc, promote.
      • The rc type is a smart default: applies minor_rc if stable, or bumps the RC number if already RC.
      • The promote type strips the RC suffix to finalize a release.
      • Example: /bump-version type=rc or /bump-version type=minor
    • /bump-progressive-rollout-version - Alias for /bump-version type=rc. Bumps with an RC suffix and enables progressive rollout.
  • ❇️ AI Testing and Review (internal link: AI-SDLC Docs):
    • /ai-prove-fix - Runs prerelease readiness checks, including testing against customer connections.
    • /ai-canary-prerelease - Rolls out prerelease to 5-10 connections for canary testing.
    • /ai-review - AI-powered PR review for connector safety and quality gates.
  • 📝 AI Documentation:
    • /ai-docs-review - AI-powered documentation review for PRs with connector changes.
    • /ai-create-docs-pr - Creates a documentation PR for connector changes, stacked on the current PR.
  • 🚀 Connector Releases:
    • /publish-connectors-prerelease - Publishes pre-release connector builds (tagged as {version}-preview.{git-sha}) for all modified connectors in the PR.
  • ☕️ JVM connectors:
    • /update-connector-cdk-version connector=<CONNECTOR_NAME> - Updates the specified connector to the latest CDK version.
      Example: /update-connector-cdk-version connector=destination-bigquery
  • 🐍 Python connectors:
    • /poe connector source-example lock - Run the Poe lock task on the source-example connector, committing the results back to the branch.
    • /poe source example lock - Alias for /poe connector source-example lock.
    • /poe source example use-cdk-branch my/branch - Pin the source-example CDK reference to the branch name specified.
    • /poe source example use-cdk-latest - Update the source-example CDK dependency to the latest available version.
  • ⚙️ Admin commands:
    • /force-merge reason="<REASON>" - Force merges the PR using admin privileges, bypassing CI checks. Requires a reason.
      Example: /force-merge reason="CI is flaky, tests pass locally"
📚 Show Repo Guidance

Helpful Resources

📝 Edit this welcome message.

@github-actions

github-actions Bot commented May 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

source-amazon-seller-partner Connector Test Results

568 tests   556 ✅  22m 40s ⏱️
  2 suites    4 💤
  2 files      8 ❌

For more details on these failures, see this check.

Results for commit d638617.

♻️ This comment has been updated with latest results.

devin-ai-integration Bot added a commit to airbytehq/airbyte-python-cdk that referenced this pull request Jun 8, 2026
@ZaneHyattAB @devin-ai-integration
fix: regenerate declarative_component_schema.py to include JsonItemsD…
67f0e2e 
…ecoder in all Union types

Ran `poe assemble` (datamodel-codegen) against the YAML schema to regenerate
the Pydantic model. This adds JsonItemsDecoder to:
- GzipDecoder.decoder
- ZipfileDecoder.decoder
- SimpleRetriever.decoder
- AsyncRetriever.decoder
- AsyncRetriever.download_decoder

The YAML schema already had JsonItemsDecoder in these anyOf unions, but the
previously generated Python model was stale and missing it. This fixes manifest
validation for GzipDecoder -> JsonItemsDecoder pipelines (needed by
airbytehq/airbyte#78360).

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration devin-ai-integration Bot mentioned this pull request Jun 8, 2026
Open
4 tasks
@devin-ai-integration

devin-ai-integration Bot commented Jun 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Updated this PR to use the CDK/SDM pre-release generated by the completed publish workflow for airbytehq/airbyte-python-cdk#1026, then merged latest master and resolved the Amazon Seller Partner conflict in favor of the JsonItemsDecoder manifest changes plus the pre-release pin.

Pinned versions:

Validation passed locally:

  • docker buildx imagetools inspect docker.io/airbyte/source-declarative-manifest:7.21.1.post9.dev27167986557 confirmed digest sha256:a5b8673bd083193321a9ed06aafdfe0e8fafa6a5848a8dea3ba95b27ec11b7dd
  • poetry check from source-amazon-seller-partner/unit_tests
  • poetry run pytest test_source.py::test_streams_count -q → 1 passed after merge
  • poetry run pytest integration/test_report_based_streams.py::TestFullRefresh::test_given_report_when_read_then_return_records --maxfail=1 -q → 22 passed after merge
  • poetry run pytest -q from source-amazon-seller-partner/unit_tests → 554 passed after merge
  • git diff --check scoped to the touched Amazon Seller Partner files

Latest push is d638617eec8.

Devin session

@ZaneHyattAB

ZaneHyattAB commented Jun 8, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor

/publish-connectors-prerelease

Pre-release Connector Publish Started

Publishing pre-release build for connector source-amazon-seller-partner.
PR: #78360

Pre-release versions will be tagged as {version}-preview.d638617
and are available for version pinning via the scoped_configuration API.

View workflow run
Pre-release Publish: SUCCESS ✅

Docker image (pre-release):
airbyte/source-amazon-seller-partner:5.7.7-preview.d638617

Docker Hub: https://hub.docker.com/layers/airbyte/source-amazon-seller-partner/5.7.7-preview.d638617

Registry JSON:

@github-actions

github-actions Bot commented Jun 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Pre-release Connector Publish Started

Publishing pre-release build for connector source-amazon-seller-partner.
PR: #78360

Pre-release versions will be tagged as {version}-preview.d638617
and are available for version pinning via the scoped_configuration API.

View workflow run
⚠️ Pre-release Publish CANCELLED for source-amazon-seller-partner.

@devin-ai-integration

devin-ai-integration Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Publish status clarification:

I also canceled a duplicate publish workflow I accidentally triggered after Zane's slash command had already started the real publish run. The canceled duplicate is https://github.com/airbytehq/airbyte/actions/runs/27171653290?pr=78360 and does not affect the successful pre-release above.

Devin session

@github-actions

github-actions Bot commented Jun 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Pre-release Connector Publish Started

Publishing pre-release build for connector source-amazon-seller-partner.
PR: #78360

Pre-release versions will be tagged as {version}-preview.d638617
and are available for version pinning via the scoped_configuration API.

View workflow run
⚠️ Pre-release Publish CANCELLED for source-amazon-seller-partner.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@ZaneHyattAB ZaneHyattAB

Labels

connectors/source/amazon-seller-partner

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@agarctfi @ZaneHyattAB @octavia-squidington-iii