test(source-activecampaign): move `acceptance-test-config.yml` to the `acceptance_tests` layout by devin-ai-integration[bot] · Pull Request #89554 · airbytehq/airbyte

devin-ai-integration · 2026-10-20T15:27:31Z

Overview

👉 TL;DR: The ActiveCampaign connector's test settings file used an old format that the current test tooling refuses to load, so CI failed before running any test and blocked the weekly dependency update PR. Specifically, this moves `source-activecampaign/acceptance-test-config.yml` from the legacy top-level `tests:` layout to `acceptance_tests: <section>: tests: [...]`, keeps every scenario that passes, and declares a `bypass_reason` for the ones the GSM test config cannot pass.

Resolves https://github.com/airbytehq/oncall/issues/13774:

https://github.com/airbytehq/oncall/issues/13774

Close on merge: yes — internally filed, no customer to confirm with.

Unblocks the up-to-date PR:

deps(source-activecampaign): 🐙 update dependencies [2026-10-20] #78556

Changes

spec (manifest.yaml), the invalid-config check (status: failed) and discovery with secrets/config.json: kept; all pass.
The GSM check (status: succeed) and basic_read: replaced with a bypass_reason stating the observed failure (see Risks).
Dropped the commented-out CAT TODO blocks (expect_records, incremental, full_refresh); the standard tests do not read them.
No version bump or changelog: only acceptance-test-config.yml changes, which CheckVersionIncrement treats as non-production (NON_PRODUCTION_PATHS in airbyte_ops_mcp/connector_qa/checks/version.py).

Risks

The GSM test config (SECRET_SOURCE-ACTIVECAMPAIGN__CREDS) is not usable: its account_username is a full hostname, so the manifest's https://{{ account_username }}.api-us1.com resolves to an invalid host (SSL: CERTIFICATE_VERIFY_FAILED ... Hostname mismatch). With the account name alone, the ActiveCampaign API returns 402 Payment Required, so the test account looks lapsed too. check and read are bypassed until the credential is refreshed.

Follow-ups

Refresh the GSM credential (fix account_username and use a paid/active account), then replace the bypasses with the original entries: config_path: secrets/config.json + status: succeed under connection, and configured_catalog_path: integration_tests/configured_catalog.json + empty_streams: [] under basic_read.

Test Coverage

This change is the test configuration itself; no connector code changes. The standard-test scenarios it restores were verified locally:

Before (master file, airbyte-cdk[dev]==7.37.1 as in CI): airbyte-cdk connector test → collected 0 items / 1 error, ValueError: Acceptance tests config not found in acceptance-test-config.yml.
After, airbyte-cdk 7.37.1: connector test exit 0 (7 passed, 14 skipped); image test exit 0 (4 passed, 6 skipped).
After, CDK branch of feat(standard-tests): fail on an empty test-scenario set unless acceptance-test-config.yml declares a bypass_reason airbyte-python-cdk#1228: connector test exit 0 (7 passed, 32 skipped), so the file still collects once empty scenario sets require a bypass_reason.
With the GSM entries kept as status: succeed / basic_read, test_check, test_docker_image_build_and_check and test_basic_read failed with the SSL hostname mismatch above, which is why they are bypassed.

Declarative-First Evaluation

Not applicable: no manifest or custom component changes; only acceptance-test-config.yml is touched.

Breaking change

Not breaking: no schema, spec, stream, state or behavior change; the released connector is unchanged.

Can this PR be safely reverted and rolled back?

YES 💚
NO ❌

Requested by the /ai-fix Hydra workflow for airbytehq/oncall#13774 (epic airbytehq/oncall#13730).

Link to Devin session: https://app.devin.ai/sessions/b7a01b6852ab43afbcd6b3ac1ec4b783
Open in Devin Desktop: https://app.devin.ai/desktop/session/b7a01b6852ab43afbcd6b3ac1ec4b783?variant=devin

…cceptance_tests layout Co-Authored-By: bot_apk <apk@cognition.ai>

devin-ai-integration · 2026-10-20T15:27:33Z

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

Disable automatic comment, CI, and merge conflict monitoring

github-actions · 2026-10-20T15:28:06Z

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

🛠️ Quick Fixes
- /format-fix - Fixes most formatting issues.
- /bump-version - Bumps connector versions, scraping changelog description from the PR title.
  - Bump types: patch (default), minor, major, major_rc, rc, promote.
  - The rc type is a smart default: applies minor_rc if stable, or bumps the RC number if already RC.
  - The promote type strips the RC suffix to finalize a release.
  - Example: /bump-version type=rc or /bump-version type=minor
- /bump-progressive-rollout-version - Alias for /bump-version type=rc. Bumps with an RC suffix and enables progressive rollout.
❇️ AI Testing and Review (internal link: AI-SDLC Docs):
- /ai-prove-fix - Runs prerelease readiness checks, including testing against customer connections.
- /ai-canary-prerelease - Rolls out prerelease to 5-10 connections for canary testing.
- /ai-review - AI-powered PR review for connector safety and quality gates.
- /ai-breaking-change-review - Step 1 for a connector breaking change. Reviews whether the change needs to be breaking, destination data-loss risk, and whether the migration guide, metadata message, and docs let users upgrade safely. Opens one conversation on the PR for each item the author needs to handle.
- /ai-breaking-change-ready - Step 2, after you've replied to and resolved each of those conversations. Checks that every one has a reply and is resolved, then pings the breaking-change reviewers in Slack for human review.
📝 AI Documentation:
- /ai-docs-review - AI-powered documentation review for PRs with connector changes.
- /ai-create-docs-pr - Creates a documentation PR for connector changes, stacked on the current PR.
🚀 Connector Releases:
- /publish-connectors-prerelease - Publishes pre-release connector builds (tagged as {version}-preview.{git-sha}) for all modified connectors in the PR.
- /enable-autopilot-rollouts - Enables autopilot progressive rollouts for the modified connector(s) in the PR, remediating "autopilot rollouts not enabled for {connector-name}" auto-merge blockers. Sets defaultRolloutMode: autopilot and enableProgressiveRollout: true, preserving any existing autopilotConfig.
  - Optional args: connector=<CONNECTOR_NAME> (defaults to the modified connectors in the PR), strategy=fast|slow|default (defaults to fast).
  - Example: /enable-autopilot-rollouts or /enable-autopilot-rollouts connector=source-faker strategy=slow
☕️ JVM connectors:
- /update-connector-cdk-version connector=<CONNECTOR_NAME> - Updates the specified connector to the latest CDK version.
  Example: /update-connector-cdk-version connector=destination-bigquery
🐍 Python connectors:
- /poe connector source-example lock - Run the Poe lock task on the source-example connector, committing the results back to the branch.
- /poe source example lock - Alias for /poe connector source-example lock.
- /poe source example use-cdk-branch my/branch - Pin the source-example CDK reference to the branch name specified.
- /poe source example use-cdk-latest - Update the source-example CDK dependency to the latest available version.
⚙️ Admin commands:
- /force-merge reason="<REASON>" - Force merges the PR using admin privileges, bypassing CI checks. Requires a reason.
  Example: /force-merge reason="CI is flaky, tests pass locally"

📚 Show Repo Guidance

Helpful Resources

Breaking Changes Guide - Breaking changes, migration guides, and upgrade deadlines
Developing Connectors Locally
Managing Connector Secrets
On-Demand Regression Tests
#connector-ci-issues
#connector-publish-updates
#connector-build-statuses

📝 Edit this welcome message.

github-actions · 2026-10-20T15:28:21Z

Note

Autopilot progressive rollouts are not enabled for the following modified connector(s):

source-activecampaign

This is a courtesy heads-up only — it does not block merge or fail any check.
To enable automatic progressive rollouts for the connector(s) above, comment
/enable-autopilot-rollouts on this PR. This sets defaultRolloutMode: autopilot
and enableProgressiveRollout: true in each connector's metadata.yaml,
preserving any existing autopilotConfig.

github-actions · 2026-10-20T15:32:49Z

`source-activecampaign` Connector Test Results

10 tests 4 ✅ 45s ⏱️
1 suites 6 💤
1 files 0 ❌

Results for commit f4ec739.

♻️ This comment has been updated with latest results.

Airbyte Support Bot (airbyte-support-bot) · 2026-10-21T11:38:12Z

↪️ Triggering /ai-prove-fix per Hands-Free AI Triage Project daily triage.

Reason: Draft fix PR with all required CI checks green at current HEAD and no /ai-prove-fix result yet.

Devin session: https://app.devin.ai/sessions/18d8ffd2eaa34edea8dbb8fea93fc2fe

octavia-bot · 2026-10-21T11:38:39Z

🔍 AI Prove Fix session starting... Running readiness checks and testing against customer connections. View playbook

🔗 Session URL: https://app.devin.ai/sessions/75faada7b8e24013b8d9a6c8001d7003

✅ Devin AI session created successfully!

Airbyte Support Bot (airbyte-support-bot) · 2026-10-21T11:39:24Z

🧪 Fix Validation Evidence

🟢 Outcome: Fix Proven

This PR changes only acceptance-test-config.yml. On the up-to-date PR, the Test source-activecampaign Connector CI job failed test collection with ValueError: Acceptance tests config not found. On this PR's HEAD the same job collects and passes (7 passed, 14 skipped / 4 passed, 6 skipped), and Connector CI Checks Summary is green. Regression tests found no SPEC/DISCOVER difference. CHECK/READ failed on both versions because the GSM credential is broken, which is a known, separate issue.

🚦 Next Steps

This PR appears ready for review and merge (mark ready for review — it is still a draft).
Follow-up: refresh the GSM credential SECRET_SOURCE-ACTIVECAMPAIGN__CREDS and restore the bypassed check succeed / basic_read scenarios.
No connector release results from this PR (no version bump; runtime image unchanged), so neither /ai-canary-prerelease nor rollout monitoring is needed.

📋 Connector & PR Details

Connector: source-activecampaign
PR: #89554
PR HEAD Validated: f4ec739a0a51a1de762cd6e53b0b5737ad23ce91
Pre-release Version Tested: n/a — regression tests built the target from PR HEAD
Comparison Baseline: latest (auto) (resolved to airbyte/source-activecampaign:0.2.14)
Evidence Harness: run_regression_tests + PR CI (Test source-activecampaign Connector)
Read Result: FAIL — 0 RECORD / 0 STATE (both versions failed on GSM credentials; inconclusive, not a regression)
True-Positive Case: n/a — fix does not loosen a connector check (test-config only; invalid-config check still runs and passes as status: failed in CI)
Detailed Results: https://github.com/airbytehq/oncall/issues/13774#issuecomment-6315148460

📝 Evidence Plan

Proving Criteria

Test source-activecampaign Connector on PR HEAD collects and runs the standard tests with no Acceptance tests config not found error, and Connector CI Checks Summary passes. The regression comparison shows no new difference between target and control.

Disproving Criteria

Collection still fails on HEAD, or the regression comparison shows a real target-vs-control difference.

Cases Attempted

Before (CI, up-to-date PR deps(source-activecampaign): 🐙 update dependencies [2026-10-20] #78556, master config): collected 0 items / 1 error → ValueError: Acceptance tests config not found in acceptance-test-config.yml.
After (CI, PR HEAD): 7 passed, 14 skipped (connector test) and 4 passed, 6 skipped (image test); job success.
Regression attempt 1 (GSM, all streams): SPEC ✅, DISCOVER ✅, CHECK ❌ both failed (Bad gateway), READ ❌ both failed. Internal failure: false.
No customer-connection escalation: the target build has the same connector code as 0.2.14 (only the test config differs). A customer-connection comparison could not exercise this change, which affects CI test collection only.

True-Positive Case

Not applicable. No connector validation logic changes. The invalid_config.json → status: failed scenario is still in the config and passes in CI.

✅ Pre-flight Checks

Viability: Fix addresses the reported issue (legacy tests: → acceptance_tests: layout)
Safety: No malicious code or dangerous patterns (YAML test config only)
Breaking Change: No breaking changes (no schema/spec/stream/state/runtime change)
Pin Exclusion: No connections pinned or queried
Reversibility: Trivially revertible; no release produced

⚠️ Coverage note: the GSM check succeed and basic_read scenarios now carry a bypass_reason, so CI no longer exercises live check/read until the credential is refreshed.

📊 Detailed Evidence Log

2026-10-20 — before: CI job on #78556 failed at collection.
PR HEAD f4ec739 — after: CI job passed; Connector CI Checks Summary success.
2026-10-21 — regression run: SPEC/DISCOVER pass; CHECK/READ both-failed (GSM creds).

Note: no customer connections were used. Details are in the linked private issue.

Airbyte Support Bot (airbyte-support-bot) · 2026-10-21T13:14:08Z

↪️ Triggering /ai-review per Hands-Free AI Triage Project daily triage.

Reason: /ai-prove-fix returned PROVEN at the current HEAD; no review yet (recheck 1).

Devin session: https://app.devin.ai/sessions/18d8ffd2eaa34edea8dbb8fea93fc2fe

octavia-bot · 2026-10-21T13:14:28Z

AI PR Review starting...

Reviewing PR for connector safety and quality.
View playbook

🔗 Session URL: https://app.devin.ai/sessions/e3a8d7f805b449c89899dd32b7e3474d

✅ Devin AI session created successfully!

Airbyte Support Bot (airbyte-support-bot) · 2026-10-21T13:15:00Z

🛡️ AI PR Review Report

🟢 Review Action: APPROVED

All 12 gates passed.

🟦 Risk Level: 1/5

Only source-activecampaign/acceptance-test-config.yml changes (legacy tests: → acceptance_tests: layout plus bypass_reason entries). No runtime code, spec, schema, or metadata changes, and no release is produced.

📋 PR Details

Connector(s): source-activecampaign
PR: #89554
HEAD SHA: f4ec739a0a51a1de762cd6e53b0b5737ad23ce91
Session: https://app.devin.ai/sessions/e3a8d7f805b449c89899dd32b7e3474d
Risk Level rationale: Test-config-only change. CAT check (succeed) and basic_read are bypassed because the GSM credential is broken. This lowers CI test coverage until the credential is refreshed (tracked under Follow-ups in the PR description), but it has no production impact.

🔍 Gate Evaluation Details

Gate-by-Gate Analysis

Gate	Status	Enforced?	Details
PR Hygiene	PASS	Yes	The description is detailed. No changelog is needed because only the non-production `acceptance-test-config.yml` changed (`Check Changelog Updated` passed). There are no human review threads.
Code Hygiene	PASS	WARNING	Exempt: no connector source paths were modified. The only changed file is test config.
Test Coverage	PASS	Yes	Exempt: the PR only modifies test configuration, so there is no source change to validate.
Code Security	PASS	Yes	No security-sensitive paths, and no auth/credential keywords in functional diff lines.
Per-Record Performance	PASS	WARNING	No record-processing code changed.
Breaking Dependencies	PASS	WARNING	No dependency files changed.
Backwards Compatibility	PASS	Yes (enforced)	No spec, schema, stream, PK/cursor, or `metadata.yaml` changes. The released image is unchanged.
Forwards Compatibility	PASS	Warning (elevates Risk Level)	`incremental`/`full_refresh`/`future_state_path` appear only in removed commented-out TODO lines, which are excluded.
Behavioral Changes	PASS	Warning (elevates Risk Level)	No operational-risk keywords in functional diff lines.
Out-of-Scope Changes	PASS	Skip	The single file is under `airbyte-integrations/connectors/source-activecampaign/`.
CI Checks	PASS	Yes	Connector CI Checks Summary, Test source-activecampaign Connector, and Lint source-activecampaign Connector all succeeded.
Live / E2E Tests	PASS	Yes	`/ai-prove-fix` reported Fix Proven for HEAD `f4ec739a`.

PR Hygiene evidence

PR Body Length (raw): ~3,500 chars. After stripping comments and whitespace: ~3,450. Visible content: ~3,300.
PR Body Preview: ## Overview <table><tr><td> **:point_right: TL;DR:** The ActiveCampaign connector's test settings file used an old
Changelog: not required. The only file changed is acceptance-test-config.yml, a non-production path, and Check Changelog Updated passed.
Peer feedback: 0 review threads and no human review comments.

Test Coverage evidence

Behavioral Change Detected: No runtime change. The indicator found is a linked issue (Resolves airbytehq/oncall#13774), but the exemption applies because the change is test-only.
Test Files Modified: airbyte-integrations/connectors/source-activecampaign/acceptance-test-config.yml (test config)
New Test Content Found: N/A (exempt)

Diff summary

The top-level tests: key becomes acceptance_tests:, with each section wrapped in tests: [...].
Kept: spec (manifest.yaml), connection with integration_tests/invalid_config.json → status: failed, and discovery with secrets/config.json.
Replaced with bypass_reason: the connection secrets/config.json → succeed case and basic_read. The reason is that the GSM credential's account_username is a full hostname (SSL hostname mismatch), and the account returns 402.
Removed: the commented-out CAT TODO blocks.

Live / E2E
The /ai-prove-fix comment validated HEAD f4ec739a0a51a1de762cd6e53b0b5737ad23ce91. Before the fix, CI hit ValueError: Acceptance tests config not found. After the fix, CI reports 7 passed, 14 skipped and 4 passed, 6 skipped. Regression tests showed SPEC/DISCOVER parity, and CHECK/READ failed on both versions because of the broken GSM credential.

📚 Evidence Consulted

Evidence

Changed files: 1 file (airbyte-integrations/connectors/source-activecampaign/acceptance-test-config.yml, +10/−21)
CI checks: Connector CI Checks Summary ✅, Test source-activecampaign Connector ✅, Lint source-activecampaign Connector ✅, Format Check ✅, Check Changelog Updated ✅, source-activecampaign Connector Test Results ✅. Excluded as non-core: Progressive Rollout Gate ✅, Validate PR Title ✅, and "Add Labels to PRs. Safe to Merge on fail" ❌ (a labeler job, not a build/lint/test check).
PR labels: connectors/source/activecampaign, hyd-fix, hyd-prove, hyd-review (applied by this review, also on airbytehq/oncall#13774)
PR description: present
Existing bot reviews: none

test(source-activecampaign): move acceptance-test-config.yml to the a…

f4ec739

…cceptance_tests layout Co-Authored-By: bot_apk <apk@cognition.ai>

devin-ai-integration Bot assigned Airbyte Support Bot (airbyte-support-bot) Oct 20, 2026

devin-ai-integration Bot added the hyd-fix Hydra: ai-fix stage has run label Oct 20, 2026

Airbyte Support Bot (airbyte-support-bot) added the hyd-prove Hydra: ai-prove-fix stage has run label Oct 21, 2026

octavia-bot Bot marked this pull request as ready for review October 21, 2026 13:14

Octavia Squidington III (octavia-squidington-iii) added the connectors/source/activecampaign label Oct 21, 2026

Airbyte Support Bot (airbyte-support-bot) added the hyd-review label Oct 21, 2026

Conversation

devin-ai-integration Bot commented Oct 20, 2026

Overview

Changes

Risks

Follow-ups

Test Coverage

Declarative-First Evaluation

Breaking change

Can this PR be safely reverted and rolled back?

Uh oh!

devin-ai-integration Bot commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026

👋 Greetings, Airbyte Team Member!

PR Slash Commands

Helpful Resources

Uh oh!

github-actions Bot commented Oct 20, 2026

Uh oh!

github-actions Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

source-activecampaign Connector Test Results

Uh oh!

Airbyte Support Bot (airbyte-support-bot) commented Oct 21, 2026

Uh oh!

octavia-bot Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Airbyte Support Bot (airbyte-support-bot) commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧪 Fix Validation Evidence

🟢 Outcome: Fix Proven

Proving Criteria

Disproving Criteria

Cases Attempted

True-Positive Case

Uh oh!

Airbyte Support Bot (airbyte-support-bot) commented Oct 21, 2026

Uh oh!

octavia-bot Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Airbyte Support Bot (airbyte-support-bot) commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🛡️ AI PR Review Report

🟢 Review Action: APPROVED

🟦 Risk Level: 1/5

Gate-by-Gate Analysis

Evidence

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Oct 20, 2026 •

edited

Loading

`source-activecampaign` Connector Test Results

octavia-bot Bot commented Oct 21, 2026 •

edited

Loading

Airbyte Support Bot (airbyte-support-bot) commented Oct 21, 2026 •

edited

Loading

octavia-bot Bot commented Oct 21, 2026 •

edited

Loading

Airbyte Support Bot (airbyte-support-bot) commented Oct 21, 2026 •

edited

Loading