Skip to content

fix(low-code): stop leaking interpolation context in undeclared-variable errors - #1241

Draft
devin-ai-integration[bot] wants to merge 1 commit into
mainfrom
devin/1792510626-jinja-undeclared-vars-no-context
Draft

devin-ai-integration[bot] wants to merge 1 commit into
mainfrom
devin/1792510626-jinja-undeclared-vars-no-context

Conversation

@devin-ai-integration

Copy link
Copy Markdown
Contributor

Summary

JinjaInterpolation._eval raised ValueError(f"Jinja macro has undeclared variables: {...}. Context: {context}"), embedding the whole interpolation context — including the full connector config (OAuth tokens, client secrets) and stream_slice. That leaked secrets and made the message non-deterministic (unordered set + varying values → no Sentry grouping).

On top of that, two concurrent-path trace emitters built TRACE messages with as_airbyte_message(), which skips filter_secrets, so even airbyte_secret fields reached the platform unmasked in internal_message / stack_trace (while the log lines for the same error were masked).

Changes:

# jinja.py — exception type unchanged (ValueError)
- f"Jinja macro has undeclared variables: {undeclared_not_in_context}. Context: {context}"
+ f"Jinja macro has undeclared variables: {sorted(undeclared_not_in_context)}. "
+ f"Template: {s!r}. Available variables: {sorted(context.keys())}."
  • Example: Jinja macro has undeclared variables: ['components_values']. Template: "{{ components_values['x'] }}". Available variables: ['config', 'stream_interval', 'stream_partition', 'stream_slice'].
  • ConcurrentReadProcessor.on_exception (both branches) and AsyncJobOrchestrator._keep_api_budget_with_failed_job: as_airbyte_message() → as_sanitized_airbyte_message(), matching AbstractSource and AirbyteEntrypoint.

Resolves https://github.com/airbytehq/oncall/issues/13791:

Close on merge: yes — internally filed, no customer to confirm with.

Review notes

  • The exception stays ValueError, so callers that catch it are unaffected; only the message text changes.
  • Not masked by this PR: values that aren't registered as secrets (e.g. a runtime access_token that isn't marked airbyte_secret). The jinja change stops these from ending up in this error. Other places that emit as_airbyte_message() (connector builder, test entrypoint wrapper) are left unchanged.
  • No connector version bump; the CDK release is cut from the PR title (fix: → patch).

Test Coverage

  • test_jinja.py::test_undeclared_variables_do_not_include_context_values: fake secrets / slice values are not in the message, and the message is identical across two different contexts (sorted names).
  • test_concurrent_read_processor.py: on_exception masks a registered secret for both AirbyteTracedException and plain exceptions (message, internal_message, stack_trace).
  • test_job_orchestrator.py: a job-start failure carrying a registered secret produces a masked TRACE message on the message repository.
  • All 5 new tests fail on main and pass with this change. Targeted suites (interpolation, concurrent read processor, async_job): 289 passed. ruff + mypy are clean.

Link to Devin session: https://app.devin.ai/sessions/567abed3e2c74b01a4c72b81f7a0deed
Open in Devin Desktop: https://app.devin.ai/desktop/session/567abed3e2c74b01a4c72b81f7a0deed?variant=devin

…ble errors

Co-Authored-By: bot_apk <apk@cognition.ai>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@github-actions

Copy link
Copy Markdown

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

💡 Show Tips and Tricks

Testing This CDK Version

You can test this version of the CDK using the following:

# Run the CLI from this branch:
uvx 'git+https://github.com/airbytehq/airbyte-python-cdk.git@devin/1792510626-jinja-undeclared-vars-no-context#egg=airbyte-python-cdk[dev]' --help

# Update a connector to use the CDK from this branch ref:
cd airbyte-integrations/connectors/source-example
poe use-cdk-branch devin/1792510626-jinja-undeclared-vars-no-context

PR Slash Commands

Airbyte Maintainers can execute the following slash commands on your PR:

  • /autofix - Fixes most formatting and linting issues
  • /poetry-lock - Updates poetry.lock file
  • /test - Runs connector tests with the updated CDK
  • /prerelease - Triggers a prerelease publish with default arguments
  • /poe build - Regenerate git-committed build artifacts, such as the pydantic models which are generated from the manifest JSON schema in YAML.
  • /poe <command> - Runs any poe command in the CDK environment
📚 Show Repo Guidance

Helpful Resources

📝 Edit this welcome message.

@github-actions

Copy link
Copy Markdown

PyTest Results (Fast)

5 843 tests  +5   5 812 ✅ +5   9m 40s ⏱️ + 1m 31s
    1 suites ±0      31 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 80f96e3. ± Comparison against base commit 8b721cd.

@github-actions

Copy link
Copy Markdown

PyTest Results (Full)

5 846 tests  +5   5 815 ✅ +5   16m 20s ⏱️ -21s
    1 suites ±0      31 💤 ±0 
    1 files   ±0       0 ❌ ±0 

Results for commit 80f96e3. ± Comparison against base commit 8b721cd.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant