Repository navigation
fix(adapter): name dbt v2 as the cause when the dbt import fails - #1558
Conversation
Recce OSS cannot run on dbt v2 (Fusion), because dbt v2 drops dbt.adapters. The import failure told users that dbt was not installed. Recce now detects dbt v2 and says that it is not supported yet. The server startup log now shows any RecceException as one line, with the traceback in the debug log. The schema guard message no longer blames dbt v2 for a manifest above v12, because dbt v2 writes manifest v12 too. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Wei-Chun, Chang <wcchang@infuseai.io>
The comment above _DBT1X_MAX_SCHEMA did not say what must not depend on the installed dbt. It now names the limits and the error that an old dbt must raise. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Wei-Chun, Chang <wcchang@infuseai.io>
…eason-recce-oss-does-not-start
Codecov Report❌ Patch coverage is
... and 3 files with indirect coverage changes 🚀 New features to boost your workflow:
|
The server startup log dropped the traceback for every RecceException, including Recce Cloud state download failures, where the traceback helps find the cause. DbtUnavailableError now marks the dbt import failure, and only that error logs as one line. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Wei-Chun, Chang <wcchang@infuseai.io>
Sentry tracks bugs, and a dbt import failure comes from an unsupported setup. The server now logs it as a warning, which the Sentry logging integration does not send as an event. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Wei-Chun, Chang <wcchang@infuseai.io>
…eason-recce-oss-does-not-start
iamcxa
left a comment
There was a problem hiding this comment.
@wcchang1115 Automated differential review of 93a1e4b (base 7c0571e). Verified by running the real import path, not just reading the diff.
What this PR does
- Converts the module-level dbt
ImportErrorinrecce/adapter/dbt_adapter/__init__.pyinto aDbtUnavailableError(aRecceException), with a dedicated message when dbt v2 / Fusion is detected viaimportlib.util.find_spec("dbt._core"). - Routes
DbtUnavailableErrorat server startup tologger.warning(traceback demoted tologger.debug) so it stays out of Sentry. - Corrects the
_DBT1X_MAX_SCHEMAcomment: dbt v2 writes manifest v12 / catalog v1, the same as dbt-core 1.x, so the ceiling is no longer a Fusion detector.
Goal achievement: ✅ Achieved — the stated goal is realised on both entry paths.
| Probe | Result |
|---|---|
Simulated dbt v2 (dbt._core present, no dbt.adapters) → import recce.adapter.dbt_adapter |
DbtUnavailableError: Recce supports dbt-core 1.x. dbt v2 (Fusion) is not supported yet., __cause__ = ModuleNotFoundError: No module named 'dbt.adapters' |
Same, via recce run |
[Error] Recce supports dbt-core 1.x. dbt v2 (Fusion) is not supported yet. — one line, no traceback, exit 1, and recce/track.py:127 takes the except RecceException branch, so event.capture_exception is not called ✅ |
Same, via recce server |
WARNING: Failed to load server context during startup: Recce supports dbt-core 1.x. dbt v2 (Fusion) is not supported yet. — one line, no traceback ✅ |
| dbt absent entirely | falls back to the install-hint message ✅ |
find_spec("dbt._core") against real dbt-core 1.8.8 |
False — no false positive on a supported install ✅ |
PR-body claim "dbt_version on the Amplitude command event already counts dbt v2 users" |
Grounded — recce/track.py:22 _installed_dbt_version() reads package metadata, not an import, and the props dict is built in the finally block, so it still fires after sys.exit(1) ✅ |
Verification summary
| Check | Result |
|---|---|
pytest tests/test_dbt.py |
8/8 pass |
pytest tests/ on PR head |
1526 passed, 28 failed, 158 errors |
pytest tests/ on base 7c0571e |
1526 passed, 28 failed, 158 errors — identical, no regression (pre-existing sandbox env failures) |
| flake8 / black / isort on the 4 changed files | clean |
| CI | all green, DCO signed |
Findings
One MEDIUM and one LOW are posted inline. Advisory items below are not blockers.
Advisory (not posted as inline comments)
A. The schema guard is now dormant — is the sibling case in scope? With _DBT1X_MAX_SCHEMA = {"manifest": 12, "catalog": 1} and the PR's own (correct) statement that dbt v2 writes v12/v1, _guard_unsupported_schema can no longer fire for any artifact a released dbt emits — the tests had to move from v20 to a synthetic v13/v2 that nothing produces. That leaves the other half of DRC-4656 unhandled: artifacts built by dbt v2 but read by a Recce running dbt-core 1.x (recce server --review, or CI generating the manifest). Those now fall through to dbt's own parser with no Recce-branded message. Worth confirming this is deliberately out of scope rather than an accidental gap.
B. Docstrings went backwards. DbtUnavailableError is pass with no docstring, while every other exception in recce/exceptions.py documents when it is raised. UnsupportedDbtSchemaError's docstring was reduced to """The version limits are in _DBT1X_MAX_SCHEMA in the dbt adapter.""", which is a pointer but no longer says what the error means. One line each would keep the file self-describing.
C. recce/server.py:299-300 logs the same event twice. In debug mode the user sees both the logger.debug line (with traceback) and the logger.warning line carrying the same text. Folding the message into the debug call, or dropping the duplicate prefix, would be tidier.
Reviewed with kc-pr-review. No fix was pushed; no merge or force-push performed.
| is_dbt_v2 = False | ||
| if is_dbt_v2: | ||
| raise DbtUnavailableError("Recce supports dbt-core 1.x. dbt v2 (Fusion) is not supported yet.") from e | ||
| raise DbtUnavailableError( |
There was a problem hiding this comment.
MEDIUM — a non-"dbt is missing" ImportError now loses its real cause.
This except ImportError block catches every failure of import agate, import dbt.adapters.factory, and from dbt.contracts.state import PreviousState — not just "dbt is absent". The most common real-world case is a transitive-dependency version skew, and that case now reaches the user as an instruction that will not fix it.
RecceException.__str__ is not overridden, so __cause__ never appears in str(e), and recce/track.py:128-129 sends the traceback to logger.debug while printing only str(e). Reproduced by shadowing agate with a module that raises ImportError("cannot import name 'MappedSequence' from 'agate._vendor' — dbt-core/agate version skew"), dbt itself installed and intact:
# base 7c0571e
ImportError: cannot import name 'MappedSequence' from 'agate._vendor' - dbt-core/agate version skew
Error: dbt module not found. Please install it by running:
pip install dbt-core dbt-<adapter>
# this PR (93a1e4b)
[Error] dbt module not found. Please install it by running:
pip install dbt-core dbt-<adapter>
The real cause is now invisible, and pip install dbt-core dbt-<adapter> is a dead end for it. Secondary effect: because DbtUnavailableError is a RecceException, recce/track.py:127 skips event.capture_exception, so these failures also stop reaching Sentry — but unlike the dbt v2 case the PR body justifies, a dependency-skew ImportError genuinely is a bug worth seeing.
Suggested fix — only claim "not found" when dbt really is absent:
if is_dbt_v2:
raise DbtUnavailableError("Recce supports dbt-core 1.x. dbt v2 (Fusion) is not supported yet.") from e
if importlib.util.find_spec("dbt.adapters") is None:
raise DbtUnavailableError(
"dbt module not found. Please install it by running:\n pip install dbt-core dbt-<adapter>"
) from e
raise # unexpected import failure - keep the original error and tracebackAt minimum, append the cause: f"...\n Original error: {e}".
There was a problem hiding this comment.
Fixed in 363ec16. Only a missing dbt and dbt v2 now raise DbtUnavailableError, and any other ImportError is raised as is, with its traceback and its Sentry report.
|
|
||
|
|
||
| def _fusion_artifact(kind: str) -> dict: | ||
| _ABOVE_CEILING = {"manifest": 13, "catalog": 2} |
There was a problem hiding this comment.
LOW — none of the new behaviour is covered by a test.
Every test change in this PR is a rename of the pre-existing schema-guard tests (TestFusionManifestFailLoud → TestNewerSchemaFailLoud, _fusion_artifact → _newer_artifact, v20 → v13/v2). The three branches the PR actually adds have zero coverage, which matches Codecov's 8 missing lines in dbt_adapter/__init__.py and 4 in server.py:
- dbt v2 detected → Fusion message
- dbt absent → install-hint message
DbtUnavailableErrorat startup →logger.warninginstead oflogger.exception
All three are testable without dbt v2 installed. For 1 and 2 I exercised the real path with a PYTHONPATH shim (/tmp/fakev2/dbt/{__init__,_core}.py, no dbt/adapters) plus a fresh interpreter — monkeypatch.syspath_prepend + importlib.reload does the same inside pytest. 3 is a plain isinstance branch that a small unit test over the startup handler can assert with caplog.
The PR checklist ticks "added or ran the appropriate tests"; worth either adding them or saying explicitly that the import-time branches are intentionally untested.
There was a problem hiding this comment.
Added two subprocess tests in 363ec16: one for the dbt v2 message, and one that checks that a non-dbt ImportError is raised as is. The second test fails without the fix.
The dbt import handler turned every ImportError into "dbt module not found". A broken dependency, such as a dbt-core and agate version mismatch, lost its real cause and its Sentry report. Only a missing dbt and dbt v2 now become DbtUnavailableError. Any other import failure is raised unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Wei-Chun, Chang <wcchang@infuseai.io>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Wei-Chun, Chang <wcchang@infuseai.io>
A warning hid that the server cannot load. The server now prints the message as an error on the console. Sentry does not read console output, so it still does not report this failure. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Wei-Chun, Chang <wcchang@infuseai.io>
iamcxa
left a comment
There was a problem hiding this comment.
@wcchang1115 Re-review of 4e58045 (delta 93a1e4b..4e58045, append-only; 3 new commits). All prior findings are resolved. Verified by running the real import path on the new head.
Prior findings
| # | Finding | Status | Evidence |
|---|---|---|---|
| MEDIUM | Non-dbt ImportError lost its real cause |
✅ Fixed (363ec165) |
recce run with a shadowed agate raising a version-skew ImportError now shows the original ImportError and traceback, not the install hint |
| LOW | New behaviour untested | ✅ Addressed | test_dbt_v2_import_names_dbt_v2 and test_unexpected_import_error_keeps_original_error exercise the real module import in a subprocess |
| A | Schema guard dormant | ✅ Resolved by explanation | dbt v2 writes manifest v12 / catalog v1, so no error is expected there |
| B | Docstrings | ✅ Fixed (af83136c) |
|
| C | Duplicate server log line | ✅ Fixed (4e580457) |
message printed once as [Error] ..., traceback only at debug level |
Runtime probes on 4e58045
| Scenario | Result |
|---|---|
dbt v2 shim (dbt._core, no dbt.adapters) → recce run |
[Error] Recce supports dbt-core 1.x. dbt v2 (Fusion) is not supported yet. — no traceback, exit 1 |
dbt v2 shim → recce server |
[Error] Failed to load server context during startup: Recce supports dbt-core 1.x. ... |
dbt package removed → recce run |
[Error] dbt module not found. Please install it by running: pip install dbt-core dbt-<adapter> |
agate version skew, dbt intact |
original ImportError re-raised with its traceback |
dbt/__init__.py itself raises ImportError |
original ImportError surfaces (chained) — not mislabelled |
Verification
| Check | Result |
|---|---|
pytest tests/test_dbt.py |
10/10 pass |
| flake8 / black / isort (changed files) | clean |
| CI | 21 pass, 3 skipped |
Advisory (non-blocking)
- The two new tests run in a subprocess, so coverage tools won't count those adapter lines, and the
server.pyDbtUnavailableErrorbranch plus the "dbt absent" branch still have no direct test. Fine to leave, but Codecov patch % will keep reporting them as missing.
LGTM.
Reviewed with kc-pr-review. No fix pushed; no merge or force-push performed.
PR checklist
What type of PR is this?
fix
What this PR does / why we need it:
With dbt v2 (Fusion) installed, Recce OSS fails at startup. Recce told users that dbt was not installed, and printed a long traceback.
This PR detects dbt v2 and shows a short message instead: Recce supports dbt-core 1.x, and dbt v2 is not supported yet.
This PR also corrects the schema version error. It said a manifest above v12 comes from dbt v2, but dbt v2 writes manifest v12, the same as dbt-core 1.x.
Which issue(s) this PR fixes:
Fixes DRC-4656
Special notes for your reviewer:
dbt_versionproperty on the Amplitudecommandevent already counts dbt v2 users.Does this PR introduce a user-facing change?:
🤖 Generated with Claude Code