Skip to content

Prevent supplier-scorecard verifier crashes on suppliers outside the comparison set - #17

Merged
sfc-gh-dhuang merged 1 commit into
Snowflake-Labs:masterfrom
genesis-gh-jlangseth:feature/supplier-scorecard-comparison-scope
Nov 12, 2026
Merged

sfc-gh-dhuang merged 1 commit into
Snowflake-Labs:masterfrom
genesis-gh-jlangseth:feature/supplier-scorecard-comparison-scope

Conversation

@genesis-gh-jlangseth

Copy link
Copy Markdown
Contributor

Summary

Three supplier-scorecard source comparisons convert metrics to numbers before checking whether the supplier has a source expectation. A supplier outside that comparison set can therefore crash verification when its metrics are NULL, even though the following guard would skip the row.

Changes

1. Apply the existing comparison scope before numeric conversion (fix)

Scope: supplier-scorecard grading | Impact: benchmark reliability

Move delivery, quality, and cost metric conversions inside their existing source-membership guards.

  • Suppliers outside each source comparison set are skipped before their metrics are dereferenced.
  • In-scope values, tolerances, and mismatch assertions retain their existing behavior.
  • No new NULL-to-zero conversion or other acceptance rule is introduced.
  • Twelve regression checks cover all three comparisons, including an out-of-scope NULL row followed by a valid supplier, matching and mismatched values, and preservation of the existing in-scope NULL failure.

Validation

On upstream a3278ad, the new regression checks produce 3 failed, 9 passed before the fix and 12 passed afterward, with no skips. The three original failures are the actual numeric-conversion exceptions in the verifier. New regression code passes Ruff.

The checks call the actual verifier comparison methods with controlled rows and source expectations. They do not run dbt, a live warehouse, a Harbor trial, or an agent. No overall task-score increase or complete live task-suite pass is claimed.

Migration / Breaking Changes

None.

Changelog Notes

  • Fixed: Supplier-scorecard source comparisons no longer crash on nullable metrics belonging to suppliers outside their comparison scope.

Refs

@sfc-gh-dhuang
sfc-gh-dhuang merged commit 8775560 into Snowflake-Labs:master Nov 12, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants