fix(gateway): pass the invalid_guard_verdict worker code through - #297
Conversation
Guard models emit invalid_guard_verdict when their first output positions carry no complete Yes/No distribution (#285), and the SDK READMEs tell callers to expect that code. The gateway's worker error allowlist predates it, so the terminal collapsed to a generic inference_error with the "internal error during generation" message on both the buffered and the streaming path. Admit the code so the typed terminal and its message reach the client. It settles like empty_model_output: terminal, non-retryable, server_error.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (2)
Included review availability: Your plan provides up to 10 included reviews per hour; 7 remain after this review. 📝 WalkthroughWalkthroughThe gateway adds ChangesGuard verdict error code handling
Suggested reviewers: Priority: ⬇️ Low Merge Risk: ⚪ Minimal · up to The new terminal error code is preserved for clients across buffered and streaming responses without changing the intended server-error behavior. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Why
Since #285 a guard model whose first output positions carry no complete Yes/No distribution terminates with
error_code="invalid_guard_verdict"and"guard model did not produce a valid thresholded verdict". The Python and TypeScript SDK READMEs already tell callers to handle that code. The gateway's worker error allowlist (queue/streaming.rs::client_safe_worker_error_code) predates it, so the terminal collapsed toinference_error/ HTTP 500 / "internal error during generation" on the buffered path and "Generation terminated with an upstream error." on the streaming path. A caller has no way to tell "the guard could not produce a verdict" from a backend fault.Reproducible with
POST /v1/generate/ibm-granite__granite-guardian-3.0-2b {"prompt": "...", "max_new_tokens": 4}(a raw prompt has no Guardian template, so the model answers in prose): the body is the generic 500 instead of the typed code.What
http_error.rs:openai_code::INVALID_GUARD_VERDICT.queue/streaming.rs: admit the code inclient_safe_worker_error_codeand pin it in the contract test.Status and OpenAI type stay the defaults (500 /
server_error), the same settlement asempty_model_output: terminal, non-retryable, tokens were consumed.Verification
cargo test --lib worker_error_public_contract: 2 passed.cargo fmt --checkandcargo clippy --lib -D warnings: clean.Summary by CodeRabbit