fix: Populate the switchyard.router_retry_recovered metric

[grahamking]

This counts when a remote model call failed and needed retrying.

Assisted-by: Codex:GPT 5.6 Sol medium
Signed-off-by: Graham King grahamk@nvidia.com

Summary by CodeRabbit

• Metrics
  • Added tracking for upstream operations that succeed after one or more retries.
  • Updated metric documentation to describe retry recovery accurately.
• Tests
  • Added coverage confirming the retry-recovery metric increments after a transient failure is successfully resolved.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 80c457a8-e33b-44cc-908e-7ccf04b1a59c

📥 Commits

Reviewing files that changed from the base of the PR and between 395c202 and 37b3088.

📒 Files selected for processing (5)

• crates/libsy-llm-client/src/client.rs
• crates/libsy-llm-client/src/metrics.rs
• crates/switchyard-server/README.md
• crates/switchyard-server/tests/server.rs
• docs/internal/metrics_reference.md

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.

---

Walkthrough

Successful upstream requests now increment a retry-recovery metric when they follow one or more retries. Integration tests simulate a transient 503 and verify the counter. Metric documentation now describes the counter behavior.

Changes

Retry recovery metric

Layer / File(s)	Summary
Retry recovery instrumentation crates/libsy-llm-client/src/metrics.rs, crates/libsy-llm-client/src/client.rs, crates/switchyard-server/README.md, docs/internal/metrics_reference.md	The client records switchyard.router_retry_recovered after a successful request that follows a retry. The metric documentation describes this behavior.
Retry recovery integration test crates/switchyard-server/tests/server.rs	The mock upstream returns a transient 503 on the first matching attempt. The test enables one retry and verifies that the recovery counter increases by one.

Estimated code review effort: 2 (Simple) | ~15 minutes

Merge Risk: ⚪ Minimal · up to 37b30

This localized change populates retry-recovery metrics and updates related tests and documentation; no actionable merge-blocking risk remains beyond normal validation.

Poem

I nibbled the retry trail,
A 503 became a tale.
One more hop, the request flew,
The recovery count grew too.
Squeak—metrics now prevail!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: populating the retry recovery metric.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://NVIDIA-NeMo.github.io/Switchyard/pr-preview/pr-474/
Built to branch gh-pages at 2026-08-18 20:17 UTC. Preview will be ready when the GitHub Pages deployment is complete.