[gaby]

The deployment state metric on the Prometheus endpoint no longer gets stuck at 2 after a deployment recovers from a cooldown. Before, a cooldown added a second, differently labeled line for the same deployment that nothing ever reset, so alerts on it kept firing after the provider came back. Now a cooldown marks the lines the deployment's own requests already report on, and the next successful request clears them

Details

TLDR

Problem this solves:

• A cooldown added a second litellm_deployment_state line for the deployment
• That line was labeled differently from the one requests update
• Nothing reset it, so it stayed at 2 after the provider recovered
• Alerts on state 2 kept firing

How it solves it:

• A cooldown now marks the lines the deployment's requests already wrote
• The next success clears every line the cooldown marked
• Failures logged during a cooldown stay at 2 until the next success
• A cooldown with no request history uses the resolved model and provider labels

Intentional product change: a cooldown now marks the line requests already write, labeled with the resolved model name (for example gpt-5.4-mini) instead of the model group name (for example prod-chat), so one deployment has one line. Users lose the group-name label on state 2, so alerts that matched it should match model_id or the resolved name

User Flow

Before: an operator who alerts on the deployment state metric keeps getting paged after the provider has recovered

1. They send POST https://litellm-domain/v1/chat/completions for prod-chat and get a 200
2. GET https://litellm-domain/metrics/ shows one litellm_deployment_state line for gpt-5.4-mini at 0
3. The provider revokes the key, the POST returns 401, and a second line for prod-chat appears at 2
4. The provider recovers and the POST returns 200, but the second line stays at 2 and the alert never clears

After: the same operator sees the metric follow the deployment, so the alert clears when the provider recovers

1. They send POST https://litellm-domain/v1/chat/completions for prod-chat and get a 200
2. GET https://litellm-domain/metrics/ shows one litellm_deployment_state line for gpt-5.4-mini at 0
3. The provider revokes the key, the POST returns 401, and the one line moves to 2
4. The provider recovers and the POST returns 200, and the line goes back to 0 so the alert clears

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• Every automated review on my PR's latest commit has finished, and I have addressed or replied to each of its findings before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

No provider key was available in this session, so the upstream is a local OpenAI-compatible stand-in that answers 200 or 401 on demand. Everything else is the real proxy hit with curl, started from a checkout of each hash below

Setup (config, upstream, helpers)

config.yaml

model_list:
  - model_name: prod-chat
    litellm_params:
      model: openai/gpt-5.4-mini
      api_base: http://127.0.0.1:9100/v1
      api_key: sk-upstream
    model_info:
      id: dep-1

router_settings:
  cooldown_time: 5

litellm_settings:
  callbacks: ["prometheus"]

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

stub_upstream.py

import json
import sys
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path

MODE = Path(sys.argv[2])
OK = {
    "id": "chatcmpl-1",
    "object": "chat.completion",
    "created": 1700000000,
    "model": "gpt-5.4-mini",
    "choices": [{"index": 0, "message": {"role": "assistant", "content": "hello"}, "finish_reason": "stop"}],
    "usage": {"prompt_tokens": 5, "completion_tokens": 1, "total_tokens": 6},
}
REVOKED = {"error": {"message": "Incorrect API key provided", "type": "invalid_request_error", "code": "invalid_api_key"}}


class Handler(BaseHTTPRequestHandler):
    def do_POST(self):
        self.rfile.read(int(self.headers.get("content-length", 0)))
        status, body = (401, REVOKED) if MODE.read_text().strip() == "401" else (200, OK)
        payload = json.dumps(body).encode()
        self.send_response(status)
        self.send_header("content-type", "application/json")
        self.send_header("content-length", str(len(payload)))
        self.end_headers()
        self.wfile.write(payload)

    def log_message(self, *args):
        pass


ThreadingHTTPServer(("127.0.0.1", int(sys.argv[1])), Handler).serve_forever()

Start the upstream and the proxy, then define two helpers

export LITELLM_MASTER_KEY="sk-$(openssl rand -hex 32)"
echo 200 > mode
python3 stub_upstream.py 9100 mode &
python litellm/proxy/proxy_cli.py --config config.yaml --port 4000

chat() {
  curl -s -o /dev/null -w 'HTTP %{http_code}\n' localhost:4000/v1/chat/completions \
    -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' \
    -d '{"model":"prod-chat","messages":[{"role":"user","content":"hi"}]}'
}
state() {
  curl -s localhost:4000/metrics/ -H "Authorization: Bearer $LITELLM_MASTER_KEY" | grep '^litellm_deployment_state{'
}

Before (f2118fc)

1. A request while the upstream is healthy

   $ chat
   HTTP 200
   

2. Read the metric

   $ state
   litellm_deployment_state{api_base="http://127.0.0.1:9100/v1",api_provider="openai",litellm_model_name="gpt-5.4-mini",model_id="dep-1"} 0.0
   

3. The provider starts rejecting the key, then the same request

   $ echo 401 > mode
   $ chat
   HTTP 401
   

4. Read the metric again, a second line at 2 appeared

   $ state
   litellm_deployment_state{api_base="http://127.0.0.1:9100/v1",api_provider="openai",litellm_model_name="gpt-5.4-mini",model_id="dep-1"} 0.0
   litellm_deployment_state{api_base="http://127.0.0.1:9100/v1",api_provider="openai",litellm_model_name="prod-chat",model_id="dep-1"} 2.0
   

5. A request during the 5 second cooldown

   $ chat
   HTTP 429
   

6. The provider recovers, the cooldown runs out, and the request goes through

   $ echo 200 > mode
   $ sleep 6
   $ chat
   HTTP 200
   

7. Read the metric, the second line is still stuck at 2

   $ state
   litellm_deployment_state{api_base="http://127.0.0.1:9100/v1",api_provider="openai",litellm_model_name="gpt-5.4-mini",model_id="dep-1"} 0.0
   litellm_deployment_state{api_base="http://127.0.0.1:9100/v1",api_provider="openai",litellm_model_name="prod-chat",model_id="dep-1"} 2.0
   

After (bd345ef)

1. A request while the upstream is healthy

   $ chat
   HTTP 200
   

2. Read the metric

   $ state
   litellm_deployment_state{api_base="http://127.0.0.1:9100/v1",api_provider="openai",litellm_model_name="gpt-5.4-mini",model_id="dep-1"} 0.0
   

3. The provider starts rejecting the key, then the same request

   $ echo 401 > mode
   $ chat
   HTTP 401
   

4. Read the metric again, the one line moved to 2 and no second line appeared

   $ state
   litellm_deployment_state{api_base="http://127.0.0.1:9100/v1",api_provider="openai",litellm_model_name="gpt-5.4-mini",model_id="dep-1"} 2.0
   

5. A request during the 5 second cooldown

   $ chat
   HTTP 429
   

6. The provider recovers, the cooldown runs out, and the request goes through

   $ echo 200 > mode
   $ sleep 6
   $ chat
   HTTP 200
   

7. Read the metric, the same line is back at 0

   $ state
   litellm_deployment_state{api_base="http://127.0.0.1:9100/v1",api_provider="openai",litellm_model_name="gpt-5.4-mini",model_id="dep-1"} 0.0
   

Mutation check

Forty-four hand-made changes to the tracker, the cooldown callback and the gauge wiring were each applied on their own, and the tests went red for every one. Forty-three were caught by the first set of tests and the last (lifting the series cap from a cooling deployment) needed one more test, which is in this PR. For example, making the next success skip the extra lines a cooldown marked, which brings the original bug back, fails test_a_success_under_a_new_label_clears_every_series_a_full_cooldown_marked, and it passes again with the change reverted

Type

🐛 Bug Fix
✅ Test

Caveats (if any)

Severe

• Cooldown lines now carry the resolved model name, not the group name
  • Alerts matching the group name on state 2 stop matching
  • Match model_id or the resolved name instead

Medium

• A deployment never picked again after a cooldown stays at 2
  • Only its next success or a restart clears the line
• Failures after a cooldown read 2 until the next success
  • That includes failures logged after the cooldown has ended
• Marks are tracked per worker process
  • Multiprocess mode keeps one line per worker, so another worker's success leaves a marked line at 2
  • Multi-worker mode was reasoned from the code, not run live

Low

• Proof uses a scripted local upstream, not a paid provider call
• A cooldown marks at most eight series per deployment
  • Older series of a wildcard deployment keep their last value
• A cooldown before any request can leave a duplicate line at 0
  • Its api_base comes from config and may differ from the request one
• litellm_deployment_cooled_down labels are unchanged on purpose
• A late success during a cooldown can briefly show 0
• Provider 401s still skip other failure metrics, a separate issue

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
Changes Prometheus deployment health semantics and label values (group vs resolved model on cooldown), which can break existing alerts; logic is observability-only but non-trivial and per-process in multiprocess mode.

Overview
Fixes litellm_deployment_state staying at 2 after a deployment recovers from router cooldown, which left a second metric line (often labeled with the model group name) that success never reset.

DeploymentStateSeriesTracker is added and wired into PrometheusLogger: healthy/partial/complete setters now route through it instead of writing the gauge once per call. On cooldown it marks the label series the request path already wrote (and related history), keeps failures during cooldown at 2 until the next success, then clears those series to 0. LRU caps limit tracked deployments and series per deployment; evicted cooling deployments are reset to healthy.

The router cooldown callback resolves model/provider via resolved_model_and_provider and uses the resolved model name for deployment state (not the group name); litellm_deployment_cooled_down still uses the group name. Unit/integration tests and Prometheus registry fixtures cover the state transitions and end-to-end router flow.

^{Reviewed by Cursor Bugbot for commit 92a0aad. Bugbot is set up for automated code reviews on this repo. Configure here.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-26T13:01:59.770566Z We couldn’t complete this review. Comment @codex review to try again.	bd345ef	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[codecov]

Codecov Report

❌ Patch coverage is 95.69892% with 4 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...ions/prometheus_helpers/deployment_state_series.py	94.93%	4 Missing ⚠️

📢 Thoughts on this report? Let us know!

[codspeed]

Merging this PR will not alter performance

1 warning

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

✅ 35 untouched benchmarks

---

_{Comparing gaby:ccr-d5d7cfc6-nfu1ya (1a1254e) with main (cfc651d)}

[gaby]

The enterprise unit tests timeout after 20mins, it was at 99% already.

[mubashir1osmani]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mubashir1osmani]

@greptile-apps

[mubashir1osmani]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 92a0aad. Configure here.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[gaby]

@mubashir1osmani Can you trigger the ci again