[theminjunchoi]

Summary

Sweep finds its top rate with a throughput step fixed at 512 concurrent requests. That's off in both directions: a small server queues the requests and reports inflated latency, and a large one tops out below its capacity. In my runs that meant a TTFT p95 of ~27s on a small server, and ~128 req/s against a real ~176 on a large one.

This adds a sweep_next profile, following the discussion in #1248 and #564. It runs like sweep, but instead of the throughput step it starts a ramp at the synchronous rate and doubles it until the server falls behind, and the highest completion rate it reached becomes the top of the range. On the llm-d simulator and the mock server it landed within ~7% of capacity. As suggested in the review, sweep itself is untouched, so the two can be compared side by side before deciding whether to replace it.

Details

• AsyncRampStrategy: the rate doubles every doubling_interval, with no upper bound.
• RampSaturationConstraint: stops sending once completions fall more than 25% short of what was sent one latency earlier, keeps measuring for three latencies, and reports the peak completion rate.
• sweep_next sizes the ramp from the synchronous step (doubling every four latencies, at least 2s) and records the peak as a sweep_ramp conclusion. It has no concurrency cap unless you set max_concurrency.
• If something like --max-seconds ends the ramp before the server saturates, the console warns that the peak is only a lower bound.
• Docs in a new Sweep Next section of benchmark.md.

I didn't use over-saturation detection as I'd planned in #564. On a server that slows down gradually, TTFT doubles well before saturation, so it stopped the ramp at ~60% of capacity.

The tradeoff is time. The ramp needs about log2(capacity × latency) doublings, so a server holding ~1000 requests at ~2s each took ~90s. Happy to expose the constants or change the defaults if you'd like.

This is stacked on #1262, since the e2e tests need the mock server to actually saturate. I'll close #1248 in favor of this one.

Test Plan

• lint-check, type-check, test-unit (3,692 passed), test-integration (34) and test-e2e (16) pass on Python 3.10. After the split, test-unit (3,799 passed), type-check and the sweep_next e2e tests pass again on Python 3.12.
• Unit tests replay a ramp against a server with fixed capacity: nothing triggers while it keeps up, saturation is caught past capacity, the peak lands within 10%, and requests cancelled by another constraint don't count as saturation. An e2e run caught that last one.
• E2E against a mock server with 4 slots: a sweep_next run saturates it and the constant step uses its peak, and a 2s step limit is reported as not saturated.

Related Issues

• Resolves Improve throughput upper limit detection #564

---

• "I certify that all code in this PR is my own, except as noted below."

Use of AI

• Includes code generated or substantially modified by an AI agent
• Includes tests generated or substantially modified by an AI agent

NOTE: the Generated-by or Assisted-by trailers should be used in git commit messages when code or tests were generated or substantially modified by an AI agent, as described in the project's DEVELOPING.md file.

---

git log

commit 04d8836
Author: MinJun Choi mj04300017@gmail.com
Date: Mon Oct 12 15:57:02 2026 +0900

Add a ramp strategy and a saturation constraint for it

The ramp strategy starts at a given rate and doubles it every interval, with
no upper bound, so finding a server's peak doesn't need a guess at its
capacity up front.

The saturation constraint decides when to stop it. While the server keeps up,
completions over a window match what was sent one latency earlier. When they
fall more than 25% short, it stops sending, lets the requests in flight finish
for three latencies, and records the highest completion rate seen over a
window. It stops judging once another constraint ends queuing, since the
requests that one cancels never complete.

Signed-off-by: MinJun Choi <mj04300017@gmail.com>

commit 853ef63
Author: MinJun Choi mj04300017@gmail.com
Date: Mon Oct 12 15:57:03 2026 +0900

Let sweep find the top of its range with a ramp

Add a peak_detection option to sweep. The default keeps the fixed
concurrency throughput step. With "ramp", the sweep runs a ramp sized from
the synchronous step instead: it starts at the synchronous rate, doubles every
four synchronous latencies (at least two seconds), and stops through the
saturation constraint. The peak becomes the top of the range and is recorded
as a profile conclusion.

If something else, like --max-seconds, ends the ramp before the server
saturates, the peak is only a lower bound, so the console warns about it. The
default concurrency cap of 512 only applies to a ramp when set explicitly.

Signed-off-by: MinJun Choi <mj04300017@gmail.com>

commit b64ca68
Author: MinJun Choi mj04300017@gmail.com
Date: Mon Oct 12 15:57:03 2026 +0900

Add tests for the ramp strategy and sweep ramp mode

Check the ramp's send schedule and doubling, and replay a ramp against a
server with fixed capacity: nothing triggers while it keeps up, saturation is
detected past capacity, the peak lands within 10% of it, and requests cancelled
by another constraint don't count as saturation. Cover sweep sizing, the peak
fallbacks, the concurrency cap and the console summary.

Assisted-by: Claude Code claude-opus-5-5
Signed-off-by: MinJun Choi <mj04300017@gmail.com>

commit 28e2941
Author: MinJun Choi mj04300017@gmail.com
Date: Mon Oct 12 15:57:03 2026 +0900

Add e2e tests for the sweep ramp

Run a ramp sweep against a mock server with a fixed number of slots: the ramp
saturates it, its peak lands near capacity and sizes the constant step, and a
short --max-seconds is reported as not saturated.

Assisted-by: Claude Code claude-opus-5-5
Signed-off-by: MinJun Choi <mj04300017@gmail.com>

commit 101f3dc
Author: MinJun Choi mj04300017@gmail.com
Date: Tue Oct 20 09:55:22 2026 +0900

Move the ramp sweep into its own sweep_next profile

Keep sweep exactly as it was and put the ramp version in a new sweep_next
profile instead of behind a peak_detection option. sweep_next runs the
synchronous step, then the ramp, then the interpolated async steps, and has no
concurrency cap unless one is set.

Signed-off-by: MinJun Choi <mj04300017@gmail.com>

commit 115a3f1
Author: MinJun Choi mj04300017@gmail.com
Date: Tue Oct 20 09:55:22 2026 +0900

Point the ramp tests at sweep_next

Restore the original sweep tests, move the ramp cases to a sweep_next test
file, and run the e2e ramp sweep with kind=sweep_next.

Assisted-by: Claude Code claude-opus-5-5
Signed-off-by: MinJun Choi <mj04300017@gmail.com>

commit 1e64de6
Author: MinJun Choi mj04300017@gmail.com
Date: Tue Oct 20 09:55:22 2026 +0900

Document sweep_next separately from sweep

Signed-off-by: MinJun Choi <mj04300017@gmail.com>

---

Assisted-by: Claude Code claude-opus-5-5
Signed-off-by: MinJun Choi mj04300017@gmail.com

[theminjunchoi]

Hi @sjmonson, now that v0.8.1 is out, would you have time to look at this? It's my attempt at the idea from your comment on #1248: a ramp in place of the fixed throughput step, cut off automatically once the server falls behind. It's opt-in for now, and I'd be glad to reshape it however you'd like.

[sjmonson]

Just a quick update. Still working through our backlog of PRs so have not had too much time to evaluate this. Just looking through the code it seems good; though I was kinda thinking we completely remove the throughput mode assuming this works out. Maybe it is better to leave it for now until we are sure this is an improvement across all use-cases. Or alternatively maybe copy the old sweep and rename this to sweep_next to make sure we don't change the old functionality by accident. Other than that my two open questions which will have to be answered experimentally are:

1. Should we completely ignore user defined constraints for sync + ramp just to prevent users from setting a ridiculously low constraint and causing the detection to fail. Most of the other profiles I don't think we should limit the user, but sweep is supposed to be a bit more hand-holding.
2. How well does this detection scale across various hardware / model combos. E.g. small model + big H200 system vs medium model on CPU.

When I find some time I'll try to answer those questions. Though if you do split this off from the old sweep I would be less concerned about getting these answered before merge.

[theminjunchoi]

@sjmonson Thanks! I split it off as sweep_next, so sweep is back to what's on main. User constraints still apply during the ramp for now, with a warning if one cuts it short. Happy to change that once there's data.