Skip to content

Parse session responses off the event loop #79

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nightlessbaron wants to merge 18 commits into
base: prod
Choose a base branch
from
Open

Parse session responses off the event loop #79

Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
0bd6425
fixes
odp Jul 2, 2026
b7d3698
add jitter
odp Jul 2, 2026
2943912
add _compact_session_record, save on bytes
odp Jul 2, 2026
d71934c
concurrency
odp Jul 3, 2026
e3112f1
add session server actor and distribute
odp Jul 3, 2026
17a4d6b
to_thread
odp Jul 3, 2026
a86619e
add a threadpool
odp Jul 3, 2026
008cf75
more to_threads
odp Jul 3, 2026
417e420
skip decoding Sample.response if possible
odp Jul 3, 2026
07b5c62
fix session server port conflict
odp Jul 8, 2026
dba65d3
lower stats logging
odp Jul 8, 2026
cf6de42
use gzip compression for /sessions/sessonid endpoint
odp Jul 9, 2026
d890537
Revert "use gzip compression for /sessions/sessonid endpoint"
odp Jul 9, 2026
8b5dfb6
session-server-side merge - first shot
odp Jul 9, 2026
92f32a4
return sample instead of samples
odp Jul 9, 2026
2096dd0
introduce _keep_records_until_first_non_completed in the session serv…
odp Jul 15, 2026
92f034a
Parse session responses off the event loop
nightlessbaron Jul 15, 2026
6bccf68
Merge branch 'prod' into agent/parse-session-responses-off-thread
nightlessbaron Aug 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions miles/rollout/generate_utils/openai_endpoint_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,11 @@
import logging
import random
from argparse import Namespace
from copy import copy, deepcopy
from typing import Any

import httpx
import orjson
from copy import copy, deepcopy
from typing import Any

from miles.rollout.generate_utils.generate_endpoint_utils import get_rollout_topk_from_response
from miles.rollout.session.session_types import (
Expand Down Expand Up @@ -182,8 +183,8 @@ async def _request(
return response.text

try:
return response.json()
except json.JSONDecodeError as exc:
return await asyncio.to_thread(orjson.loads, response.content)
except orjson.JSONDecodeError as exc:
logger.info(
"[session-client] json_decode_error phase=%s method=%s url=%s status=%d bytes=%d attempt=%d/%d",
phase,
Expand Down
Loading