[earakely-scale]

What

Two changes to the TRL training path.

1. No call holds a session past its client's deadline

AgentEnvEnvironment already ended a tool call that missed the timeout_s deadline in its action's metadata. The other two calls on a session had no deadline on the server:

• reset_async waited on the deploy and the opening of the first turn.
• end_turn waited on the next turn or the verifiers' score.

The client gave up on both after reset_timeout. A client that had gone away therefore left the server's handler waiting: the session stayed open, its sandboxes stayed up, and the server's graceful shutdown waited on the handler until it was killed.

• Server: reset_async takes timeout_s, and end_turn honors timeout_s metadata as tool calls do. Past either deadline, the server ends the episode with an error and the session goes on. Without a deadline, nothing changes.
• Answering at once: a call past its deadline, a tool call's included, answers immediately, and its episode tears down after. The teardown stays tracked, so the session's next reset, a close and the server's shutdown wait for it. Before, the reply waited for the teardown, which can outlast the client's grace.
• Helper: agent_env.training.trl now sends reset_timeout as the deadline of each reset and each end_turn, the left-open turns it ends included. Each message waits GRACE_SECONDS longer than its deadline, so the server answers first.

2. A CI job that runs TRL

TRL marks environment_factory experimental, and until now CI never ran TRL. The new tst/trl tier, run by .github/workflows/trl.yml, has one test:

• GRPOTrainer trains a step on CPU through the helper, against the OpenEnv server build_app makes over the Episode API's fake steps.
• As in TRL's own environment tests, a tiny model's generate is stood in for: in each group, one rollout adds a crate and ends its turn, and the other answers without a tool.
• The test checks that TRL resets with the row, calls the tools, and reads get_reward. Every rollout is scored (1, 0, 1, 0), parameters change, and every episode is torn down.

When it runs:

• against TRL 1.16 and the latest. TRL 1.15 can't train on CPU, since its log-prob path is a Triton GPU kernel, so 1.16 is the oldest the job can run;
• on pull requests that touch agent_env/training, agent_env/openenv or the tier;
• weekly, and on demand.

It allows no skips, and it isn't a required check.

AsyncGRPO needs a vLLM server, so the tier doesn't cover it. Its pickling contract is covered by the unit test that plays a rollout in a spawned process.

Supporting changes:

• The trl_installed capability: trl and torch aren't in the dev extra, so make test skips the tier with a declared reason.
• Make target: make trl-test.
• Docs: AGENTS.md and CONTRIBUTING.md list the tier and the workflow.

Tests

• Reset and end_turn past their deadlines: a new test in tst/unit/openenv/environment_test.py holds a reset before its first turn, and an end_turn before the score, past 0.5 s deadlines. Teardown is held while they're called. Both answer with an error observation before their episodes are torn down, and both episodes are torn down once teardown is released. Without the server change, or with replies that wait for teardown, it hangs.
• Client deadlines: test_a_reset_and_an_end_turn_carry_the_resets_deadline_so_end_turn_waits_for_the_score_past_a_tools (the slow-verifier test, extended) checks that the helper sends reset_timeout with both resets and the end_turn.
• The new tier: tst/trl/grpo_test.py runs in the new workflow on this pull request.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; the earlier timeout-reply issue is fixed.

Summary

The PR adds server deadlines for reset and end_turn, sends those deadlines from the TRL helper, and adds a CPU TRL training check.

• OpenEnv ends reset and end-turn calls when their deadlines pass.
• CI trains one CPU step through the TRL helper against two TRL versions.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Call misses deadline] --> B[Track episode cleanup]
  B --> C[Return error without waiting]
  B --> D[Cleanup continues]
  D --> E[Next reset, close, or shutdown waits]

Loading

_{Reviews (3) · Last reviewed commit: "fix(openenv): a call past its deadline a..." · Reviewed by Greptile}