[sergiopaniego]

The Wordle tutorial trained Qwen3-1.7B with GRPO from the base model and the environment's raw reward. That recipe barely learns. The base model almost never wins, and the raw reward pays a win the same at any speed, so a policy with thinking on learns to think less and use all six guesses. This rewrites the tutorial around a recipe that works, and adds its scripts.

• examples/wordle/: the whole pipeline, each script self-contained and pinned (trl==1.16.0, openenv==0.10.0):
  • generate_sft_data.py: a solver plays 1,000 games against the environment (in-process) and writes them as tool-calling conversations, in the format TRL renders during GRPO.
  • sft.py: SFT warmup on those games.
  • grpo.py: GRPO from the SFT model, with a reward split in three terms (a win that pays more the faster it comes, partial credit from the best guess, a penalty for invalid moves).
  • evaluate.py and make_gif.py: play models on fixed words and render a side-by-side GIF.
  • wordle.ipynb: the same pipeline for Colab. On a T4 it launches SFT and GRPO on Hugging Face Jobs and compares the published models.
• docs/source/tutorials/wordle-grpo.md: rewritten as "Train a Wordle Agent", with the results first, the hardware, time and cost of every step, and why the SFT warmup is needed. The tutorials index and the sidebar point to it and to the new notebook.
• envs/textarena_env/README.md: links the tutorial (sync_env_docs.py --fix refreshes its docs page).

Results on the same 30 words, played greedily: Qwen3-1.7B wins 0, the SFT model 10 and the SFT + GRPO model 20. The models and the dataset are in the openenv org (Qwen3-1.7B-Wordle-SFT, Qwen3-1.7B-Wordle-SFT-GRPO, wordle-tool-calls-sft), and the Wordle Arena Space replays their games. The full run costs about $6 on one A100.

The openenv/wordle Space was redeployed from v0.10.0, so it reports how a game ended (#1455), and allows 64 concurrent sessions.

This PR was prepared with AI assistance.

---

Note

Low Risk
Documentation and example scripts only; no production runtime or auth changes. Training scripts assume users duplicate the Wordle Space with MAX_CONCURRENT_ENVS=64.

Overview
Replaces the old GRPO-only Wordle tutorial (TRL notebook–centric) with an end-to-end SFT warmup → GRPO recipe for Qwen3-1.7B on the OpenEnv TextArena Wordle Space, plus runnable assets in examples/wordle/.

New pipeline: generate_sft_data.py (solver → tool-calling SFT dataset), sft.py, grpo.py (split rewards: fast wins, partial greens, invalid penalties; thinking on; up to 12 tool rounds), evaluate.py / make_gif.py, secrets30.txt, wordle.ipynb, and a small README. Docs rename the tutorial to Train a Wordle Agent, lead with benchmark results and per-step cost/time, and point the tutorials index, sidebar, and TextArena docs at the new flow and Colab notebook (OpenEnv repo, not TRL’s old notebook).

Tests under tests/scripts/ mock training/eval/GIF logic (guess budgets, fp32 SFT weights, notebook Hub URL validation) without GPUs or Hub calls.

^{Reviewed by Cursor Bugbot for commit 0a24871. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Alignment Review Report

Automated Checks

• Lint: PASS — usort check src/ tests/ and ruff check src/ tests/ are clean; ruff format --check src/ tests/ envs/ flags 47 pre-existing files, none touched by this diff (this PR only changes docs/, envs/textarena_env/README.md, and examples/wordle/). Running ruff check/ruff format --check directly against the new examples/wordle/*.py files (outside the hook's scope) is also clean.
• Debug code: CLEAN — check-debug.sh only scans src/, which this PR doesn't touch. No print/breakpoint/pdb debug code in the new examples/wordle/*.py files beyond intentional CLI/progress output, and no TODO/FIXME added.

Open RFCs Context

• RFC 005 (Agentic Harness Integration) and RFC 006 (Agentic RL through Harness Interception) are both In Review but cover installed-CLI-agent harnesses (OpenClaw, Claude Code, opencode, Pi) and token-capture proxies — not relevant here, since this tutorial uses the standard TextArena HTTP/MCP client wired into TRL's environment_factory, the same pattern already used by rl-training-2048 and end_to_end_walkthrough.
• RFC 002 (Env Spec, In Review) is the RFC behind the "rewards inside environment" principle — see the alignment flag below.
• No other Draft/In Review RFC (000, 001, 003, 004, 008, 010, 011, 012) touches docs or training-example conventions in a way this PR conflicts with.

Tier 1: Fixes Required

• examples/wordle/generate_sft_data.py:4, examples/wordle/grpo.py:6, examples/wordle/evaluate.py:6, examples/wordle/wordle.ipynb:21, docs/source/tutorials/wordle-grpo.md:40 — the pip-installable dependency URL still reads git+https://github.com/huggingface/OpenEnv.git@v0.10.0#subdirectory=envs/textarena_env, while this same PR's d5f95805 commit ("point its links at openenv/OpenEnv") updated the surrounding prose links (Colab badges, GitHub source tree, raw.githubusercontent.com URLs in the notebook) to openenv/OpenEnv. Worth a pass to make these consistent with the new canonical org, matching this PR's own stated goal — GitHub's repo-rename redirect means installs won't break, but it reintroduces the exact link the commit set out to replace.

Tier 2: Alignment Discussion

Principle Conflicts

ALIGNMENT FLAG: Reward shaping defined in the training script, not the environment

• Principle/RFC at stake: "Rewards inside environment" (PRINCIPLES.md, backed by RFC 002) — "Domain knowledge encapsulated in env, not external"; INVARIANTS.md adds that external reward augmentation should go through the server-side Transform pipeline.
• The concern: examples/wordle/grpo.py defines reward_win, reward_partial, and reward_invalid as plain Python functions passed to GRPOTrainer(reward_funcs=...), reading env.num_guesses, env.won, env.best_greens, env.invalid_moves off the client-side WordleEnv wrapper rather than through a server-side Transform. This is consistent with three other existing examples (browsergym_trl_hf_sandbox.py, pelican_svg_grpo.py, sophistry_bench_sprint_grpo.py), so it's a pre-existing pattern this PR follows rather than originates — but it's still worth a human check on whether environment_factory + client-side reward_funcs is meant to be the sanctioned exception to this invariant, or whether the invariant/examples need reconciling.
• Suggested reviewer: @Darktex (author of the "Rewards inside environment" row in PRINCIPLES.md and RFC 002 co-author)

RFC Conflicts

None identified.

Summary

• 1 mechanical issue to fix (stale huggingface/OpenEnv install URLs left behind by this PR's own link-update commit, 5 occurrences)
• 1 alignment point for human review (client-side reward shaping vs. "rewards inside environment")
• 0 RFC conflicts to discuss

  

_{Sent by Cursor Automation: Pre-review}

[cursor]

ALIGNMENT FLAG: reward_win/reward_partial/reward_invalid live in this training script and read fields off the client-side WordleEnv wrapper, rather than the environment emitting a single domain-aware reward. That's a tension with "Rewards inside environment" in PRINCIPLES.md (RFC 002) and the Transform-pipeline guidance in INVARIANTS.md. It matches existing precedent (browsergym_trl_hf_sandbox.py, pelican_svg_grpo.py, sophistry_bench_sprint_grpo.py), so likely fine as a sanctioned environment_factory pattern — flagging for @Darktex to confirm rather than as a blocker.

[sergiopaniego]

The environment keeps its own reward (1.0 for a win, a letter score for a loss), and the tutorial documents it. These three functions are a training choice for this recipe. With the raw reward, a policy with thinking on learned to think less and use all six guesses, so the script pays more for faster wins and adds partial credit. Moving this into a server-side Transform would change what the shared openenv/wordle Space returns for everyone using it, so it stays in the script, like the other environment_factory examples. Happy to change it if @Darktex prefers a Transform.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Stale Bugbot comment from a previous run.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit f6c8fd5. Configure here.}