RiT: Rubrics-in-Thinking Reinforcement Learning
for Improved Reasoning in Large Language Models

Xiaobin Tian · Shuai Yuan · Muyun Ding · Haonan Chen · Xiaoxi Jiang

Qwen Large Model Application Team, Alibaba

Overview

Rubrics-in-Thinking Reinforcement Learning (RiT) introduces thinking-rubric supervision into reinforcement learning for large reasoning models. Rather than rewarding only the final answer, RiT evaluates whether the intermediate reasoning is coherent, complete, relevant, and aligned with the task. The thinking reward is fused with an outcome reward and bounded by a hard gate, preventing plausible-looking reasoning from compensating for a poor final response.

This repository contains the official experimental code for RiT: Rubrics-in-Thinking Reinforcement Learning for Improved Reasoning in Large Language Models (Findings of ACL 2026).

RiT integrates a thinking-rubric reward with the response reward through gated fusion.

Contents

• Overview
• Contents
• Quick start
• Released data
• Experiment configuration
• Multi-node training
• Development and testing
• Acknowledgements
• License
• Citation

Quick start

1. Install the environment

Requirements:

• Linux with CUDA-capable GPUs
• Python 3.10 or newer
• A local policy-model directory
• Git LFS for the released JSONL datasets

This repository includes the veRL snapshot used by the training entry point under verl/. Start from a CUDA/PyTorch environment compatible with veRL and vLLM 0.8.4, then install the recorded dependencies:

git lfs install
git lfs pull
python3 -m pip install -r requirements/requirements-rlvr.txt
python3 -m pip install --no-deps -e .

The launcher discovers the bundled snapshot automatically. To use another compatible veRL checkout, set VERL_DIR to the directory whose top level contains the verl Python package.

If the LLM critic runs in a separate environment, install its dependencies there:

python3 -m pip install -r requirements/requirements-verifier.txt
python3 -m pip install --no-deps -e .

2. Prepare the data

The launcher accepts JSONL or a JSON array and converts it to veRL parquet format. A reasoning example is:

{
  "prompt": [{"role": "user", "content": "Question or instruction"}],
  "data_source": "strategyqa",
  "golden_answer": "Yes",
  "thinking_rubrics_prompt": "Verifier instructions and rubric JSON schema"
}

prompt may instead be a string input or question, and golden_answer may be named answer.

Experiment	Required supervision
RiT, reasoning	thinking_rubrics_prompt and an answer
RiT, open-ended	thinking_resp_rubrics_prompt
ORM, reasoning	golden_answer or answer
ORM, open-ended	rubrics_prompt

3. Start the LLM critic

The LLM critic is required for RiT and for the open-ended ORM baseline:

Method	Reasoning	Open-ended
RiT	Required	Required
ORM	Not required	Required

The paper uses Qwen3-235B-A22B-Instruct for rubric scoring:

python3 -m rit.verifier_server \
  --model /path/to/Qwen3-235B-A22B-Instruct \
  --tensor-parallel-size 8

The server listens on 127.0.0.1:5000 by default with a 32,768-token context, 4,096 maximum output tokens, and temperature 0.0. Use --host 0.0.0.0 only on a trusted network when remote workers need access.

4. Launch training

The minimal command runs the default Qwen + RiT + reasoning configuration:

bash scripts/run_RiT.sh \
  --train-jsonl Datasets/training/Reasoning/StrategyQA.boxed.rubrics.jsonl \
  --test-jsonl Datasets/testing/Reasoning/StrategyQA.test.boxed.jsonl \
  --model-path /path/to/policy-model

--test-jsonl is optional and defaults to the training file. Inspect the resolved configuration without starting training:

bash scripts/run_RiT.sh \
  --train-jsonl /path/to/train.jsonl \
  --model-path /path/to/policy-model \
  --dry-run

Run bash scripts/run_RiT.sh --help and python3 -m rit.verifier_server --help for the complete command-line reference.

Released data

Datasets/ contains the pre-generated rubric prompts and benchmark splits used by the launcher. JSONL files are stored with Git LFS because the largest file exceeds GitHub's 100 MB file limit.

Benchmark	Training records	Test records	Training files
LongBench-Write	60	60	combined response/thinking rubrics
WritingBench	500	500	combined response/thinking rubrics
StrategyQA	1,279	229	answer-only and thinking-rubric variants
LogiQA 2.0 NLI	30,908	3,942	answer-only and thinking-rubric variants
TruthfulQA MC1	653	164	answer-only and thinking-rubric variants

Experiment configuration

Setting	Reasoning	Open-ended
RL algorithm	GRPO	GRPO
Samples per rollout	8	8
Actor learning rate	2e-6	2e-6
Global batch size	128	32
Default training steps	500	200
Fusion weight $\alpha$	1.0	0.5
Gate	hard minimum	hard minimum

The DeepSeek-R1-Distill-Llama-8B LongBench-Write experiment is the documented exception and uses --fusion-alpha 1.

Launcher defaults not determined by task are:

Option	Default
--model-family	Qwen
--task-type	reasoning
--method	RiT
--train-pp / --train-tp / --gen-tp	2 / 2 / 8
--gate-type	min
--rubrics-version	v1
--proxy-url	127.0.0.1:5000

Reward functions are selected automatically:

Method	Task	Score function
RiT	reasoning	compute_score_with_format_think_rubrics
RiT	open-ended	compute_score_with_thinking_resp_rubrics
ORM	reasoning	compute_score_with_format_orm
ORM	open-ended	compute_score_with_resp_rubrics

Multi-node training

Multi-node runs use NNODES, RANK, MASTER_ADDR, NPROC_PER_NODE, and a LOCAL_OUTPUT_DIR shared by every node. Export the same settings and run the same launcher command on each node, changing only RANK.

Development and testing

The lightweight regression suite does not require GPUs or veRL:

python3 -m unittest discover -s tests -v
for script in scripts/*.sh; do bash -n "$script"; done

Optional formatting and shell checks:

python3 -m pip install -r requirements/requirements-precommit.txt
pre-commit run --all-files

.pre-commit-config.yaml is not needed at runtime. It is intentionally kept as a contributor safeguard: Black formats project Python files and ShellCheck checks shell scripts, while the large datasets and bundled veRL code are excluded. Removing it would only remove these automated local checks.

Repository layout:

src/rit/                     Python package, training entry point, and rewards
Datasets/                    Released benchmark splits and rubric prompts
examples/data_preprocess/    JSON/JSONL to veRL parquet conversion
scripts/                     Unified launcher and distributed setup
requirements/                Training, verifier, and development dependencies
tests/                       CPU-only regression tests
assets/                      Framework figure
pyproject.toml               Python package metadata and src-layout configuration
verl/                        Bundled upstream veRL snapshot (0.5.0.dev)

The CPU suite verifies parsing, reward fusion, data normalization, chat templates, and CLI defaults. Full training additionally depends on the documented GPU topology, model checkpoints, CUDA stack, and verifier service; use --dry-run to validate launcher choices before allocating a cluster.

Acknowledgements

RiT is built on veRL and uses vLLM and Transformers. We thank the developers of these open-source projects and the authors of LongBench-Write, WritingBench, StrategyQA, LogiQA, and TruthfulQA for making their work available to the research community.

License

This project is released under the Apache License 2.0. See LICENSE. Third-party models, datasets, and dependencies remain subject to their respective licenses and terms of use.

Citation

If you use RiT, please cite the paper:

@inproceedings{tian-etal-2026-rit,
  title     = {{R}i{T}: Rubrics-in-Thinking Reinforcement Learning for Improved Reasoning in Large Language Models},
  author    = {Tian, Xiaobin and Yuan, Shuai and Ding, Muyun and Chen, Haonan and Jiang, Xiaoxi},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year      = {2026},
  pages     = {3944--3957},
  doi       = {10.18653/v1/2026.findings-acl.192},
  url       = {https://aclanthology.org/2026.findings-acl.192/}
}

GitHub citation metadata is also provided in CITATION.cff.