Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,6 @@ jobs:
run: python -m pytest
- name: Example contract smoke
run: bash examples/native/cpu_contract_smoke.sh
- name: New user Hello World
run: harborrl hello --out-dir /tmp/harborrl-hello
- name: Training dry run
run: python -m harborrl.cli train --config examples/native/train_qwen_native.yaml --dry-run
- name: Public tree audit
Expand Down Expand Up @@ -79,8 +77,6 @@ jobs:
python -m venv /tmp/harborrl-wheel
/tmp/harborrl-wheel/bin/pip install dist/*.whl
/tmp/harborrl-wheel/bin/harborrl --help
cd /tmp
/tmp/harborrl-wheel/bin/harborrl hello --out-dir /tmp/harborrl-wheel-hello
- uses: actions/upload-artifact@v4
with:
name: harborrl-dist
Expand Down
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,8 @@ docs/agents/*
!docs/agents/environment-check.md
!docs/configuration-wizard.md
!docs/deployment.md
!docs/math-distill.md
!docs/sandbox-rollout.md
!docs/rollout-runtime-lifecycle.md
!docs/train-inference-consistency.md
!docs/web-console.md
practices/**/VALIDATION.md
Expand Down
40 changes: 19 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,11 @@

## News

- **2026-10-26** Added an [offline configuration wizard](docs/configuration-wizard.md) and a [local Web console](docs/web-console.md) for configuration, validation, dry-runs, training launch, and logs.
- **2026-10-26** Added [controller/GPU deployment configuration](docs/deployment.md), optional [OpenSandbox execution](docs/sandbox-rollout.md), and opt-in [train/inference consistency checks](docs/train-inference-consistency.md).
- **10/2026** 📣📣📣 HarborRL released
- **2026-11-12** Added [gateway horizontal sharding](docs/deployment.md) with static routing, elastic auto-growth (remote-first), and automatic retirement.
- **2026-11-12** Added [Hub deployment mode](docs/deployment.md) for clusters with one-way network paths: nodes pull tasks over HTTP, no cross-machine SSH required.
- **2026-11-12** Added [online math training](docs/math-distill.md) (GRPO, DAPO, process rewards, OPD, PRM distillation) on the shared Slime/Megatron + SGLang pipeline.
- **2026-11-12** Added [configuration wizard](docs/configuration-wizard.md), [web console](docs/web-console.md), and [sandbox rollout](docs/sandbox-rollout.md) with audited cleanup.
- **2026-11-12** 10/2026 📣 HarborRL released

## About

Expand All @@ -42,18 +44,23 @@ HarborRL supports flexible combinations of **Models × Algorithms × Tasks × Ha

## AI-assisted Configure

For offline option browsing, configuration validation, and interactive or
answer-file setup, see the [configuration CLI guide](docs/configuration-wizard.md).

For mathematical data validation and scoring utilities, see the [math adapter guide](docs/math-distill.md).

For the offline options catalog, interactive configuration, and answer-file setup,
see the [configuration CLI guide](docs/configuration-wizard.md).
Start with `harborrl hello` for a CPU walkthrough requiring no model, GPU or task
JSON. It executes scripted attempts and a real verifier, then exports a GRPO batch
with fixture tokens. Continue with `harborrl options` or `harborrl init --help`.
The wizard detects one shared Local environment, calculates model fingerprints,
and supports incomplete drafts (`:save` / `--resume`).
Start with `harborrl options` or `harborrl init --help`.

Run `harborrl web` to open the local browser console for configuration, validation,
drafts, filled Local/Sandbox examples, Hello World, dry-runs, training launch, and logs. See the [web console guide](docs/web-console.md)
dry-runs, training launch, and logs. See the [web console guide](docs/web-console.md)
for remote-controller access and lifecycle details.

For the optional math PRM/OPD reference trainer, see
[Math training pipeline](docs/math-distill.md). It includes explicit synthetic
fixtures for testing real optimizer updates and checkpoint restore.

If you want an AI agent to help deploy this repository and verify training,
ask it to first read and strictly follow
[docs/agents/environment-check.md](docs/agents/environment-check.md).
Expand Down Expand Up @@ -122,15 +129,10 @@ Expected result: uv executes the installed entry point and tests without activat
For native training, install the pinned backend stack in a writable location:

```bash
bash scripts/bootstrap_backends.sh "<Path to HarborRL backends>"
. "<Path to HarborRL backends>/harborrl-backend.env"
bash scripts/bootstrap_backends.sh <Path to HarborRL backends>
. <Path to HarborRL backends>/harborrl-backend.env
```

The generated file exports `SLIME_DIR`, `MEGATRON_DIR`, and `SGLANG_IMAGE` to child
processes. For files generated by older bootstrap scripts, run `set -a` before
sourcing and `set +a` afterward. See [rollout lifecycle and recovery](docs/rollout-runtime-lifecycle.md)
for cleanup markers and CPU-only regression checks.

### Train

After installation, choose one of the two configuration styles:
Expand Down Expand Up @@ -169,11 +171,7 @@ task/

Here, `task.toml` declares resources, the agent, and the verifier; `instruction.md` is the agent-facing instruction; `environment/Dockerfile` builds the isolated environment; and `tests/test.sh` verifies the result and writes `reward.json` or `reward.txt`. For training, tasks are locked into a catalog with their task ID, upstream revision, local path, content digest, and reward parsing rules; see [examples/native/catalog.json](examples/native/catalog.json).

The bundled runnable example is `hello_world`. It allows internet access so Harbor can install the pinned Claude Code CLI, reserves 4 GiB each for memory and storage, and writes a 0/1 reward to `/logs/verifier/reward.json`. Its profile allows four turns so the agent can observe a tool result and finish. Offline deployments need the exact CLI version preinstalled and a reachable policy gateway. After changing a task, regenerate its catalog digest.

The example pins the runtime-validated Claude Code `2.1.141`. The adapter disables prompt caching and cache warmup, compaction, and nonessential traffic, and always disallows `Agent` and `Task` even with a custom tool denylist. CLI `2.1.0` still makes an independent file-path-extraction model request after Bash and is incompatible with the single-context rollout contract. Probe other CLI versions before use; native session audit rejects auxiliary contexts, summaries, and unconsumed model responses.

Real training tasks are not repackaged by this repository: fetch Harbor-native datasets or tasks from Harbor Hub, and use the [compatibility summary](configs/harbor_hub/public_compatibility_summary.md) to understand validated coverage. After fetching a task root, validate it locally with `harborrl hub inspect` and lock it into your own catalog.
The bundled runnable example is `hello_world`. Real training tasks are not repackaged by this repository: fetch Harbor-native datasets or tasks from Harbor Hub, and use the [compatibility summary](configs/harbor_hub/public_compatibility_summary.md) to understand validated coverage. After fetching a task root, validate it locally with `harborrl hub inspect` and lock it into your own catalog.

## License

Expand Down
30 changes: 10 additions & 20 deletions README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,11 @@

## News

- **2026-10-26** 新增[离线配置向导](docs/configuration-wizard.md)和[本地 Web 控制台](docs/web-console.md),支持配置、校验、dry-run、训练启动和日志查看。
- **2026-10-26** 新增 [controller/GPU 部署配置](docs/deployment.md)、可选的 [OpenSandbox 执行能力](docs/sandbox-rollout.md),以及按需启用的[训练与推理一致性检查](docs/train-inference-consistency.md)。
- **10/2026** 📣📣📣 HarborRL released
- **2026-11-12** 新增[网关水平分片](docs/deployment.md):静态路由、弹性自动生长(远端优先)与自动缩容。
- **2026-11-12** 新增 [Hub 部署模式](docs/deployment.md):适配单向网络集群,节点经 HTTP 拉取任务,无需跨机 SSH。
- **2026-11-12** 新增[在线数学训练](docs/math-distill.md)(GRPO、DAPO、过程奖励、OPD、PRM 蒸馏),统一接入 Slime/Megatron + SGLang 管线。
- **2026-11-12** 新增[配置向导](docs/configuration-wizard.md)、[Web 控制台](docs/web-console.md)与带审计清理的 [Sandbox 执行](docs/sandbox-rollout.md)。
- **2026-11-12** 10/2026 📣 HarborRL 发布

## 项目介绍

Expand All @@ -46,12 +48,8 @@ HarborRL 支持 **Models × Algorithms × Tasks × Harnesses (MATH)** 的灵活
[docs/agents/environment-check.md](docs/agents/environment-check.md)。

离线选项目录、交互式配置和 answer-file 配置方式见[配置 CLI 指南](docs/configuration-wizard.md)。
首次使用可以先运行 `harborrl hello`:无需模型、GPU 或已有 task JSON,
即可执行脚本任务、真实 verifier、0/1 奖励收集和使用示例 token 的 GRPO batch 导出。
接着使用 `harborrl options` 或 `harborrl init --help`。Local 模式会检测并共用
一套源码目录与 Python;模型指纹自动计算,未填完的草稿支持 `:save` 保存和 `--resume` 恢复。
运行 `harborrl web` 可打开浏览器控制台,查看或套用 Local/Sandbox 示例、
随时保存草稿、一键运行 Hello World,并完成校验、dry-run、训练启动和日志查看;
可以从 `harborrl options` 或 `harborrl init --help` 开始。
运行 `harborrl web` 可打开本地浏览器控制台,完成配置、校验、dry-run、训练启动和日志查看;
远程访问与生命周期说明见 [Web 控制台指南](docs/web-console.md)。

推荐的实际操作流程是把下面这段提示词交给 agent,并把尖括号内容替换成你的机器信息:
Expand Down Expand Up @@ -112,14 +110,10 @@ uv run pytest
如需原生训练,请把固定版本的 native 训练栈安装到可写位置:

```bash
bash scripts/bootstrap_backends.sh "<Path to HarborRL backends>"
. "<Path to HarborRL backends>/harborrl-backend.env"
bash scripts/bootstrap_backends.sh <Path to HarborRL backends>
. <Path to HarborRL backends>/harborrl-backend.env
```

生成的文件会将 `SLIME_DIR`、`MEGATRON_DIR` 和 `SGLANG_IMAGE` 导出给子进程。
对于旧版 bootstrap 脚本生成的文件,请在 source 前执行 `set -a`,之后执行 `set +a`。
清理标记的恢复步骤和无 GPU 回归命令见[rollout 生命周期与恢复](docs/rollout-runtime-lifecycle.md)。

### 训练

完成安装后,从两种配置方式中选择一种:
Expand Down Expand Up @@ -158,11 +152,7 @@ task/

其中 `task.toml` 声明资源、agent 与 verifier,`instruction.md` 是给 agent 的任务说明,`environment/Dockerfile` 构建隔离执行环境,`tests/test.sh` 负责验证并输出 `reward.json` 或 `reward.txt`。训练时,这些任务会被锁定到 catalog 中,记录任务 ID、上游 revision、本地路径、内容 digest 和 reward 解析规则;示例见 [examples/native/catalog.json](examples/native/catalog.json)。

仓库内置的可直接运行示例是 `hello_world`。示例允许联网,以便 Harbor 安装固定版本的 Claude Code CLI;内存与存储各配置 4 GiB,verifier 将 0/1 奖励写入 `/logs/verifier/reward.json`。profile 允许四轮交互,供 agent 执行工具、读取结果并正常结束。离线部署需要预装完全一致的 CLI 版本,并确保能够访问 policy gateway。修改任务内容后,需要重新生成 catalog 中的内容摘要。

示例固定使用经过运行验证的 Claude Code `2.1.141`。adapter 默认关闭 prompt caching 及缓存预热、上下文压缩和非必要流量;自定义工具禁用列表也会保留 `Agent` 和 `Task`。CLI `2.1.0` 在 Bash 执行后仍会发起独立的文件路径提取请求,不符合单上下文 rollout 契约。使用其他 CLI 版本前需要重新验证;原生会话审计会拒绝辅助上下文、摘要和未消费的模型响应。

实际训练任务不由本 repo 重新打包,而是从 Harbor Hub 获取符合 Harbor native task 格式的 dataset/task;可先查看[兼容性汇总](configs/harbor_hub/public_compatibility_summary_zh.md)了解已验证覆盖情况。取回 task root 后,先用 `harborrl hub inspect` 做本地静态校验,再锁定进自己的 catalog。
仓库内置的可直接运行示例是 `hello_world`。实际训练任务不由本 repo 重新打包,而是从 Harbor Hub 获取符合 Harbor native task 格式的 dataset/task;可先查看[兼容性汇总](configs/harbor_hub/public_compatibility_summary_zh.md)了解已验证覆盖情况。取回 task root 后,先用 `harborrl hub inspect` 做本地静态校验,再锁定进自己的 catalog。

## License

Expand Down
81 changes: 81 additions & 0 deletions docs/architecture/p0_p1_config_and_boundaries_zh.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# P0/P1:配置生效与模块边界

本次变更延续已有 GPU 验收的默认训练路径,保持 README、schema-2 和用户命令不变。涉及导出代码的收敛保留原 GRPO 数学公式与权重规则;本轮验证采用 CPU 合同、旧实现数值对照、shell dry-run 和包构建,不代表新增 GPU 实测。

## 配置如何生效

优先级:YAML 原值 → 按顺序应用 `--set` → 显式 `-model/-harness/-task/-alg` 选择。后端路径、可见 GPU、代理及动态库等部署环境仍由声明的环境变量提供。结构化启动不继承环境中的 `LR`、采样预算、checkpoint 开关或其他未声明训练选项,避免覆盖 YAML。

`training.learning_rate` 现在显式映射为 `LR`,进入 Slime `--lr`。旧默认 `1e-6` 保持不变;此前被忽略的非默认 YAML 学习率现在会生效。

配置字段到环境的直接映射统一维护于 `harborrl/backends/slime_v032/config.py::CONFIG_ENV`;`field_usage()` 同时记录不通过环境消费的字段,测试要求覆盖全部 schema-2 字段。

| 配置 | 实际消费者 |
| --- | --- |
| `execution.backend`、`harness.name` | 当前执行/harness 组合校验及 runner |
| `tasks.catalog` | 静态 task lock 校验;运行时生成锁定 catalog 和 prompt data |
| `harness.profile` | 最终 profile 文件校验;runner 配置 |
| `harbor.python/version/workers` | runner Python、worker runtime probe、worker 调度 |
| `gateway.host/port/advertised_url` | gateway listener 和 worker 访问入口 |
| `gateway.tokenizer_digest/template_digest/audited_raw_logprobs` | 生成证据与训练资格检查 |
| `model.checkpoint/reference/args_file` | `--hf-checkpoint`、`--load`、`--ref-load` 与模型 preset |
| `training.learning_rate/num_rollout/save_interval` | `--lr`、`--num-rollout`、`--save-interval` |
| `training.backend_contract` | 后端 selector 与版本/hooks 检查 |
| `sampling.group_size/groups_per_batch` | `--n-samples-per-prompt`、`--rollout-batch-size` |
| `sampling.max_tokens/max_context` | response/context 预算 |
| `sampling.max_attempts` | 完整组 slot 重试限制 |
| `deployment.*` | GPU 数量、TP、split/colocate 校验和启动参数 |
| `output.root` | run、checkpoint 与指标根目录 |

严格配置校验与文件检查分离:`validate()` 不读 harness/task 文件,也不执行 live probe;`validate_files()` 校验最终选定的 harness profile;`tasks.catalog.catalog()` 校验任务锁;doctor 执行 live 部署检查。`load_config()` 保留加载后校验 profile 的兼容行为,但在 override 完成后检查文件,因此可以替换已经不存在的旧 profile 路径。

## 运行证据

每次结构化训练新增:

- `effective-config.json`:解析后配置、task selectors、组件标识、运行时环境映射、声明的部署信息、doctor 观察到的包版本/后端 commit、源码证据与字段消费者。
- `effective-command.json`:shell 默认值和模型 preset 展开后的实际可执行 argv,在调用 Slime 前记录。
- `dry-run-commands/<command digest>.json`:直接运行 shell dry-run 时的命令证据,允许同目录重复检查不同参数。

有效配置快照不转储整个父进程环境,不记录代理 URL 或环境中的 provider 凭据。原有 `launch.json`、源码 manifest、source patch 和任务产物保留。

`harborrl train --dry-run` 仍只解析并输出 plan,不启动 doctor 或训练;shell 命令证据是在实际执行 shell launcher 时生成的。若进程在进入 launcher 前失败,最终命令文件可能尚不存在。

## 模块职责

| 模块 | 职责 |
| --- | --- |
| `config/native.py` | schema-2 解析、类型/约束、路径规则、最终 profile 校验;保留旧 catalog/launch_plan 薄封装 |
| `backends/slime_v032/config.py` | Slime 配置映射与字段消费者清单 |
| `tasks/catalog.py` | native catalog 与 task lock 检查 |
| `tasks/content.py` | task 内容 digest 与共享 TOML reader |
| `platform/preflight.py` | live doctor;task digest 每次 catalog 检查一次 |
| `platform/evidence.py` | 输入准备、源码证据、有效配置与最终命令记录 |
| `platform/native_train.py` | 流程派发、run 目录、子进程、信号转发;旧 doctor/准备函数保留封装 |
| `export/native.py` | canonical immutable GRPO TrainingBatch |
| `rollout/exporters/native.py` | canonical batch 的旧 record 格式视图 |

模型 tool parser 与 harness 插件接口留给 P2;本阶段不提前扩大支持组合,也不改变 serving、loss、同步屏障或 checkpoint 生命周期。

## 历史工具清单与处置

| 历史路径 | 当前调用关系 | 处置 |
| --- | --- | --- |
| `data/harbor/__main__.py` | 模块 CLI → inspector/materializer | 保留历史入口,标注 TerminalEnv 兼容用途 |
| `data/harbor/inspector.py` | 历史 CLI、materializer、probe | 保留;共享 digest/TOML 从 `tasks/content.py` 兼容导出 |
| `data/harbor/materializer.py` | 历史 CLI;生成 task.yaml、Compose、wrapper | 保留以支持旧记录复现,native 训练不使用 |
| `data/harbor/probe.py`、`receipt.py` | 旧 TerminalEnv 动态探测和奖励解析 | 保留并标注兼容用途 |
| `data/harbor/native_inspector.py` | native catalog 与 hub inspect | 直接依赖 `tasks/content.py`,不再导入历史 inspector/materializer |
| `rollout/exporters/native.py` | GroupSlots、offline smoke、旧合同调用者 | 保留格式入口,数学及 batch 校验共用 canonical exporter |

未删除本地环境、缓存、runs 或验收记录。旧接口在确认无复现/外部调用需求后再单独决定退役。

## 回归重点

1. YAML 原值与 `--set` 的学习率均进入最终 `--lr`;预算、GPU、checkpoint 和输出参数通过命令级检查。
2. 父环境的训练变量不能覆盖 YAML;有效配置不包含父环境凭据与代理值。
3. 非法 section、字段类型、override YAML 和 catalog entry 返回可定位的 CLI 错误。
4. 有效组和零方差组、多 turn、不同轨迹长度的旧导出字段与权重保持一致;缺失证据、不完整组和重复身份继续拒绝。
5. native task inspection 不导入历史转换路径;原始 verifier 与 task digest 算法保持不变。

本次验证结果:102 项 CPU 测试通过;Ruff、shell 语法、示例 smoke、训练 plan dry-run、公开树审计通过。旧/新 exporter 在有方差与零方差、多 turn、不同长度样本上的 JSON 输出逐字一致。使用满足 pyproject 构建要求的隔离环境成功构建 sdist/wheel,并在仓库外安装 wheel 后通过 CLI dry-run 与新增模块导入。
Loading
Loading