-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathindex.html
More file actions
389 lines (364 loc) · 20.5 KB
/
Copy pathindex.html
File metadata and controls
389 lines (364 loc) · 20.5 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning.">
<title>Robo-ValueRL</title>
<link rel="icon" href="data:,">
<link rel="stylesheet" href="styles.css">
</head>
<body>
<header class="site-header">
<a class="brand" href="#top" aria-label="Robo-ValueRL home">
<img class="project-logo" src="assets/logos/robo-valuerl-logo.png" alt="Robo-ValueRL">
</a>
<div class="header-logo-group" aria-label="Affiliated labs">
<a class="lab-logo-link" href="https://gewu-lab.github.io/" aria-label="GeWu-Lab website">
<img class="lab-logo gewu-logo" src="assets/logos/gewu-lab.png" alt="GeWu-Lab">
</a>
<span class="logo-separator" aria-hidden="true">×</span>
<a class="lab-logo-link" href="https://www.x-humanoid.com/" aria-label="X-Humanoid website">
<img class="lab-logo xhumanoid-logo" src="assets/logos/x-humanoid.png" alt="X-Humanoid">
</a>
</div>
</header>
<nav class="side-nav" aria-label="Project sections">
<a href="#intro">Intro</a>
<a href="#tasks">Tasks</a>
<a href="#method">Method</a>
<a href="#results">Results</a>
<a href="#condition">Condition</a>
<a href="#citation">Citation</a>
</nav>
<main id="top">
<section id="intro" class="hero section-shell reveal">
<p class="eyebrow">Offline-to-Online Robotic Reinforcement Learning</p>
<h1>Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning</h1>
<div class="paper-meta" aria-label="Authors and affiliations">
<p class="authors">
<a href="https://xwinks.github.io/">Wenke Xia</a><sup>1,*</sup>,
<a href="https://scholar.google.com/citations?user=hcqLADMAAAAJ&hl=zh-CN">Pei Ren</a><sup>2,*</sup>,
<a href="https://namelesscrew.github.io/">Wenbo Yu</a><sup>3</sup>,
Yizhuo Zhang<sup>1</sup>, Jifan Li<sup>1</sup>, Yixue Zhang<sup>2,4</sup>,
<a href="https://nuomizai.github.io/">Yinuo Zhao</a><sup>2</sup>, Qingyang Gao<sup>2</sup>,
<br class="author-break">
<span class="author-second-line"><a href="https://www.microsoft.com/en-us/research/people/jianf/">Jianlong Fu</a><sup>5</sup>,
Jian Tang<sup>2</sup>, Ji-Rong Wen<sup>1</sup>, <a href="https://chezhengping.xyz/">Zhengping Che</a><sup>2,†</sup>,
<a href="https://dtaoo.github.io/">Di Hu</a><sup>1,†</sup></span>
</p>
<ol class="affiliations">
<li><sup>1</sup> Gaoling School of Artificial Intelligence, Renmin University of China</li>
<li><sup>2</sup> Beijing Innovation Center of Humanoid Robotics</li>
<li><sup>3</sup> Beijing Forestry University</li>
<li><sup>4</sup> Peking University</li>
<li><sup>5</sup> Microsoft Research</li>
</ol>
<p class="contribution-note">
<span>* Equal contribution</span>
<span>† Corresponding author</span>
</p>
</div>
<div class="hero-grid">
<p class="hero-copy">
Robo-ValueRL studies how value-function reliability shapes offline-to-online robotic reinforcement learning.
It learns history-conditioned value estimates, propagates them into quality-conditioned policy pretraining,
and uses reliable value guidance to stabilize online improvement from real robot rollouts.
</p>
<div class="hero-note" aria-label="Project highlights">
<span>240h offline data</span>
<span>3,000+ online rollouts</span>
<span>86% chip insertion</span>
<span>84% block disassembly</span>
</div>
</div>
<div class="resource-grid" aria-label="Project resources">
<a class="resource-link" href="https://arxiv.org/abs/2607.09866">
<span>Paper</span>
<strong>arXiv</strong>
</a>
<a class="resource-link" href="https://github.com/Open-X-Humanoid/Robo-ValueRL">
<span>Code</span>
<strong>GitHub</strong>
</a>
<a class="resource-link" href="https://huggingface.co/collections/X-Humanoid/robo-valuerl">
<span>Model & Dataset</span>
<strong>Hugging Face</strong>
</a>
<a class="resource-link" href="assets/community/robovaluerl-wechat-community.png">
<span>Community</span>
<strong>QR Code</strong>
</a>
</div>
<div class="video-placeholder hero-video overview-video" data-video="assets/videos/overview.mp4" data-video-ready="true" role="img" aria-label="Robo-ValueRL overview video">
<span class="play-mark" aria-hidden="true"></span>
<div>
<strong>Overview Video</strong>
<p>Robo-ValueRL system introduction.</p>
</div>
</div>
</section>
<section id="tasks" class="section-shell reveal">
<div class="section-kicker">Task Videos</div>
<div class="section-heading task-heading">
<h2>Two real-robot testbeds for value-guided policy improvement.</h2>
</div>
<div class="task-showcase" data-task-showcase data-task-autoplay="true">
<div class="task-copy-stack" role="tablist" aria-label="Task demo selector">
<button class="task-copy task-selector-card is-active" type="button" data-task-target="chip" aria-current="true">
<p class="task-label">Precision-critical manipulation</p>
<h3>Millimeter-level chip insertion</h3>
<p>The robot grasps a PCB, adjusts it to a feasible insertion pose, then grasps and inserts a chip into millimeter-scale clearance.</p>
<strong class="metric">86% final success</strong>
</button>
<button class="task-copy task-selector-card" type="button" data-task-target="block" aria-current="false">
<p class="task-label">Generalization-critical coordination</p>
<h3>Generalizable block disassembly</h3>
<p>The robot disassembles randomly placed blocks and sorts them into color-matched plates under randomized plate layouts.</p>
<strong class="metric">84% final success</strong>
</button>
</div>
<div class="task-video-rail" aria-label="Task videos">
<article class="task-video-panel is-active" data-task-target="chip" data-task-media="chip" tabindex="0" aria-label="Play chip insertion video">
<div class="video-placeholder task-video portrait-video" data-video="assets/videos/chip-insertion.mp4" data-video-ready="true" data-aspect="368 / 640" role="img" aria-label="Chip insertion video">
<span class="play-mark" aria-hidden="true"></span>
<div>
<strong>Chip Insertion</strong>
<p>Video loading area</p>
</div>
</div>
</article>
<article class="task-video-panel" data-task-target="block" data-task-media="block" tabindex="0" aria-label="Play block disassembly video">
<div class="video-placeholder task-video portrait-video" data-video="assets/videos/block-disassembly.mp4" data-video-ready="true" data-aspect="368 / 640" role="img" aria-label="Block disassembly video">
<span class="play-mark" aria-hidden="true"></span>
<div>
<strong>Block Disassembly</strong>
<p>Video loading area</p>
</div>
</div>
</article>
</div>
</div>
</section>
<section id="method" class="section-shell reveal">
<div class="section-kicker">Methodology</div>
<div class="section-heading">
<h2>Reliable values become a data-utilization interface.</h2>
<p>The framework links value estimation, quality-conditioned policy learning, and residual online adaptation in one offline-to-online pipeline.</p>
</div>
<figure class="figure-panel pipeline-panel">
<img src="assets/figures/pipeline.png" alt="Robo-ValueRL offline-to-online reinforcement learning pipeline">
</figure>
<div class="method-video-stack" aria-label="Robo-ValueRL method modules">
<article class="method-module">
<div class="method-copy">
<span>01</span>
<h3>History-conditioned value estimator</h3>
<p>Recent visual history reduces ambiguity from occlusions, repeated motions, and visually similar task stages. The three clips can later show value progress, error sensitivity, and history ablations.</p>
</div>
<div class="video-switcher" data-switcher data-autoplay="true" data-interval="8000">
<div class="switcher-tabs" role="tablist" aria-label="Value estimator videos">
<button type="button" class="is-active" data-panel="value-progress">Value Progress</button>
<button type="button" data-panel="error-sensitivity">Error Sensitivity</button>
<button type="button" data-panel="history-ablation">History Ablation</button>
</div>
<div class="switcher-stage">
<div class="switcher-panel is-active" data-panel-id="value-progress">
<div class="video-placeholder method-video" data-video="assets/videos/value-progress.mp4" data-aspect="1008 / 586" data-marker="Value Signal" data-note="Dense value estimates rise along successful progress." role="img" aria-label="Value progress video placeholder">
<span class="play-mark" aria-hidden="true"></span>
<div><strong>Value Progress</strong><p>Estimator demo placeholder</p></div>
</div>
</div>
<div class="switcher-panel" data-panel-id="error-sensitivity">
<div class="video-placeholder method-video" data-video="assets/videos/error-sensitivity.mp4" data-aspect="1008 / 586" data-marker="Error Cue" data-note="Sharp drops highlight unstable or low-quality states." role="img" aria-label="Error sensitivity video placeholder">
<span class="play-mark" aria-hidden="true"></span>
<div><strong>Error Sensitivity</strong><p>Estimator demo placeholder</p></div>
</div>
</div>
<div class="switcher-panel" data-panel-id="history-ablation">
<div class="video-placeholder method-video" data-video="assets/videos/history-ablation.mp4" data-aspect="1008 / 586" data-marker="History Window" data-note="Temporal context separates visually similar stages." role="img" aria-label="History ablation video placeholder">
<span class="play-mark" aria-hidden="true"></span>
<div><strong>History Ablation</strong><p>Estimator demo placeholder</p></div>
</div>
</div>
</div>
</div>
</article>
<article class="method-module">
<div class="method-copy">
<span>02</span>
<h3>Quality-conditioned consistency policy</h3>
<p>Value differences become action-quality indicators for efficient one-step action generation under high-frequency control. This group can show quality labels and task-specific policy behavior.</p>
</div>
<div class="video-switcher" data-switcher data-autoplay="true" data-interval="8000">
<div class="switcher-tabs" role="tablist" aria-label="Policy pretraining videos">
<button type="button" class="is-active" data-panel="quality-labels">Policy Demo 1</button>
<button type="button" data-panel="chip-policy">Policy Demo 2</button>
</div>
<div class="switcher-stage">
<div class="switcher-panel is-active" data-panel-id="quality-labels">
<div class="video-placeholder method-video" data-video="assets/videos/quality-labels.mp4" data-aspect="1008 / 736" role="img" aria-label="Policy demo 1 video placeholder">
<span class="play-mark" aria-hidden="true"></span>
<div><strong>Policy Demo 1</strong><p>Policy demo placeholder</p></div>
</div>
</div>
<div class="switcher-panel" data-panel-id="chip-policy">
<div class="video-placeholder method-video" data-video="assets/videos/chip-policy.mp4" data-aspect="1008 / 736" role="img" aria-label="Policy demo 2 video placeholder">
<span class="play-mark" aria-hidden="true"></span>
<div><strong>Policy Demo 2</strong><p>Policy demo placeholder</p></div>
</div>
</div>
</div>
</div>
</article>
<article class="method-module">
<div class="method-copy">
<span>03</span>
<h3>Online residual adaptation</h3>
<p>High-quality rollout segments train a lightweight adapter while preserving the pretrained base behavior. Later videos can show the residual gate, self-correction, and failure recovery.</p>
</div>
<div class="single-method-video">
<div class="video-placeholder method-video method-video-direct" data-video="assets/videos/residual-gate.mp4" data-aspect="672 / 384" data-direct-autoplay="true" role="img" aria-label="Residual gate video">
<span class="play-mark" aria-hidden="true"></span>
<div><strong>Residual Gate</strong><p>Adaptation demo placeholder</p></div>
</div>
</div>
</article>
</div>
</section>
<section id="results" class="section-shell reveal">
<div class="section-kicker">Results</div>
<div class="section-heading">
<h2>Reliability metrics predict downstream policy quality.</h2>
<p>Across 240 hours of offline demonstrations and over 3,000 online rollout trajectories, reliable value estimates support stronger offline scaling and more stable online improvement.</p>
</div>
<div class="stat-grid">
<div>
<strong>240h</strong>
<span>offline demonstrations</span>
</div>
<div>
<strong>3,000+</strong>
<span>online rollout trajectories</span>
</div>
<div>
<strong>+26%</strong>
<span>chip offline gain</span>
</div>
<div>
<strong>+34%</strong>
<span>block offline gain</span>
</div>
</div>
<div class="result-layout">
<figure class="figure-panel">
<img src="assets/figures/value-diagnostics.png" alt="Value estimation metrics and downstream success rate">
</figure>
<div class="result-copy">
<h3>From value reliability to policy improvement</h3>
<p>Reliable value functions preserve global progress, maintain local fluency, and detect execution errors. These properties align with downstream policy success.</p>
<ul>
<li>Value-guided offline RL scales better than quality-agnostic behavior cloning.</li>
<li>Reliable value guidance filters noisy online rollouts.</li>
<li>Residual adaptation improves behavior without overwriting the base policy.</li>
</ul>
</div>
</div>
<div class="evidence-grid">
<figure class="figure-panel compact">
<img src="assets/figures/offline-results.png" alt="Offline pretraining scaling with heterogeneous data">
<figcaption>Offline pretraining scaling with heterogeneous data</figcaption>
</figure>
<figure class="figure-panel compact">
<img src="assets/figures/online-results.png" alt="Success rate and failure-mode distribution">
<figcaption>Success rate and failure-mode distribution</figcaption>
</figure>
</div>
<div class="recovery-band">
<div>
<p class="task-label">Detailed demos</p>
<h3>Recovery behavior belongs with the evidence.</h3>
<p>Failure recovery, self-correction, and online adaptation are treated as method/result evidence rather than intro tasks.</p>
</div>
<div class="detailed-demo-gallery" aria-label="Detailed recovery behavior demos">
<figure class="demo-card demo-card-main">
<div class="video-placeholder demo-video" data-video="assets/videos/detailed-chip-2.mp4" data-aspect="1280 / 720" data-direct-autoplay="true" role="img" aria-label="Detailed chip recovery demo 1">
<span class="play-mark" aria-hidden="true"></span>
<div><strong>Chip Recovery 1</strong><p>Detailed demo loading area</p></div>
</div>
</figure>
<figure class="demo-card">
<div class="video-placeholder demo-video" data-video="assets/videos/detailed-chip-3.mp4" data-aspect="1280 / 720" data-direct-autoplay="true" role="img" aria-label="Detailed chip recovery demo 2">
<span class="play-mark" aria-hidden="true"></span>
<div><strong>Chip Recovery 2</strong><p>Detailed demo loading area</p></div>
</div>
</figure>
<figure class="demo-card">
<div class="video-placeholder demo-video" data-video="assets/videos/detailed-chip-4.mp4" data-aspect="1280 / 720" data-direct-autoplay="true" role="img" aria-label="Detailed chip recovery demo 3">
<span class="play-mark" aria-hidden="true"></span>
<div><strong>Chip Recovery 3</strong><p>Detailed demo loading area</p></div>
</div>
</figure>
<figure class="demo-card demo-card-portrait">
<div class="video-placeholder demo-video portrait-demo-video" data-video="assets/videos/detailed-block-portrait.mp4" data-aspect="720 / 1280" data-direct-autoplay="true" role="img" aria-label="Detailed block recovery portrait demo">
<span class="play-mark" aria-hidden="true"></span>
<div><strong>Block Recovery</strong><p>Portrait demo loading area</p></div>
</div>
</figure>
</div>
</div>
</section>
<section id="condition" class="section-shell condition-section reveal">
<div class="section-kicker">Condition</div>
<div class="section-heading">
<h2>Value estimates become conditions for policy improvement.</h2>
<p>Robo-ValueRL treats value reliability as an interface between heterogeneous robot experience and downstream policy learning.</p>
</div>
<div class="condition-panel">
<div class="condition-copy">
<p>
A history-conditioned value estimator infers task progress from multi-view observations and visual history.
Its value differences are converted into action-quality indicators, allowing the VLA policy to condition on
high-quality behavior during offline pretraining.
</p>
<p>
During online improvement, the same value guidance filters rollout segments and activates a lightweight
residual adapter only where targeted correction is useful. This keeps adaptation focused while preserving
the pretrained behavior prior.
</p>
<div class="condition-points" aria-label="Conditioning signals">
<span>History-conditioned value</span>
<span>Action-quality condition</span>
<span>Value-guided residual adaptation</span>
</div>
</div>
<figure class="condition-figure">
<img src="assets/figures/condition-teaser.png" alt="Robo-ValueRL value conditioning overview">
</figure>
</div>
</section>
<section id="citation" class="section-shell citation-section reveal">
<div class="section-kicker">Citation</div>
<div class="section-heading">
<h2>Cite Robo-ValueRL.</h2>
<p>Please cite the arXiv paper if you find Robo-ValueRL useful.</p>
</div>
<pre class="bibtex" id="bibtex">@misc{xia2026robovaluerlreliablevalueestimation,
title={Robo-ValueRL: Reliable Value Estimation for Offline-to-Online Reinforcement Learning},
author={Wenke Xia and Pei Ren and Wenbo Yu and Yizhuo Zhang and Jifan Li and Yixue Zhang and Yinuo Zhao and Qingyang Gao and Jianlong Fu and Jian Tang and Ji-Rong Wen and Zhengping Che and Di Hu},
year={2026},
eprint={2607.09866},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2607.09866},
}</pre>
<button class="copy-button" type="button" data-copy-target="bibtex">Copy BibTeX</button>
</section>
</main>
<footer class="site-footer">
<span>Robo-ValueRL</span>
<a href="https://github.com/GeWu-Lab/Robo-ValueRL.git">GitHub</a>
</footer>
<script src="script.js"></script>
</body>
</html>