From 21f2fce687eba02f4ee24a900229eb9d458c68ca Mon Sep 17 00:00:00 2001 From: Yuxin Wang <3275317723@qq.com> Date: Mon, 24 Aug 2026 14:16:02 +0400 Subject: [PATCH] Add physics-consistent video generation discussion --- .../action-items.md | 5 + .../assets/.gitkeep | 0 .../idea-map.md | 28 + .../index.html | 41 + .../meeting-note.html | 21 + .../meeting-note.md | 3 + .../metadata.yaml | 42 + .../presentation.html | 1817 +++++++++++++++++ .../references.html | 75 + .../references.md | 57 + .../summary.html | 79 + discussions/index.html | 3 +- index.html | 6 +- people/index.html | 2 +- sitemap.xml | 4 + tags/index.html | 10 +- 16 files changed, 2184 insertions(+), 9 deletions(-) create mode 100644 discussions/2026-08-24-yuxin-physics-consistent-video-generation/action-items.md create mode 100644 discussions/2026-08-24-yuxin-physics-consistent-video-generation/assets/.gitkeep create mode 100644 discussions/2026-08-24-yuxin-physics-consistent-video-generation/idea-map.md create mode 100644 discussions/2026-08-24-yuxin-physics-consistent-video-generation/index.html create mode 100644 discussions/2026-08-24-yuxin-physics-consistent-video-generation/meeting-note.html create mode 100644 discussions/2026-08-24-yuxin-physics-consistent-video-generation/meeting-note.md create mode 100644 discussions/2026-08-24-yuxin-physics-consistent-video-generation/metadata.yaml create mode 100644 discussions/2026-08-24-yuxin-physics-consistent-video-generation/presentation.html create mode 100644 discussions/2026-08-24-yuxin-physics-consistent-video-generation/references.html create mode 100644 discussions/2026-08-24-yuxin-physics-consistent-video-generation/references.md create mode 100644 discussions/2026-08-24-yuxin-physics-consistent-video-generation/summary.html diff --git a/discussions/2026-08-24-yuxin-physics-consistent-video-generation/action-items.md b/discussions/2026-08-24-yuxin-physics-consistent-video-generation/action-items.md new file mode 100644 index 0000000..af42c17 --- /dev/null +++ b/discussions/2026-08-24-yuxin-physics-consistent-video-generation/action-items.md @@ -0,0 +1,5 @@ +# Action Items + +- [ ] Yuxin: present the method roadmap and clarify model capability versus system capability. +- [ ] All participants: identify one claim whose current evidence does not support its strongest interpretation. +- [ ] All participants: propose one controlled comparison that could distinguish physical generalization from template matching. diff --git a/discussions/2026-08-24-yuxin-physics-consistent-video-generation/assets/.gitkeep b/discussions/2026-08-24-yuxin-physics-consistent-video-generation/assets/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/discussions/2026-08-24-yuxin-physics-consistent-video-generation/idea-map.md b/discussions/2026-08-24-yuxin-physics-consistent-video-generation/idea-map.md new file mode 100644 index 0000000..033c995 --- /dev/null +++ b/discussions/2026-08-24-yuxin-physics-consistent-video-generation/idea-map.md @@ -0,0 +1,28 @@ +# Idea Map + +```mermaid +graph TD + A[Physics-consistent video generation] --> B[External physics and control] + A --> C[Generator learning] + A --> D[Inference-time alignment] + A --> E[Evaluation evidence] + B --> B1[Infer initial state] + B --> B2[Compute motion] + B2 --> B3[Render or complete interactions] + C --> C1[Simulation data] + C --> C2[Physical attributes] + C --> C3[Representation alignment] + C --> C4[Physics-aware rewards] + C --> C5[Latent physical dynamics] + D --> D1[World-model scoring] + D --> D2[Trajectory search and sampling guidance] + E --> E1[Open-domain human or VLM judgment] + E --> E2[Comparison with real futures] + E --> E3[Equations and physical invariants] + B3 --> F[System capability] + C5 --> G[Model capability] + D2 --> F + F --> H[Single-variable counterfactual tests] + G --> H + E3 --> H +``` diff --git a/discussions/2026-08-24-yuxin-physics-consistent-video-generation/index.html b/discussions/2026-08-24-yuxin-physics-consistent-video-generation/index.html new file mode 100644 index 0000000..9f3f3be --- /dev/null +++ b/discussions/2026-08-24-yuxin-physics-consistent-video-generation/index.html @@ -0,0 +1,41 @@ + + + + + + + Physics-Consistent Video Generation · CausalVideoGroup + + + + +
+ +
+

This discussion maps how physical knowledge enters a video-generation system through explicit simulation, training signals, latent dynamics, and inference-time alignment, while separating improvements to the generator from improvements to the full system.

+ +
+
+ + + diff --git a/discussions/2026-08-24-yuxin-physics-consistent-video-generation/meeting-note.html b/discussions/2026-08-24-yuxin-physics-consistent-video-generation/meeting-note.html new file mode 100644 index 0000000..7d975bd --- /dev/null +++ b/discussions/2026-08-24-yuxin-physics-consistent-video-generation/meeting-note.html @@ -0,0 +1,21 @@ + + + + + + + Meeting Note · Physics-Consistent Video Generation: Who Teaches the Generator Physics? · CausalVideoGroup + + + + +
+ +

The discussion is scheduled for 2026-08-24. Replace this notice after the presentation with changed views, evidence boundaries, critiques, research opportunities, concrete experiments, and explicitly assigned action items.

+
+ + + diff --git a/discussions/2026-08-24-yuxin-physics-consistent-video-generation/meeting-note.md b/discussions/2026-08-24-yuxin-physics-consistent-video-generation/meeting-note.md new file mode 100644 index 0000000..1b2bed2 --- /dev/null +++ b/discussions/2026-08-24-yuxin-physics-consistent-video-generation/meeting-note.md @@ -0,0 +1,3 @@ +# Meeting Note + +The discussion is scheduled for 2026-08-24. Replace this notice after the presentation with changed views, evidence boundaries, critiques, research opportunities, concrete experiments, and explicitly assigned action items. diff --git a/discussions/2026-08-24-yuxin-physics-consistent-video-generation/metadata.yaml b/discussions/2026-08-24-yuxin-physics-consistent-video-generation/metadata.yaml new file mode 100644 index 0000000..92c7823 --- /dev/null +++ b/discussions/2026-08-24-yuxin-physics-consistent-video-generation/metadata.yaml @@ -0,0 +1,42 @@ +title: "Physics-Consistent Video Generation: Who Teaches the Generator Physics?" +date: 2026-08-24 +topic_slug: physics-consistent-video-generation +leader: + name: "Yuxin Wang" + short_name: yuxin +status: before-meeting +summary: >- + A design-space view of how simulators, explicit controls, training signals, + latent dynamics, and inference-time alignment can make generated videos obey + real-world physical laws, and what current benchmarks can actually establish. +tags: + - video-generation + - physics-consistency + - world-model + - controllable-generation + - benchmark + - evaluation +related_projects: [] +papers: + - title: "PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation" + url: https://arxiv.org/abs/2409.18964 + role: system + note: Adds explicit rigid-body simulation before diffusion refinement. + - title: "PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Drop" + url: https://arxiv.org/abs/2503.09595 + role: method + note: Tests physics post-training and extrapolation through controlled free fall. + - title: "VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models" + url: https://arxiv.org/abs/2505.23656 + role: method + note: Distills spatiotemporal relations into the generator. + - title: "What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewards" + url: https://arxiv.org/abs/2512.00425 + role: method + note: Turns Newtonian motion proxies into verifiable rewards. + - title: "Inference-time Physics Alignment of Video Generative Models with Latent World Models" + url: https://arxiv.org/abs/2601.10553 + role: system + note: Guides inference-time search with a frozen latent world model. +external_videos: [] +slides: presentation.html diff --git a/discussions/2026-08-24-yuxin-physics-consistent-video-generation/presentation.html b/discussions/2026-08-24-yuxin-physics-consistent-video-generation/presentation.html new file mode 100644 index 0000000..480cbec --- /dev/null +++ b/discussions/2026-08-24-yuxin-physics-consistent-video-generation/presentation.html @@ -0,0 +1,1817 @@ + + + + + + + 物理一致性视频生成|谁在教生成模型物理? + + + +
↗CAUSAL VIDEO GROUP · DISCUSSION
+
01 / 29
+
+
+
+
+
+

CausalVideoGroup · 2026.08.24

+

物理一致性视频生成

+

谁在教生成模型物理?从外部模拟、显式控制,到训练内化与推理时修正。

+

Yuxin Wang · Group Discussion

+
+
+
+
+
状态 × 规律
→ 未来
+
+
+ + +
+
+ +
+
+ 02 +

The opening tension

+

漂亮的视频,不等于正确的世界

+

单帧可以逼真,连续演化仍可能违背重力、接触、守恒与材料响应。

+
+
+
+ 重力失效

落体突然悬停、上升或加速度漂移。

+
+
+
+ 碰撞失真

接触后方向、速度或形变没有因果响应。

+
+
+
+ 质量不守恒

液体或物体凭空增减、分裂后体积异常。

+
+
+
+ 穿透与接触错位

物体越过边界,却没有遮挡或反作用。

+
+
+
+ 支撑关系消失

人体或物体失去支撑后仍维持原动作。

+
+
+
第一道分界:时间平滑与外观连续,是物理一致性的必要基础,但不是充分证据。
+ + +
+
+ +
+
+ 03 +

Definition

+

什么是物理一致性?

+
+
+
生成视频中的物体运动、相互作用和状态变化,应符合真实世界的物理规律。
+
我们评的是演化:给定状态与条件,后续变化是否可由合理机制解释。
+
+
+
运动学位置、速度、加速度x(t), v(t), a(t)
+
动力学力、质量、接触响应F = ma
+
守恒与约束质量、动量、能量、边界ΔQ ≈ 0
+
材料与场刚体、流体、弹塑性、光热state → state'
+
+
+ + +
+
+ +
+
+ 04 +

The landscape

+

谁在教生成模型物理?

+

不是一条“外部 → 内部”的替代史,而是多种知识来源与作用位置持续扩展。

+
+
模拟器与显式状态先算状态和轨迹,再交给生成器渲染。
+
↔
+
VLM / 学习式动力学计划运动、预测轨迹、提供训练监督。
+
↔
+
数据、表征与 reward通过训练或 post-training 改变生成权重。
+
↔
+
搜索、引导与修正推理时从已有分布中找更合理结果。
+
+
+
位置问题:物理知识在生成器、辅助模块、输入,还是推理系统?
+
证据问题:模型在服从给定结果,还是根据物理条件推断结果?
+
+ + +
+
+ +
+
+ 05 +

Source × stage

+

物理知识可以从哪里进入生成系统?

+
+ + + + + + + + + + + +
知识来源 ↓ / 进入阶段 →生成前规划训练 / post-training生成中控制推理后评价 / 修正
模拟器与显式状态初态 → 轨迹模拟数据 / 属性flow / track / map重建再模拟
VLM / 视频理解模型事件与粗轨迹表征 / reward提示与 guidancejudge / verifier
真实或模拟视频—SFT / preference—真实未来参照
人类与模型反馈规则描述DPO / RL在线选择排序与重试
可测量物理规律方程先验loss / reward约束引导定律残差
+
没有单一最优入口:关键是约束位于哪里、推理依赖什么,以及它能提供多强的保证。
+
+ + +
+
+ +
+
+ 06 +

Audit lens

+

阅读每种方法时,我们检查什么?

+
+
+
+
生成器权重或建模对象是否改变?
+
辅助模块动力学、VLM、reward 是否另训?
+
输入需要轨迹、力、材料或真值吗?
+
推理是否增加采样、搜索或修正?
+
+
证据阶梯:案例 / VLM 分数 → 可测规律 → 参数外推 → 单变量干预 → 跨外观机制稳定。
+
+
+ Weak evidence +

“整个系统更好”

+

只说明组合后的输出改善,无法定位知识来自哪里。

+

“模型学到机制”

+

需要权重层面的改变,以及对未见条件和受控干预的稳定响应。

+
Strong evidence · mechanism-level
+
+
+ + +
+
+ +
+
+ 07 +

Part I · Explicit control

+

最直接的方案:让模拟器算,让生成模型画

+

先得到物理上可接受的运动骨架,再把视觉真实感交给生成器。

+
+
+
输入文本 / 单图 / 干预
+
→
+
初始状态估计几何、材料、接触、速度
+
→
+
物理演化模拟器或学习式动力学
+
→
+
控制信号轨迹、flow、属性图
+
→
+
视频生成渲染、补纹理与细节
+
+
+
谁变了生成器辅助模块输入推理系统SYSTEM-LEVEL
+ + +
+
+ +
+
+ 08 +

External simulator

+

模拟器给出确定骨架,生成器补回开放外观

+
+
+
2023
+ GPT4Motion + GPT-4 写 Blender 脚本;物理引擎制作基本运动,再由扩散模型视觉化。 + +
+
+
2024
+ PhysGen + 单图理解 → 图像空间刚体模拟 → 视频扩散细化。 + +
+
+
2026
+ PSIVG + 模板视频 → 4D 恢复 → 模拟轨迹 → 生成修正,形成 simulator-in-the-loop。 + +
+
+
边界:轨迹更明确,不代表世界状态就估对了;错误的几何、材料或接触会被“精确地模拟”。
+
谁变了生成器辅助模块输入推理系统SYSTEM-LEVEL
+ + +
+
+ +
+
+ 09 +

State identifiability

+

真正困难的,往往不是求解器,而是初始状态

+
+
+
几何三维形状、厚度、遮挡后的结构
+
物性质量、材料、摩擦、恢复系数
+
关系接触、支撑、连接与边界条件
+
动态初始速度、角速度、外力与约束
+
+
+
image → ? state → simulator → future单张图像只提供投影;多个隐藏状态可以解释同一外观,却导向不同未来。
+
+

PhysMotion

+

从单图构造 3D Gaussian,并以连续介质 / MPM 模拟中间运动,再用扩散模型细化外观。

+
+
不可辨识性:如果输入不能唯一确定初态,再精确的模拟也只能给出条件性的正确结果。
+
+
+
谁变了生成器辅助模块输入推理系统SYSTEM-LEVEL
+ + +
+
+ +
+
+ 10 +

Outcome vs cause

+

控制的是运动结果,还是物理条件?

+
+
+ Outcome-like control +

运动结果类

+

mask sequence · depth / edge · optical flow · 2D / 3D trajectory

+
大部分未来已经给出:评测更接近控制服从与渲染质量。
+
“照这条路径走”
+
+
+ Cause-like condition +

物理条件类

+

位置 / 速度 · force · mass · friction · restitution · material

+
仍需推断演化:同一场景下改变条件,未来应按规律变化。
+
“受这个力以后会怎样?”
+
+
+
trajectory 更像答案;force 更像原因。两者都可用,但支持的“物理理解”结论不同。
+ + +
+
+ +
+
+ 11 +

Boundary case · InterDyn

+

只控制一个物体,模型能否推断其他物体的响应?

+
+
+
+
初始帧场景与对象
+
+
+
driving entity用户给 mask sequence
+
→
+
视频扩散生成完整交互
+
+
+
未来 A被撞物向左旋转
+
未来 B被撞物滑动后停下
+
未来 C多个对象连锁响应
+
+
+
+ CONTROL / INFERENCE BOUNDARY +
InterDyn 让模型补全未被直接控制的交互对象。
+
积极证据:模型不是逐像素复刻全部未来,而是在控制条件下生成其他对象的响应。
+
审计问题:mask sequence 是否已经编码了大部分碰撞时机与运动结果?
+
+
+
谁变了SVD 主干冻结控制分支mask sequence推理系统HYBRID
+ + +
+
+ +
+
+ 12 +

VLM planning

+

VLM 成为物理 planner

+
+
+
把开放世界的“常识”先翻译成粗运动计划,再交给视频模型实现。
+
风险链:计划幻觉 → 临时运动偏差 → flow 误差 → 生成器放大。
+
+
+
+
文本 / 图像开放场景
+
→
+
VLM 推理事件顺序 + 粗轨迹
+
→
+
临时运动合成并提取 optical flow
+
→
+
视频生成受 flow 控制
+
+
+

VLIPP 的取舍

+

覆盖面比手写模拟器开放;但“语义上合理的轨迹”并不自动满足精确物理规律。

+
+
边界归类:DiffPhy 也用 LLM/MLLM,但它以训练目标监督并微调生成器,因此放到下一部分。
+
+
+
谁变了 · VLIPP生成器VLM planner控制输入推理系统SYSTEM-LEVEL
+ + +
+
+ +
+
+ 13 +

Learned dynamics

+

用学习式动力学替代手写模拟器

+
+
+ LATENT DYNAMICS MODULE +

NewtonGen

+
state + parameter
→ Neural Newtonian Dynamics
+
    +
  • 学习位置、速度与一般动力学演化
  • +
  • 输出 latent dynamics guidance
  • +
  • 强调初值与动力学参数控制
  • +
+
谁变了视频生成器NND条件搜索
+
+
+ 3D TRAJECTORY MODULE +

PhysCtrl

+
3D points + force + material
→ trajectory diffusion
+
    +
  • 覆盖 elastic、sand、plasticine、rigid
  • +
  • 以 3D 点轨迹驱动视频生成
  • +
  • 训练数据来自大规模物理模拟
  • +
+
谁变了视频生成器动力学网络力 / 物性搜索
+
+
+
共同结论:物理知识主要在独立动力学模块;视频生成器接收其输出,不能据此直接宣称 backbone 内化了完整动力学。
+ + +
+
+ +
+
+ 14 +

Training transfers simulator knowledge

+

模拟器退出推理,但显式物理条件没有消失

+
+
+
+
Simulator系统变化 friction / restitution / deformation / force
+
→
+
100K+ videos像素对齐属性图 + 视频
+
→
+
ControlNet SFTfrozen video backbone
+
+
+
VLM reward针对属性控制问答
+
→
+
Inference初图 + 文本 + 属性图
+
→
+
Video无需在线模拟器
+
+
+
+ TRAINING + EXPLICIT CONDITION +
PhyCo 把模拟知识转成可学习的控制接口。
+
仍然显式:推理期虽无模拟器或几何重建,用户仍需提供物理属性图。
+
精确说法:训练的是 ControlNet;官方项目页说明视频扩散 backbone 保持冻结。
+
+
+
谁变了frozen backboneControlNet + VLM reward属性图在线模拟HYBRID
+ + +
+
+ +
+
+ 15 +

Part II · Training and inference

+

从替模型计算答案,到改变模型如何生成

+

五种监督接口作用于权重;推理策略则在不改权重时改变系统输出。

+
+
数据
模拟 / 真实视频
+
训练目标
temporal / physics loss
+
表征对齐
foundation features
+
preference
human / VLM / real data
+
规律 reward
measurable proxy
+
latent dynamics
state prediction
+
inference
search / guidance
+
+
+
Model-level:生成器权重、表征或预测对象被改变。
+
System-level:权重不变,借助 evaluator、搜索或采样策略改善结果。
+
+ + +
+
+ +
+
+ 16 +

Controlled post-training

+

PISA:少量物理数据究竟能教会什么?

+
+
+
+
受控模拟自由落体少量专门数据
+
→
+
SFT改变生成行为
+
→
+
Physics reward继续校正
+
+
+

Seen range

训练高度、速度与外观附近,行为改善最明显。

+

OOD parameters

未见高度和速度下,轨迹外推并不稳定。

+

New composition

一个落体行为不能证明可组合的通用物理。

+
+
+
+
它证明了post-training 能教行为;也暴露了“行为模板”与“物理规律”之间的距离。
+
替代解释:模型可能只是拟合“物体向下移动”的视频模板,而没有形成可干预的重力机制。
+
+
+
谁变了生成器权重reward model显式输入搜索MODEL-LEVEL
+ + +
+
+ +
+
+ 17 +

Physics curriculum

+

让训练数据更像一门物理课

+
+
+

WISA

+

把定律拆成文本描述、定性类别和定量属性;camera-ready 使用 WISA-80K。

+
+
+

Synthetic video

+

系统改变初态与参数,得到可配对、可测量、可干预的训练样本。

+
+
+

PhysVid

+

为局部时间块描述状态、交互与约束;推理时加入负物理提示。

+
+
+

DiffPhy

+

LLM 生成物理上下文,MLLM 提供训练监督,共同微调视频扩散模型。

+
+
+
+
更明确:监督从“这段视频不错”变成状态、事件、参数和约束。
+
仍有限:模型可能学习类别或运动模板;时间连贯性也不能替代力与材料建模。
+
+
谁变了生成器权重数据 / 监督模块推理输入视方法而定搜索MODEL-LEVEL
+ + +
+
+ +
+
+ 18 +

Representation alignment

+

向“更懂视频”的模型借表征

+
+
+
+
Frozen video encoder时空 token 关系 / motion subspace
+
↓
+
Alignment loss把 teacher 关系蒸馏进生成器中间层
+
↓
+
Video diffusion继续完成去噪与生成
+
+
+
+
+

VideoREPA

+

对齐 foundation model 的时空 token 关系,迁移更强的视频结构先验。

+
+
+

MoAlign

+

指出通用表征纠缠外观与运动,先学习由 optical flow 监督的 motion subspace。

+
+
证据边界:benchmark 提升说明表征有用,但不能单独证明它对应可解释、可干预的物理状态。
+
+
+ + +
+
+ +
+
+ 19 +

Preference and reward

+

用反馈纠正生成器

+
+
+
2024
+ AI Feedback + VLM 二元反馈优于简单 CLIP / flow reward,用于动态交互。 + +
+
+
2025.06
+ RDPO + 从真实视频反向采样,自动构造物理 preference pairs。 + +
+
+
2025.08
+ PhysHPO + instance / state / motion / semantic 四层 preference。 + +
+
+
2025.11
+ PhysCorr + 区分对象内稳定与对象间交互的 dual reward。 + +
+
+
+
问题一:feedback 来自人、VLM、真实视频,还是可验证规律?
+
问题二:生成器是否只学会了 evaluator 的偏好捷径?
+
+
谁变了生成器权重reward / preference pipeline显式输入通常不增加搜索MODEL-LEVEL
+ + +
+
+ +
+
+ 20 +

Verifiable reward

+

把物理规律做成可验证 reward

+
+
+
+
生成视频candidate x
+
→
+
Frozen utilitiesoptical flow / appearance features
+
→
+
Physics proxyv, a, mass-like signal
+
→
+
Rewardlaw residual ↓
+
+
恒加速度速度变化应近似线性Δv / Δt ≈ const.
+
质量守恒防止缩小 / 消失的退化解m(t) ≈ m(0)
+
碰撞 / 运动把可测关系写成局部约束residual → 0
+
+
+
NewtonRewards:从“看起来合理”走向“可重复计算”。
+
更可信:reward 的计算规则公开,可对每个样本复核。
+
更狭窄:规律越具体、越容易验证,覆盖的开放世界现象通常越少。
+
+
+
谁变了生成器权重冻结测量工具显式输入搜索MODEL-LEVEL
+ + +
+
+ +
+
+ 21 +

Latent physical state

+

从通用表征,到潜在物理动力学

+
+
+
+

DreamWorld

+

联合像素与多种 foundation-model feature,把几何、语义和时空知识整合为 world representation。

+
+
↓
+
+

Phantom

+

不只预测未来画面,同时预测 latent physical dynamics,让控制演化的状态进入模型预测对象。

+
+
+
+
+
observed frames可见外观与运动
+
→
+
latent physics抽象物理状态
+
→
+
future dynamics状态演化
+
→
+
future video视觉结果
+
+
+

可解释?

latent 对应哪些状态或属性?

+

可干预?

改变质量、力或材料能否定向改变未来?

+

可迁移?

换外观后,动力学机制是否保持?

+
+
+
+
谁变了生成器预测对象 / 权重多源 teacher显式条件搜索MODEL-LEVEL
+ + +
+
+ +
+
+ 22 +

Inference-time alignment

+

也许模型已经包含有用信息,只是默认采样没有利用

+
+
+
+
同一 prompt多个 seed / trajectory
+
→
+
候选 A · B · C · …并行或渐进生成
+
→
+
Physics scorerworld model / DiT feature probe
+
→
+
保留 / 引导更合理的轨迹
+
+
+
+

WMReward

+

以 latent world model 的预测 surprise 评价候选,并支持 Best-of-N 与梯度引导。

+
+
+

Seeking Physics in Diffusion Noise

+

从中间 DiT 特征识别 plausibility,早停低分去噪轨迹。

+
+
+
+
+ SYSTEM-LEVEL +
额外计算能找到更好的样本,不等于权重获得了新的物理知识。
+
更强的推断:冻结特征里至少存在可预测物理合理性的信号。
+
成本:候选数量、scorer 偏差与额外算力必须计入比较。
+
+
+
谁变了生成器权重world model / verifier输入搜索 / guidanceSYSTEM-LEVEL
+ + +
+
+ +
+
+ 23 +

Sampling dynamics

+

也许生成过程破坏了已有的运动先验

+
+
+
+
早期少步去噪粗糙外观,但运动骨架可能更合理
+
↓
+
视觉细化阶段纹理、边缘与细节逐步增强
+
↓
+
完整输出外观更好,运动相位可能被侵蚀
+
+
+
+
+

PhaseLock / Physics in 2-Steps

+

先从少步轨迹抽取 motion prior,再以 latent delta guidance 注入完整去噪。

+
+
+

Self-Refining Video Sampling

+

复用生成器作 denoising autoencoder;以 predict-and-perturb 内循环,并按 uncertainty 选择性精炼。

+
+
物理失败可能来自知识不足,也可能来自表示、目标或解码过程。
+
+
+ + +
+
+ +
+
+ 24 +

Synthesis

+

“模型更强”,还是“系统更强”?

+
+ + + + + + + + + +
路线改变了什么主要证据仍不能证明什么
物理数据生成器权重行为与 benchmark 改善跨规律的通用机制
表征对齐中间表征时空 / 物理分数改善可解释物理状态
preference / reward生成器权重偏好与指标提升evaluator 之外的泛化
latent dynamics预测对象latent 与视频表现改善稳定、可干预机制
inference-time alignment搜索 / 采样系统输出改善权重内化物理
+
+
Model-level claim:必须指出到底更新了 backbone、adapter、ControlNet 还是独立动力学网络。
+
System-level claim:必须报告额外 evaluator、候选数、推理预算与失败模式。
+
+
+ + +
+
+ +
+
+ 25 +

Part III · Evaluation

+

不同 benchmark 评测的不是同一个对象

+
+
开放文本生成

给 prompt,看最终视频是否语义正确、物理合理。

+
真实观测续写

给初始帧或前缀,与真实物理实验的未来比较。

+
轨迹与规律

提取运动,计算方程残差、守恒量或不变量。

+
模型内部偏好

比较模型对合理 / 不合理视频分配的概率。

+
评价器能力

先验证 VLM 或自动 metric 是否能识别真实物理。

+
+
一条排行榜无法同时回答“输出好不好”“模型懂不懂”“评价器准不准”。
+ + +
+
+ +
+
+ 26 +

Open-world judgement

+

开放世界 benchmark:覆盖广,但依赖判断者

+
+
+

VideoPhy → VideoPhy-2

+

先拆语义遵从与物理常识;再扩展到动作中心、细粒度规则和难例。

+
+
+

PhyGenBench / Eval

+

覆盖力学、光学、热学与材料属性;以关键现象、时序和整体自然度分层判断。

+
+
+

PhyWorldBench

+

从基础运动到复合、人 / 动物与 anti-physics 提示,扩大开放场景覆盖。

+
+
+
+
目标事件发生?semantic adherence
+
→
+
看起来物理合理?human / VLM judgement
+
→
+
违反哪条规则?diagnostic questions
+
+
共同盲点:画面质量可能干扰物理判断;VLM 可能合理化错误;开放提示通常缺少可测初态与真值。
+ + +
+
+ +
+
+ 27 +

Three different questions

+

从真实未来,到物理规律,再到模型内部

+
+
+ REFERENCE FUTURE +

Physics-IQ

+

受控真实实验;给初始观测,比较生成未来与真实重复实验的运动区域和演化。

+
问:预测是否接近这个真实系统的未来?
+
+
+ LAW RESIDUAL +

Morpheus

+

从视频提取二维轨迹,以动力方程和能量、加速度、动量等不变量评分。

+
问:即使未来不同,它是否仍满足对应规律?
+
+
+ LIKELIHOOD PROBE +

LikePhys

+

构造视觉匹配的 valid / invalid 视频对,比较扩散模型自身的 likelihood preference。

+
问:模型内部是否给合理视频更高概率?
+
+
+
越接近定量验证,条件通常越受控;越接近开放生成,物理真值越难确定。
+ + +
+
+ +
+
+ 28 +

Benchmark audit

+

benchmark 本身也需要审计

+
+
+
+
原始 benchmark提示、掩码、计分规则
+
→
+
逐样本审计混淆因素与错误真值
+
→
+
Verified version修正提示、样本与聚合
+
+
+

57.6%

samples 涉及至少一种潜在测量错误

+

69 / 198

prompts 不清楚(34.8%)

+

τ = 0.46

Original / Verified 六模型排名一致性

+
+
Physics-IQ Verified 的提醒:prompt、GT activation、计分与聚合都改变后,两版分数不再是同一把尺;只能审慎比较排名重排。
+
+
+
循环验证训练 reward 与最终 evaluator 是否同一个 VLM?avoid
+
sanity check真实视频与已知反例能否得到合理分数?required
+
随机性报告多 seed,而不是只展示 best-of-N。N + cost
+
错误分解语义失败、视觉失败与物理失败分开。separate
+
证据组合开放域判断 + 可测规律 + 人评。triangulate
+
+
+ + +
+
+ +
+
+ 29 +

Takeaway & discussion

+

我们真正缺少什么?

+
+ 结论:外部模拟、显式控制、训练内化和推理优化将长期并存。尚未解决的是:怎样让模型获得能跨场景泛化、响应物理变量变化,并长期稳定的物理机制。 +
+
+
+
01
+

物理状态应该放在哪里?

+

作为可解释的显式条件,还是隐含在生成模型 latent 中?

+ representation +
+
+
02
+

谁获得了能力?

+

inference-time 搜索生成更合理的视频,应该算模型能力还是系统能力?

+ model vs system +
+
+
03
+

怎样证明学到机制?

+

能否固定场景,只改变 force、mass、friction,检验结果是否按规律变化?

+ controlled intervention +
+
+ + +
+
+
+
← → / Space 翻页 · O 总览 · N 讲稿 · F 全屏 · 虚线术语可悬停
+
+ + + + + +
+
+ + + + + diff --git a/discussions/2026-08-24-yuxin-physics-consistent-video-generation/references.html b/discussions/2026-08-24-yuxin-physics-consistent-video-generation/references.html new file mode 100644 index 0000000..5d6f950 --- /dev/null +++ b/discussions/2026-08-24-yuxin-physics-consistent-video-generation/references.html @@ -0,0 +1,75 @@ + + + + + + + Key References · Physics-Consistent Video Generation: Who Teaches the Generator Physics? · CausalVideoGroup + + + + +
+ +

These six papers anchor the talk's main routes: explicit simulation, post-training, representation alignment, verifiable reward, inference-time alignment, and law-level evaluation.

+

1. PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation

+ +

2. What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewards

+ +

3. PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Drop

+ +

4. VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models

+ +

5. Inference-time Physics Alignment of Video Generative Models with Latent World Models

+ +

6. Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systems

+
+
+ + + diff --git a/discussions/2026-08-24-yuxin-physics-consistent-video-generation/references.md b/discussions/2026-08-24-yuxin-physics-consistent-video-generation/references.md new file mode 100644 index 0000000..cba916c --- /dev/null +++ b/discussions/2026-08-24-yuxin-physics-consistent-video-generation/references.md @@ -0,0 +1,57 @@ +# Key References + +These six papers anchor the talk's main routes: explicit simulation, post-training, representation alignment, verifiable reward, inference-time alignment, and law-level evaluation. + +## 1. PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation + +- Link: https://arxiv.org/abs/2409.18964 +- Type: system +- Why important: It connects single-image understanding, rigid-body simulation, and diffusion refinement. +- Role in this topic: Representative of computing physical evolution outside the video generator. +- Main limitation: Hidden geometry, material, contact, and force are not uniquely recoverable from one image. +- Discussion question: How should the system represent multiple plausible initial states? + +## 2. What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewards + +- Link: https://arxiv.org/abs/2512.00425 +- Type: method +- Why important: It converts measurable motion proxies and Newtonian structure into post-training rewards. +- Role in this topic: Representative of moving from general preference feedback toward verifiable physical constraints. +- Main limitation: Reliable measurement requires controlled phenomena, so coverage narrows as verification becomes stricter. +- Discussion question: Which additional laws can become reliable rewards without sacrificing broad scene coverage? + +## 3. PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Drop + +- Link: https://arxiv.org/abs/2503.09595 +- Type: method +- Why important: Controlled free fall reveals both post-training gains and weak extrapolation. +- Role in this topic: Diagnostic anchor for separating behavior teaching from rule learning. +- Main limitation: The model may fit falling-video statistics without learning a transferable gravity mechanism. +- Discussion question: Which split best rules out memorized falling templates? + +## 4. VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models + +- Link: https://arxiv.org/abs/2505.23656 +- Type: method +- Why important: It distills token relations from a video-understanding model into a generator. +- Role in this topic: Representative of using learned representations as a physics teacher. +- Main limitation: Teacher features may encode visual-motion statistics rather than intervenable physical state. +- Discussion question: Which counterfactual test can separate physics from visual correlation? + +## 5. Inference-time Physics Alignment of Video Generative Models with Latent World Models + +- Link: https://arxiv.org/abs/2601.10553 +- Type: system +- Why important: WMReward scores candidate trajectories with a frozen latent world model. +- Role in this topic: Representative of improving outputs through inference without updating generator weights. +- Main limitation: Better candidates must already exist, the proxy can fail, and search adds cost. +- Discussion question: How should generator, verifier, and test-time-compute gains be separated? + +## 6. Evaluating Newtonian Mechanics in Video Generative Models with Real Physical Systems + +- Link: https://arxiv.org/abs/2504.02918 +- Type: benchmark +- Why important: Morpheus checks extracted trajectories against equations and physical invariants. +- Role in this topic: Representative of moving from plausibility judgments to law-level measurement. +- Main limitation: Controlled rigid-body scenes and trajectory-extraction error restrict coverage. +- Discussion question: Which non-rigid phenomena admit useful measurable invariants? diff --git a/discussions/2026-08-24-yuxin-physics-consistent-video-generation/summary.html b/discussions/2026-08-24-yuxin-physics-consistent-video-generation/summary.html new file mode 100644 index 0000000..25288ed --- /dev/null +++ b/discussions/2026-08-24-yuxin-physics-consistent-video-generation/summary.html @@ -0,0 +1,79 @@ + + + + + + + Physics-Consistent Video Generation · Topic Summary + + + + +
+ +
+

Topic overview

+

Physics consistency means that motion, interaction, and state change in a generated video follow real-world physical laws. Temporal smoothness, appearance continuity, natural motion, and high visual quality are useful prerequisites, but they do not by themselves establish correct gravity, contact, material response, conservation, or causal order.

+ +

Why now?

+

Recent work has moved beyond the simple recipe of letting a simulator compute motion and a generator render it. Physical information may come from simulation, real video, explicit properties, foundation-model features, VLM feedback, measurable laws, or the generator's own latent states. It may enter during planning, training, conditioning, sampling, or evaluation. Benchmarks have expanded at the same time, so “better physics” can no longer be summarized by one score.

+ +

Background: three distinctions

+
    +
  1. Source × stage: simulators, data, VLMs, learned representations, and physical laws can enter before generation, during training, as explicit conditions, or at inference time.
  2. +
  3. Model × system: updating the video backbone, training an auxiliary dynamics or reward model, and adding test-time search are distinct interventions.
  4. +
  5. Motion outcome × physical condition: masks, flow, and trajectories often specify how an object moves; force, mass, friction, restitution, and material properties ask the system to infer the resulting motion.
  6. +
+ +

Method landscape

+
    +
  1. Explicit dynamics and control: PhysGen and related systems infer state, simulate motion, then use a video model for rendering or refinement. Their constraints are interpretable, but hidden geometry, material, contact, and force are often underdetermined.
  2. +
  3. Data, representation, and reward: PISA teaches a controlled behavior through post-training; VideoREPA borrows spatiotemporal relations from a video encoder; VLM or law-based rewards optimize generated motion. These methods change the generator, but may still learn templates or evaluator shortcuts.
  4. +
  5. Latent dynamics and inference-time alignment: some methods add physical state to the prediction target, while others use a frozen world model, intermediate features, or early motion priors to search, select, or refine samples without updating the generator.
  6. +
  7. Evaluation: open-ended judges offer breadth, real futures provide controlled evidence, equations test measurable laws, and likelihood preference probes internal model ranking. These protocols answer different questions.
  8. +
+ +

Representative evidence and its boundary

+

PhysGen shows that explicit state estimation and simulation can guide an image-to-video pipeline, but the guarantee belongs to the full pipeline and depends on the inferred state. PISA shows that small amounts of simulated data and reward can change free-fall behavior while exposing weak out-of-distribution generalization. VideoREPA improves physics benchmarks through representation alignment, yet assumes that the teacher features encode transferable physics rather than stronger visual statistics. Verifiable rewards are easier to audit but cover fewer phenomena; inference-time alignment can find better samples without proving that the generator learned new knowledge.

+ +

Failure cases

+

Current methods degrade with hidden initial state, contact-rich interaction, material change, deformation, long horizons, unseen parameters, and combinations of physical rules. Explicit pipelines inherit state-estimation errors; control may be confused with physical inference; reward optimization may exploit an evaluator; and inference-time search only works when the candidate distribution already contains a better trajectory.

+ +

Connections to our research

+

The topic connects long-video generation, causal and autoregressive modeling, controllable generation, world models, and video evaluation. Long rollouts amplify small dynamics errors, while shared reward and evaluation models can create circular evidence.

+ +

Research-quality prompts

+
Central claim: Progress comes from better interfaces that turn physics into controllable and testable supervision, not from one route replacing all others.
+
Hidden assumption: Trajectories, flow, VLM scores, learned features, and video likelihood can separate physical mechanisms from visual correlation.
+
Failure boundary: Evidence remains strongest for short, observable, single-event, and controlled phenomena.
+
Connection: Every claim should identify what changed in the generator, auxiliary modules, and inference system.
+
Concrete experiment: Fix the backbone, data, and compute, vary one supervision interface, then test unseen parameters, appearances, rule combinations, and longer horizons.
+
Discussion seed: If inference-time search reliably finds better physics, is that model capability, latent capability in the distribution, or system capability?
+ +

Open questions

+
    +
  1. Which parts of geometry, material, contact, and force are identifiable from one image?
  2. +
  3. How can evaluation separate following a supplied trajectory from inferring motion from physical conditions?
  4. +
  5. Should physical state be explicit and writable, or remain inside a generator's latent representation?
  6. +
  7. What experiment would show that latent dynamics are stable and intervenable rather than merely benchmark-useful?
  8. +
  9. How can training rewards and final evaluators remain independent enough to avoid circular validation?
  10. +
  11. How should a benchmark represent multiple physically valid futures and long-horizon error accumulation?
  12. +
+ +

Possible project

+

Build a small counterfactual benchmark that fixes scene and appearance while changing one physical variable at a time. Compare simulated-data fine-tuning, representation alignment, preference optimization, verifiable reward, and inference-time search on the same backbone, and report both law-level error and system cost.

+
+
+ + + diff --git a/discussions/index.html b/discussions/index.html index b674670..83bab80 100644 --- a/discussions/index.html +++ b/discussions/index.html @@ -2,7 +2,8 @@
-

2026-08-10 · Yichang Jian

From Video Scores to World Consistency

A data-backed discussion of how video-generation evaluation moves from perceptual proxy scores toward geometric, revisit, and out-of-sight world consistency.

before-meeting video-generation world-model benchmark evaluation

Read discussion →
+

2026-08-24 · Yuxin Wang

Physics-Consistent Video Generation: Who Teaches the Generator Physics?

A design-space view of how simulators, explicit controls, training signals, latent dynamics, and inference-time alignment can make generated videos obey real-world physical laws, and what current benchmarks can actually establish.

before-meeting video-generation physics-consistency world-model controllable-generation benchmark evaluation

Read discussion →
+

2026-08-10 · Yichang Jian

From Video Scores to World Consistency

A data-backed discussion of how video-generation evaluation moves from perceptual proxy scores toward geometric, revisit, and out-of-sight world consistency.

before-meeting video-generation world-model benchmark evaluation

Read discussion →

2026-07-20 · Jinyuan Hu

Multi-Shot Video Generation: From Joint Denoising to Online Directing

A technical map of multi-shot video generation, organized by how shots are generated and how history enters the model: holistic joint denoising, shot-level autoregression, causal streaming, and system-level directing.

before-meeting video-generation multi-shot-video long-video-generation causal-generation agentic-video

Read discussion →

2026-07-13 · Yifan Shen

Forcing Series: The Evolution of AR Video Distillation

A technical map of methods that distill bidirectional video diffusion models into few-step autoregressive generators that produce frames in temporal order and support streaming output.

before-meeting video-generation autoregressive-video video-distillation streaming-generation

Read discussion →
diff --git a/index.html b/index.html index 6378ab4..78c23cb 100644 --- a/index.html +++ b/index.html @@ -30,9 +30,9 @@

Video generation from a causal perspective.

Weekly reading group

Topic discussions

Each entry prepares the necessary background before the meeting, then records the assumptions, failure cases, and experiments identified during discussion.

-

2026-08-10 · Yichang Jian

From Video Scores to World Consistency

A data-backed discussion of how video-generation evaluation moves from perceptual proxy scores toward geometric, revisit, and out-of-sight world consistency.

before-meeting video-generation world-model benchmark evaluation

Read discussion →
-

2026-07-20 · Jinyuan Hu

Multi-Shot Video Generation: From Joint Denoising to Online Directing

A technical map of multi-shot video generation, organized by how shots are generated and how history enters the model: holistic joint denoising, shot-level autoregression, causal streaming, and system-level directing.

before-meeting video-generation multi-shot-video long-video-generation causal-generation agentic-video

Read discussion →
-

2026-07-13 · Yifan Shen

Forcing Series: The Evolution of AR Video Distillation

A technical map of methods that distill bidirectional video diffusion models into few-step autoregressive generators that produce frames in temporal order and support streaming output.

before-meeting video-generation autoregressive-video video-distillation streaming-generation

Read discussion →
+

2026-08-24 · Yuxin Wang

Physics-Consistent Video Generation: Who Teaches the Generator Physics?

A design-space view of how simulators, explicit controls, training signals, latent dynamics, and inference-time alignment can make generated videos obey real-world physical laws, and what current benchmarks can actually establish.

before-meeting video-generation physics-consistency world-model controllable-generation benchmark evaluation

Read discussion →
+

2026-08-10 · Yichang Jian

From Video Scores to World Consistency

A data-backed discussion of how video-generation evaluation moves from perceptual proxy scores toward geometric, revisit, and out-of-sight world consistency.

before-meeting video-generation world-model benchmark evaluation

Read discussion →
+

2026-07-20 · Jinyuan Hu

Multi-Shot Video Generation: From Joint Denoising to Online Directing

A technical map of multi-shot video generation, organized by how shots are generated and how history enters the model: holistic joint denoising, shot-level autoregression, causal streaming, and system-level directing.

before-meeting video-generation multi-shot-video long-video-generation causal-generation agentic-video

Read discussion →
diff --git a/people/index.html b/people/index.html index 0d3d50d..919a29f 100644 --- a/people/index.html +++ b/people/index.html @@ -5,7 +5,7 @@

Group Leader · guangyi

Guangyi Chen

Discussion Leaders

Discussion Leader · peiyuan

Peiyuan Zhu

No published projects or discussions yet.

-

Discussion Leader · yuxin

Yuxin Wang

No published projects or discussions yet.

+
diff --git a/sitemap.xml b/sitemap.xml index 06ae613..35e66d3 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -8,9 +8,13 @@ https://causalvideogroup.github.io/discussions/ https://causalvideogroup.github.io/people/ https://causalvideogroup.github.io/tags/ + https://causalvideogroup.github.io/discussions/2026-08-24-yuxin-physics-consistent-video-generation/ https://causalvideogroup.github.io/discussions/2026-08-10-yichang-video-generation-benchmark/ https://causalvideogroup.github.io/discussions/2026-07-20-jinyuan-multi-shot-video-generation/ https://causalvideogroup.github.io/discussions/2026-07-13-yifan-forcing-ar-video-distillation/ + https://causalvideogroup.github.io/discussions/2026-08-24-yuxin-physics-consistent-video-generation/summary.html + https://causalvideogroup.github.io/discussions/2026-08-24-yuxin-physics-consistent-video-generation/references.html + https://causalvideogroup.github.io/discussions/2026-08-24-yuxin-physics-consistent-video-generation/meeting-note.html https://causalvideogroup.github.io/discussions/2026-08-10-yichang-video-generation-benchmark/summary.html https://causalvideogroup.github.io/discussions/2026-08-10-yichang-video-generation-benchmark/references.html https://causalvideogroup.github.io/discussions/2026-08-10-yichang-video-generation-benchmark/meeting-note.html diff --git a/tags/index.html b/tags/index.html index a6f1bfd..b624d60 100644 --- a/tags/index.html +++ b/tags/index.html @@ -4,14 +4,16 @@

1 discussion(s) · 0 project(s)

agentic-video

1 discussion(s) · 0 project(s)

autoregressive-video

-

1 discussion(s) · 0 project(s)

benchmark

+

2 discussion(s) · 0 project(s)

benchmark

1 discussion(s) · 0 project(s)

causal-generation

-

1 discussion(s) · 0 project(s)

evaluation

+

1 discussion(s) · 0 project(s)

controllable-generation

+

2 discussion(s) · 0 project(s)

evaluation

1 discussion(s) · 0 project(s)

long-video-generation

1 discussion(s) · 0 project(s)

multi-shot-video

+

1 discussion(s) · 0 project(s)

physics-consistency

1 discussion(s) · 0 project(s)

streaming-generation

1 discussion(s) · 0 project(s)

video-distillation

-

3 discussion(s) · 0 project(s)

video-generation

-

1 discussion(s) · 0 project(s)

world-model

+

4 discussion(s) · 0 project(s)

video-generation

+

2 discussion(s) · 0 project(s)

world-model