diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/action-items.md b/discussions/2026-09-28-peiyuan-video-editing-frontiers/action-items.md new file mode 100644 index 0000000..bb47710 --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/action-items.md @@ -0,0 +1,3 @@ +# Action items + +No post-meeting tasks have been assigned. Candidate experiments and open questions are documented in the [topic summary](summary.html). diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/.gitkeep b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/bernini.png b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/bernini.png new file mode 100644 index 0000000..8a7cbc7 Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/bernini.png differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/coinve-framework.png b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/coinve-framework.png new file mode 100644 index 0000000..1bb115a Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/coinve-framework.png differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/dog_bone_prediction.jpg b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/dog_bone_prediction.jpg new file mode 100644 index 0000000..1102454 Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/dog_bone_prediction.jpg differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/dog_bone_prediction.mp4 b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/dog_bone_prediction.mp4 new file mode 100644 index 0000000..5d90494 Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/dog_bone_prediction.mp4 differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/dog_bone_source.jpg b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/dog_bone_source.jpg new file mode 100644 index 0000000..50c3bad Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/dog_bone_source.jpg differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/dog_bone_source.mp4 b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/dog_bone_source.mp4 new file mode 100644 index 0000000..36be9a9 Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/dog_bone_source.mp4 differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/dog_bone_target.jpg b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/dog_bone_target.jpg new file mode 100644 index 0000000..d2027d2 Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/dog_bone_target.jpg differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/dog_bone_target.mp4 b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/dog_bone_target.mp4 new file mode 100644 index 0000000..70eece1 Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/dog_bone_target.mp4 differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/dynaedit.png b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/dynaedit.png new file mode 100644 index 0000000..0a9becc Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/dynaedit.png differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/gemini_omni_substitution_edited.jpg b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/gemini_omni_substitution_edited.jpg new file mode 100644 index 0000000..6f00357 Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/gemini_omni_substitution_edited.jpg differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/gemini_omni_substitution_edited.mp4 b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/gemini_omni_substitution_edited.mp4 new file mode 100644 index 0000000..551513e Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/gemini_omni_substitution_edited.mp4 differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/gemini_omni_substitution_source.jpg b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/gemini_omni_substitution_source.jpg new file mode 100644 index 0000000..ba8c5ad Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/gemini_omni_substitution_source.jpg differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/gemini_omni_substitution_source.mp4 b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/gemini_omni_substitution_source.mp4 new file mode 100644 index 0000000..1d22e49 Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/gemini_omni_substitution_source.mp4 differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/goku.png b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/goku.png new file mode 100644 index 0000000..b2b9432 Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/goku.png differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/joyai-video-edit.png b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/joyai-video-edit.png new file mode 100644 index 0000000..e900b80 Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/joyai-video-edit.png differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/bernini-joint.svg b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/bernini-joint.svg new file mode 100644 index 0000000..689c03f --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/bernini-joint.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/bernini-planner.svg b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/bernini-planner.svg new file mode 100644 index 0000000..046f3b1 --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/bernini-planner.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/bernini-renderer.svg b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/bernini-renderer.svg new file mode 100644 index 0000000..5308090 --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/bernini-renderer.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/flow-init.svg b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/flow-init.svg new file mode 100644 index 0000000..60131c3 --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/flow-init.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/flow-source.svg b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/flow-source.svg new file mode 100644 index 0000000..089bed9 --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/flow-source.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/flow-target.svg b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/flow-target.svg new file mode 100644 index 0000000..e83fe38 --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/flow-target.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/flow-update.svg b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/flow-update.svg new file mode 100644 index 0000000..0b23203 --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/flow-update.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/flow-velocity.svg b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/flow-velocity.svg new file mode 100644 index 0000000..df6a541 --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/flow-velocity.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/sga-aggregate.svg b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/sga-aggregate.svg new file mode 100644 index 0000000..d795f25 --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/sga-aggregate.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/sga-project.svg b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/sga-project.svg new file mode 100644 index 0000000..e6a8481 --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/sga-project.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/sga-weight.svg b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/sga-weight.svg new file mode 100644 index 0000000..ce17433 --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/math/sga-weight.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/motionv2v.png b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/motionv2v.png new file mode 100644 index 0000000..0900e76 Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/motionv2v.png differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/mse-framework.png b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/mse-framework.png new file mode 100644 index 0000000..9680d35 Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/mse-framework.png differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_ours_t1.jpg b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_ours_t1.jpg new file mode 100644 index 0000000..8b36b25 Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_ours_t1.jpg differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_ours_t1.mp4 b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_ours_t1.mp4 new file mode 100644 index 0000000..961cc80 Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_ours_t1.mp4 differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_ours_t2.jpg b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_ours_t2.jpg new file mode 100644 index 0000000..d019428 Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_ours_t2.jpg differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_ours_t2.mp4 b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_ours_t2.mp4 new file mode 100644 index 0000000..eb8b97e Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_ours_t2.mp4 differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_ours_t3.jpg b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_ours_t3.jpg new file mode 100644 index 0000000..acb311f Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_ours_t3.jpg differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_ours_t3.mp4 b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_ours_t3.mp4 new file mode 100644 index 0000000..8407a09 Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_ours_t3.mp4 differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_source.jpg b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_source.jpg new file mode 100644 index 0000000..a574f94 Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_source.jpg differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_source.mp4 b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_source.mp4 new file mode 100644 index 0000000..423ffcf Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/multiturn_clothing_source.mp4 differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/propfly.png b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/propfly.png new file mode 100644 index 0000000..558165f Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/propfly.png differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/void.png b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/void.png new file mode 100644 index 0000000..3c9e155 Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/void.png differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/world-in-world.png b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/world-in-world.png new file mode 100644 index 0000000..4073f38 Binary files /dev/null and b/discussions/2026-09-28-peiyuan-video-editing-frontiers/assets/world-in-world.png differ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/evidence.html b/discussions/2026-09-28-peiyuan-video-editing-frontiers/evidence.html new file mode 100644 index 0000000..8163fc2 --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/evidence.html @@ -0,0 +1,19 @@ + +Examples & evaluation notes · CausalVideoGroup +
CausalVideoGroup
+

2026-09-28 · Peiyuan Zhu

Examples & evaluation notes

+

Qualitative editing examples

+

Action substitution

The requested change replaces walking toward the door with drinking water. In the shown Gemini Omni example, drinking is added near the end while much of the walking remains. Check both the presence of the target event and removal of the source event. View the example.

+

Clothing edits across turns

The sequence edits clothing colors, preserves the edits in the next turn, then restores both colors. It does not test undoing only one change while preserving another. View the sequence.

+

Dog–bone interaction

The source contains a yellow-brown bone on the ground. The archived target has a differently shaped white bone in the mouth from the first frame, so it does not demonstrate the original bone being picked up. The Gemini Omni Flash output leaves the original bone on the ground and changes the dog's coat. Model attribution follows the presenter's identification of the result. These three clips belong to one example; they are not three independent trials. View the comparison.

+

These examples diagnose event timing, object continuity, interaction execution, and preservation. They do not establish overall model rankings or failure rates.

+

Evaluation snapshot — 2026-09-26

The three results shown in the deck use different metrics and protocols. Each is a diagnostic of the judge, and their scales must not be combined.

+ + + + +
SubsetJudge and protocolNo-edit scoreInterpretation
VEFX Instance Motion33VEFX-Reward-4B, official duration-dependent sampling at 4 frames/s; IF, 1–42.7089Above the tested editing outputs under this protocol.
Omni Temporal-200Qwen3-VL-32B with the Omni rubric and all decoded video frames; Overall, 0–10033.340Above editing outputs with results available under the full-video protocol.
IVEBench Subject Motion Short64Qwen2.5-VL-72B with official IS preprocessing; IS, 1–51.6875This setup gives the unedited output a low instruction score.
+

Source-as-output deliberately performs no edit and is excluded from the editor ranking. High preservation or smoothness cannot establish action completion. In the Omni sensitivity study, switching the same Qwen3-VL-32B judge from 12 sampled frames to all decoded frames changes the no-edit Overall score from 17.608 to 33.340. The direction of change also depends on judge size, so more frames are not automatically a more accurate evaluation.

+

Local configuration names

FM-only denotes a Bernini-R 1.3B adapter trained with direct flow-matching supervision and source-plus-instruction inference. Zebra denotes a reference-conditioned controller whose pipeline additionally generates a pseudo-reference. Neither name denotes an independently published paper. Their extra inputs and generation cost must be reported when comparing results; neither has a full-video record in the frozen Omni comparison used here.

+

These values are transcribed from the presenter's frozen 2026-09-26 aggregate report used to prepare the deck. The public page documents the displayed results and protocol limits; it is not a raw benchmark release or a new evaluation run.

+ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/idea-map.md b/discussions/2026-09-28-peiyuan-video-editing-frontiers/idea-map.md new file mode 100644 index 0000000..ecedec3 --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/idea-map.md @@ -0,0 +1,10 @@ +# Video Editing Frontiers — Idea Map + +- Editing mechanisms: inference transport, instruction training, semantic planning and rendering. +- Controllable changes: local attributes, composition, motion, object interaction, counterfactual effects. +- Persistent editing: long video, multiple shots, history, instruction updates and undo. +- World models: state recovery, interventions, consequences and prediction feedback. +- Agents: executable plans, tools, verification, repair and fixed-budget evaluation. +- Evidence: source-object identity, event completion, non-target preservation and calibrated judges. + +The dog–bone example connects object identity and contact to state recovery and judge validity. Multi-turn editing connects versioned state to agent rollback and session-level evaluation. diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/index.html b/discussions/2026-09-28-peiyuan-video-editing-frontiers/index.html new file mode 100644 index 0000000..83c7adb --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/index.html @@ -0,0 +1,11 @@ + +Video Editing Frontiers · CausalVideoGroup +
CausalVideoGroup
+

2026-09-28 · Peiyuan Zhu

Video Editing Frontiers

+

Editing mechanisms, controllable changes, and persistent video editing.

+

Led by Peiyuan Zhu. A bilingual, 46-slide survey with dedicated method explanations, paper figures, playable examples, and technical hover notes.

+
Open presentationRead topic summaryReferencesExamples & evaluation notes
+

Discussion focus

How can a video editor change an event while keeping the intended source objects and unrelated content? The discussion connects sampling and learning mechanisms to object interaction, edit history, world models, agents, and reliable evaluation.

+

Open problems · Dog–object interaction · World models · Editing agents

+

Use arrow keys to navigate. Hover over underlined terms for technical explanations; paper names link to their sources. Videos can play together on each example slide. Meeting notes will be added after the discussion.

+ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/meeting-note.html b/discussions/2026-09-28-peiyuan-video-editing-frontiers/meeting-note.html new file mode 100644 index 0000000..b8e7179 --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/meeting-note.html @@ -0,0 +1,26 @@ + + + + + + + Meeting note · Video Editing Frontiers · CausalVideoGroup + + + +
+ CausalVideoGroup + +
+
+

2026-09-28 · Peiyuan Zhu

Meeting note

Video Editing Frontiers

+

This page contains the pre-meeting presentation. Discussion outcomes have not yet been recorded.

+
+
+ + + diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/meeting-note.md b/discussions/2026-09-28-peiyuan-video-editing-frontiers/meeting-note.md new file mode 100644 index 0000000..db993d6 --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/meeting-note.md @@ -0,0 +1,7 @@ +# Meeting note + +This page contains the pre-meeting presentation. Discussion outcomes have not yet been recorded. + +- [Presentation](presentation.html) +- [Topic summary](summary.html) +- [References](references.html) diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/metadata.yaml b/discussions/2026-09-28-peiyuan-video-editing-frontiers/metadata.yaml new file mode 100644 index 0000000..0145e31 --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/metadata.yaml @@ -0,0 +1,39 @@ +title: "Video Editing Frontiers" +date: 2026-09-28 +topic_slug: video-editing-frontiers +leader: + name: "Peiyuan Zhu" + short_name: peiyuan +status: before-meeting +summary: >- + A survey of video editing mechanisms and controllable changes, with method + deep dives, interactive examples, and open problems in object interaction, + editing history, world models, agents, and evaluation. +tags: + - video-editing + - action-editing + - world-model + - agents + - benchmark +related_projects: [] +papers: + - title: "FlowEdit (ICCV 2025)" + url: https://arxiv.org/pdf/2412.08629 + role: method + - title: "DynaEdit" + url: https://arxiv.org/pdf/2603.17989 + role: method + - title: "Bernini" + url: https://arxiv.org/pdf/2605.22344 + role: method + - title: "World in World" + url: https://arxiv.org/pdf/2609.11548 + role: method + - title: "MMLVE-Agent" + url: https://arxiv.org/pdf/2608.26809 + role: system + - title: "CoVEBench" + url: https://arxiv.org/pdf/2606.08415 + role: benchmark +external_videos: [] +slides: presentation.html diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/presentation.html b/discussions/2026-09-28-peiyuan-video-editing-frontiers/presentation.html new file mode 100644 index 0000000..ceaa30c --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/presentation.html @@ -0,0 +1,224 @@ +Video Editing Frontiers · Peiyuan Zhu · CausalVideoGroup +
VIDEO EDITING
Video Editing Frontiers
Source video + instruction + controls
目标变化 / 非目标保持 / 时间一致性
Editing mechanisms
FlowEdit · DynaEdit +SAMA · Bernini · TIDE
Sampling, learning & representation
Controllable changes
Local · motion · physics +World models
Region, dynamics & interaction
Persistent editing
Long video · multi-shot +Multi-turn · agents
State, memory & feedback
DATA & SUPERVISION
BENCHMARKS & EVALUATION
+
TOPIC
VIDEO EDITING
What changes when we edit a video?
从修改内容到执行机制：同一任务可以沿不同技术路线实现。
Editing mechanisms
编辑能力怎样获得？
Sampling
/ FlowEdit / DynaEdit
Learning
SAMA / Bernini / TIDE
Transfer
Qwen-Video-Edit / EditVid
源轨迹复用 → 条件生成 → 图像先验迁移
Controllable changes
目标变化由什么表示？
Appearance
Mask / keyframe / reference
Dynamics
Action / point tracks / camera
Interaction
Object removal / physical intervention
区域绑定、运动重写与必要的连带变化
Persistent editing
修改跨越哪些边界？
Time
Long video / streaming
Shots
Cross-shot identity / references
Turns
Memory / overwrite / undo
从单段输出走向连续执行与历史管理
Data & evaluation 配对是否只改变目标？评测是否能识别未执行、串扰和历史漂移？
02
+
TOPIC
VIDEO EDITING
Change, ,
输入条件决定编辑难度；目标修改决定哪些内容应该保留。
01
Change
Appearance：换衣服、替换对象 +Motion：改变动作或相机轨迹 +Interaction：删除对象及其后果
“把红衣改黑”与“取消接球者”需要不同的变化范 +围。
02
Identity：仍是同一个人或对象 +Context：未指定的背景与事件 +History：前几轮仍有效的修改
删除接球者后，球继续按原轨迹运动，可能反而 +是不正确的保持。
03
Text：语义目标 +Mask / reference：范围与外观 +Keyframe / tracks：像素与路径
目标首帧、逐帧 mask、target tracks 都向系统 +提供了额外答案。
INSIGHT
比较模型时固定输入接口，并分别检查 edit success 与 non-target preservation。
VACE · Kiwi-Edit · MotionV2V · JAVEdit
03
+
TOPIC
I / MODELS & PRIORS / TRAINING-FREE EDITING
Training-free editing
Source coupling 的位置决定了哪些结构容易被保留，哪些结构能够被重写。
MECHANISM / METHOD
SOURCE EVIDENCE
INFERENCE / LEARNING
& feature reuse
FateZero · TokenFlow
source trajectory
→
attention / feature injection
重建偏差会传到目标视频；强复用限制几何变化。
FlowEdit → DynaEdit
→
target − source
通过差分方向修改源状态，采样稳定性成为关键。
Keyframe &
AnyV2V · EditVid
edited keyframe / matches
→
cross-frame propagation
依赖跨帧对应，遮挡与新显露区域最容易失效。
Shared backbone
无需训练编辑器；仍调用已经训练的图像或视频生成模型。
Compute
反演步数、source / target 双分支、候选数和外部首帧编辑都影响成本。
INSIGHT
主要 trade-off：source preservation 与允许重写的时空结构。
FateZero · TokenFlow · AnyV2V · DynaEdit · EditVid · et al.
04
+
METHOD
I / MODELS & PRIORS / TRAINING-FREE EDITING
FlowEdit
-free editing：直接构造 source → target 的传输路径。
EDIT ODE
Source query
Target query
Editing direction
State update
主编辑状态由完整源视频初始化。每一步累加目标与 +源条件的 。
两次 query 共享噪声残差，减少条件差分中混入的随 +机变化。
Video adaptation
原工作用 SD3 / FLUX 做图像编辑；DynaEdit 换成 +Wan ，在完整视频 上运算。
State vs. query
主状态代表正在修改的内容；source / target query 是为了调用 flow model +构造的辅助带噪状态。
Edit strength
提高最高噪声时刻可释放更大的结构变化，也更容易丢失源布局。
INSIGHT
FlowEdit 提供编辑 ODE；视频时间先验来自替换后的 video backbone。
FlowEdit (ICCV 2025) · DynaEdit
05
+
METHOD
I / MODELS & PRIORS / TRAINING-FREE EDITING
DynaEdit
把 FlowEdit 推到最大噪声查询区间，让动作与动态事件可以被改写。
dynaedit.png
早期生成多个候选方向，外推 clean edit，再用 source +similarity 选择更对齐的结构。
后期逐步增强 之间的噪声相关性，减少独立噪声 +造成的抖动。
Wan2.1 -14B 读取源 / 目标首帧。新对象和外观目标常由外 +部图像编辑器提供。
Backbone
Frozen Wan2.1 -14B
Training
None; inference-time sampling
Input
Full source + captions + first frames
INSIGHT
选择早期粗结构， 稳定后期采样；两者作用在不同阶段。
DynaEdit
06
+
METHOD
I / MODELS & PRIORS / TRAINING-FREE EDITING
DynaEdit: candidate selection & noise correlation
截断噪声区间更容易保留源运动； 可改粗结构，但需要稳定早期路径与后期采样。
/ coarse structure
/ solver stability
候选间使用不同噪声；每个候选内部的 source / target query 共享噪声。论文 +配置在最早 3 步用 5 个候选，之后回到 1 个。
Early steps
弱相关 / 更多探索
候选可以形成不同动作、相机与布局。
Late steps
更强相关 / 更少抖动
降低每步重采样噪声引入的高频不稳定。
相关性沿 调度。固定所有步骤的噪声容易锁住早期错误 +结构。
Preservation boundary
全局 cosine similarity 是软偏好；没有 protected-region mask 或局部不变保 +证。
Cost & sensitivity
多候选增加前几步模型调用。、 temperature 与目标首帧都会影响结 +果。
INSIGHT
最大噪声 query 仍从源视频初始化主编辑状态；候选聚合约束方向，噪声相关性约束采样稳定性。
DynaEdit · FlowEdit (ICCV 2025)
07
+
TOPIC
I / MODELS & PRIORS / IMAGE-TO-VIDEO TRANSFER
Image-to-video transfer
Qwen-Video-Edit 训练视频接口；EditVid 在推理时连接冻结图像编辑器。
Qwen-Video-Edit
Train the temporal interface +Wan Video 将视频压为 + 适配 Qwen 图像 +先训 projections，再适配编辑主干
训练迁移：新的 video representation
EditVid
Reuse the frozen image editor +FLUX.2 Klein 保持冻结 + 连接邻近帧 + 补远程联系
推理迁移：新的 temporal coupling
Learned transfer
Qwen-Video-Edit 先训 projections，再以 Ditto-1M 等数据适配 。可选 +Wan enhancement 单独计入成本。
Inference transfer
EditVid 恢复 source trajectory； difference 产生 soft mask，融合编 +辑与源区域。
INSIGHT
两条路线都借用图像先验，但训练需求、时序约束和推理成本不同。
Qwen-Video-Edit · EditVid
08
+
TOPIC
I / MODELS & PRIORS / INSTRUCTION & UNIFIED MODELS
Learning an instruction editor
三个位置引入语义监督：与像素联合预测、先预测目标语义、或为输入赋予角色。
Method familySemantic pathwayVisual pathway
联合生成 semantic tokens +SigLIP target anchors 提供训练监督source / noisy target 同一 +恢复任务学习 motion prior
Bernini 预测 target features +作为连续 semantic plan / reference +Wan renderer 合成目标
Gemma 提供语义条件 + 区分 token 用途 条件保留视觉细节 +progressive multi-task training
Common structure
/ reference 提供证据，noisy target 学习目标条件分布。
Key distinction
学联合表示；Bernini 学连续 target plan； 统一多种任务与条件角 +色。
INSIGHT
这些模型都接受指令，但语义监督进入网络的位置、数据和训练对象不同。
· Bernini ·
09
+
METHOD
I / MODELS & PRIORS / INSTRUCTION & UNIFIED MODELS
SAMA
Semantic Anchoring & Motion Alignment：在同一个 中学习语义修改与视频恢复。
Semantic anchoring
64 local + 1 global tokens
Target anchor frame 的 SigLIP features 提 +供语义监督。
Motion alignment
Cube / speed / tube corruption
通过恢复视频学习 source motion prior。
Paired editing
Instruction triplets
Source + instruction + target
约 1.029M video-edit triplets 适配联合编 +辑。
Training targets
Stage 0 使用约 2.5M image-edit pairs，并加入 Cube Inpainting、Speed +Perturbation、Tube Shuffle。
Inference
从噪声开始，与 target video 联合生成。Target anchor +frame 只用于训练监督。
INSIGHT
Motion Alignment 通过恢复任务学习运动理解与保持；目标动作重写仍需单独验证。
SAMA
10
+
METHOD
I / MODELS & PRIORS / INSTRUCTION & UNIFIED MODELS
Bernini
semantic planning：先预测目标视觉语义，再用 source-conditioned diffusion 渲染。
bernini.png
Bernini · Figure 3 (A–C): inputs, semantic planner and renderer
从 source、instruction、references 预测 target + features，再得到上下文化 hidden states。
Condition interface
MLP connector 将 hidden states 接到 ；原 +生 T5 文本条件仍然保留。
/ reference features 保留视 +觉证据，只有 target 被逐步去噪。
INSIGHT
Source preservation 同时依赖 clean visual conditions 与配对监督。
Bernini
11
+
METHOD
I / MODELS & PRIORS / INSTRUCTION & UNIFIED MODELS
Bernini: how the and renderer are trained
先学语义转换，再学视频渲染，最后轻量联合优化。
I /
256p / 2 fps
更新 + decoder，保留语 +言与多模态理解。
II / renderer
480p / 16 fps
训练 renderer + lightweight text +encoder，后期偏向高质量 editing pairs。
III / joint
480p / 16 fps
轻量联合训练 与 renderer，后期加入 +CoT 数据。
Natural visual transitions
来自同一真实视频的两个 clips；近 30M image pairs 来自 +tutorial keyframes。它们提供变化先验。
Instruction-aligned refinement
传播式 V2V、motion-aware V2V 与 reference 数据补充编辑监督。Stage II +的相邻 pair 采样比例逐渐降到 0。
INSIGHT
相邻真实片段有助于学习变化；高质量 editing pairs 进一步约束指令与非目标保持。
Bernini
12
+
METHOD
I / MODELS & PRIORS / INSTRUCTION & UNIFIED MODELS
TIDE
用 统一 generation、editing 与 reference-conditioned video tasks。
Semantic condition
Text + visual evidence
→
Frozen Gemma3-12B-IT
→
Semantic tokens
解释指令与参考内容，提供跨模态语义。
Visual condition
Source / target / references
→
→
LTX-2.3 visual
Clean conditions 保留细节，noisy target 使用 训练。
Role-aware tokens
Source、target、edit reference、generation reference 使用不同角色和索 +引，降低条件用途混淆。
Training scope
完整训练 LTX-2.3 视觉 14B ，Gemma 冻结； + 统一任务接口。
INSIGHT
提供可学习的条件区分；仍需用具体任务检验保持与参考绑定。
TIDE
13
+
TOPIC
I / MODELS & PRIORS / INSTRUCTION & UNIFIED MODELS
Discrete editing & joint audio–visual editing
改变生成表示，或增加输出模态，会引入不同的训练目标与评价对象。
· discrete tokens
Binary token refinement +Stage I 学 source-conditioned identity +Stage II 用 路由修改
source evidence / bit correction
· audio + video
Joint denoising +LTX-2.3 联合处理视听 reference +任务涉及声音、画面及二者同步
visual compliance / audio / sync
supervision
约 0.6M pairs；Stage II 冻结 identity reference，用 correction router 分配 +bit 修改。
supervision
约 100K 视听样本、五类人物相关任务；画面、声音与同步分别评价。
INSIGHT
离散编辑改变采样表示；音视频编辑增加了需要同时满足的输出约束。
· · TIDE
14
+
TOPIC
II / TASKS & CONTROL / LOCAL & COMPOSITIONAL
Local & compositional control
空间范围、目标外观和操作绑定，分别由哪些条件提供？
Where
显式给定需要修改的像素区域
VACE / VOID
Edited keyframe
What it looks like
给定一个时刻的目标外观，再传播
AnyV2V / PropFly
Reference image
Which appearance
迁移身份、材质或风格等参照
TIDE / Bernini
Which operation goes where
多条指令分别绑定主体与区域
CoinVE
Region binding
预测错误的区域，会把正确的操作施加给错误的人或物体。
Temporal coverage
一张首帧无法约束后续才出现的对象；遮挡后重现需要跨时刻的对应信息。
INSIGHT
局部保持的关键是区域绑定，以及边界外的信息如何进入生成过程。
VACE · VOID · AnyV2V · PropFly · TIDE · et al.
15
+
METHOD
II / TASKS & CONTROL / LOCAL & COMPOSITIONAL
PropFly
：从 teacher 的 guidance 差异中学习外观传播。
01
Frozen teacher
同一 noisy input +低 / 高 估计
→
02
构造条件变化对应的 +one-step clean estimates
→
03
Trained propagator
VACE adapter 学习 +Guidance-Modulated FM
提供目标外观；完整源视频提供运动和场景。首帧通常由外部图像编辑器得 +到。
What is trained
冻结 Wan teacher，更新 VACE-style adapter 的条件投影和 attention 模 +块，目标为 Guidance-Modulated FM。
INSIGHT
PropFly 学的是外观变化沿时间的传播，source motion 是主要保留对象。
PropFly
16
+
METHOD
II / TASKS & CONTROL / LOCAL & COMPOSITIONAL
CoinVE
将多条指令分别绑定到 region、subject 与 attention query。
coinve-framework.png
Instruction queries
每条操作使用独立 instruction query；Qwen3-VL 为其建立语义 +表示，再送入区域与控制分支。
Mask predictor 定位区域，GateNet 调整操作作用范围，降低不 +同主体间的 。
区域感知的 query 混合控制 。
Backbone
Wan-14B + Qwen3-VL-8B
Data
CoinVE-200K; 2–5 operations / sample
Labels
Per-operation masks + composite mask
INSIGHT
评测同时看 per-operation success、all-operation success 与 non-target preservation。
CoinVE · CoVEBench · ReBind
17
+
TOPIC
II / TASKS & CONTROL / MOTION & ACTION
Motion, action & camera control
文本动作、身体结构、点轨迹与相机路径提供不同强度的动态约束。
Event / intent
“走近杯子，拿起，再喝水”
DynaEdit / Goku
Pose / depth
Body / geometry
规定骨架或结构轨迹
结构条件编辑器
Space–time path
指定点的身份、位置与可见性
MotionV2V
Camera control
Viewpoint
改变视点，并处理遮挡与新显露
NeoVerse / WiW
“拿杯子并喝水”包含靠近、接触、持杯移动与杯口接触，不能由单个轨迹分数 +代替。
Motion preservation
传播外观通常希望保留源运动；动作替换则需要主动释放这部分约束。
INSIGHT
Action editing 需要检查事件顺序与物体交互；平滑运动不能替代动作完成。
MotionV2V · DynaEdit · Goku · NeoVerse · World in World
18
+
METHOD
II / TASKS & CONTROL / MOTION & ACTION
MotionV2V
用 source / target point tracks 指定运动变化，保留 source RGB 作为视觉条件。
01
Track source
TAPNext 提取 +稀疏 2D point tracks
→
02
Specify target
用户 Bézier curves +编辑轨迹与时序
→
03
训练控制分支 +冻结 CogVideoX-5B
Frame interpolation、time resampling 与 spatial transforms 生成可训练的 +运动差异与对应 tracks。
Control granularity
冻结 CogVideoX-5B，训练类似 ControlNet 的 分支。Tracks 指定位 +移，接触关系仍依赖生成先验。
INSIGHT
源视频作为 进入网络；target tracks 提供额外运动监督。
MotionV2V
19
+
CASE
II / TASKS & CONTROL / MOTION & ACTION
“The man does not walk toward the door; instead, he drinks water.”
Source / walks toward the door
Edited / walking retained, drinking added
Target event
末段出现喝水动作。
Source event
原走路仍被保留，未完成替换。
Diagnosis
Action appending ≠ substitution.
DIAGNOSIS
一次成功替换需要同时满足目标动作出现与被替换动作消失。
Editing examples
20
+
METHOD
II / TASKS & CONTROL / PHYSICS &
VOID
deletion：删除对象后，重新生成受它影响的交互结果。
void.png
删除区、受影响区、重叠区、保护区采用不同标签。
先生成反事实结果，再用 flow 引导可选的细化。
Causal scope
删除接球者后，球的后续运动也需要重新生成。
Condition
Source RGB + + description
Pipeline
VLM / segmentation → generator → refine
Task
Object-and-interaction removal
roles
删除区 / 受影响区 / 重叠区 / 保护区分别编码。重叠区允许反事 +实运动经过旧对象位置。
INSIGHT
“需要保留什么”取决于干预：被删除对象造成的后果可以随之改变。
VOID
21
+
METHOD
II / TASKS & CONTROL / PHYSICS &
PhysEditWorld
：改变 gravity，用同一初态与控制输入观察后续轨迹。
Replay A
scene + initial state + actions
gravity g₁
→
trajectory A
Replay B
scene + initial state + actions
gravity g₂
→
trajectory B
匹配的是控制输入与 camera policy，跟随相机的实际轨迹仍会变化。
Gravity 配对用于 Wan / LingBot 的 训练；VGGT 加速度是筛选与尺度 +处理后的代理，定量范围有限。
INSIGHT
匹配重放提供物理反事实监督；真实 source-video editing 仍需独立验证。
PhysEditWorld · What-If World
22
+
TOPIC
II / TASKS & CONTROL / WORLD MODELS
World models × video editing
共同点是可干预的生成状态；操作对象和已验证任务有明显区别。
State / interventionRepresentative workWhat the evidence supports
Visual evidence / cameraWorld in World观测与几何证据约束 camera re-rendering
Explicit scene / viewpointNeoVerse + completion；附录含外观编辑
Ongoing / actionActionSplice生成中修正 movement / camera actions
Physical variable / replayPhysEditWorldGravity 配对对条件生成与 的训练效用
Source video editing
以已有观测为条件，修改外观、视角或交互结果，检查非目标内容。
intervention
在生成进程中修改动作或状态，检查响应时间、前缀保持与后续一致性。
INSIGHT
Camera interventions 检验视角一致性；action / gravity interventions 还需检验轨迹与交互后果。
World in World · NeoVerse · ActionSplice · PhysEditWorld
23
+
METHOD
II / TASKS & CONTROL / WORLD MODELS
World in World
：将观测、warp、几何与历史作为可查询的视觉证据。
world-in-world.png
为观测、warp 与几何状态记录 camera / time / +validity。
决定从哪里取证据。
Evidence-wise
分别调节证据影响，编辑仍需要首帧、mask 与描述。
Backbone
Frozen LingBot-World-2
Metadata
Camera / time / validity
Edit input
Edited first frame + mask + description
对应关系限定 attention 读取的位置，evidence-wise 分别 +调节不同证据。
INSIGHT
主要定量证据来自 camera re-rendering；长期 memory 等能力需区分已验证与展望。
World in World · NeoVerse · GaussianDWM++
24
+
TOPIC
II / TASKS & CONTROL / WORLD MODELS
Scene editing & intervention
从显式场景、正在生成的 ，到共享随机性的反事实分支。
MECHANISM / METHOD
SOURCE EVIDENCE
INFERENCE / LEARNING
Explicit scene state
monocular video
→
重建可随时间变化的三维表示，再由视频生成器补全新视角中缺失的内容。
Action update during sampling
ongoing generated
→
correct at
新控制到达当前求解步后，受训 修正内部状态，继续生成后续视频。
Paired hypothetical outcomes
→
branch under interventions
共享随机性用于定义分叉对照；当前实验验证尚待开展。
timing
r 是求解步；视频边界 m 是输出时间。CST-T 在前者修正后者 +的后缀。
Evidence boundary
CST-T 的严格前缀保持针对 ；最终像素仍需检验。
INSIGHT
操作重建场景； 修正生成中 ； 定义共享随机性的分支对照。
· ·
25
+
TOPIC
III / TIME & INTERACTION / LONG & STREAMING
Long video & streaming
Offline duration、causal output、real-time response 对应不同系统目标。
MECHANISM / METHOD
SOURCE EVIDENCE
INFERENCE / LEARNING
MLV-Edit
Training-free extension
offline
→
+ anchor
全局锚点缓解漂移，但过强约束会压制真实变化。
Trained streaming editor
source-aligned
→
+
少步输出降低计算量，仍需处理自己的历史误差。
SVEET /
Adaptation & reuse
existing streaming model
→
transfer control / reuse prior
前者训练离线控制，后者组织 training-free 双分支采样。
What persists
边界传递像素、、 或 reference，决定状态能保留多 +久。
What to report
首帧延迟、稳态吞吐、峰值显存、总时长，以及随时长增长的漂移。
INSIGHT
报告 latency、throughput、memory 与 long-horizon drift，才能解释“实时”。
MLV-Edit · LiveEdit · JoyAI-Video-Edit · SVEET ·
26
+
METHOD
III / TIME & INTERACTION / LONG & STREAMING
JoyAI-Video-Edit
从双向编辑模型到 streaming editor：训练、少步采样与系统优化共同作用。
01
Editing pretraining
T2I / T2V / I2I / V2V +获得双向编辑能力
→
02
+压缩为 two-step sampling
→
03
Long-horizon
+ +causal 输出视频
稳态 8 RGB frames / ； 与 保留 +有限历史，source 持续输入。
30.19 FPS / B200 / 720×1280，长期定量证据到一分钟。少步 distillation、 +FP8、kernel 与 优化共同构成该结果。
INSIGHT
吞吐率不能直接换成首帧延迟；open-ended 接口仍要测长期漂移。
JoyAI-Video-Edit
27
+
METHOD
III / TIME & INTERACTION / LONG & STREAMING
SVEET
在双向模型上学 control，再迁移到冻结的 causal generator。
Bidirectional training
Wan2.1-VACE-1.3B +学习 stylization / inpainting / depth + +减少对双向时间上下文的依赖
Task-specific conditioning branches
Causal transfer
Frozen causal backbone +对 更新做 +抑制与 冲突的方向 +保留实时 的计算结构
Transfer trained control, preserve causal prior
Visibility mismatch
双向训练能看到未来；causal 只能读历史。适配需要处理这项 feature +gap。
Condition budget
Stylization、inpainting、depth-to-video 使用不同条件；mask / depth 必须 +计入接口。
INSIGHT
部分任务需要 mask 或 depth；冻结主干仍包含离线控制训练。
SVEET
28
+
METHOD
III / TIME & INTERACTION / MULTI-SHOT EDITING
MSEditor
跨过 shot cut，把同一主体与目标外观绑定到不同视角。
mse-framework.png
用深度与相机重投影构造多视角代理，使同一外观修改能对应到 +不同视点。
Supervisory Adapter 连接条件，将多个镜头与编辑目标打包为共 +同 token 序列。
Sparse
通过稀疏交互传递共享主体与目标外观。
Backbone
Wan2.1-14B
Input
Source + masks + edited first frame
Training
Depth / camera reprojection proxies
INSIGHT
真正困难的测试是主体缺席后重现，以及大视角变化后的编辑保持。
MSEditor
29
+
METHOD
III / TIME & INTERACTION / MULTI-SHOT EDITING
MMLVE-Agent
Shot assignment、visual memory 与 feedback 构成跨镜头执行流程。
01
Locate & assign
切分镜头，定位实体 +分配需要执行的指令
→
02
Render with memory
before / after references +调用 HappyHorse 1.0
→
03
正负反馈修订 prompt +重试相关镜头
Memory unit
保存视觉参照，供多个镜头使用。
Evidence scale
25-case benchmark；有效分母与服务拒绝处理影响均值。
INSIGHT
跨镜头一致性既可以来自模型内 attention，也可以来自模型外的 reference 管理。
MMLVE-Agent · MSEditor
30
+
TOPIC
III / TIME & INTERACTION / MULTI-TURN & AGENTS
Multi-turn editing
检索过去的视觉内容，还需要知道哪些修改仍然有效。
Source
original identity
不可变参照
→
Turn 1
red shirt → black
新增修改
→
Turn 2
change background
保持衣服修改
→
Turn 3
undo shirt edit
背景修改继续有效
检索相关历史，按重要性动态 tokenization / compression，再注入 。
需要记录哪些修改仍有效，以及被覆盖、被撤销的对象与范围。
INSIGHT
Undo 需要原始参照与有效修改记录；仅检索历史画面无法区分应保留与应撤销的变化。
· Editing examples
31
+
CASE
III / TIME & INTERACTION / MULTI-TURN & AGENTS
Apply, hold,
将左侧人物的衣服改黑、右侧人物的毛衣改为深蓝，保持后再恢复原色。
Source
Turn 1 / apply
Turn 2 / hold
Turn 3 /
Binding
两名人物分别接收自己的颜色修改。
Persistence
第二轮维持两项有效修改。
Restoration
第三轮恢复原衣服颜色。
DIAGNOSIS
观察同一段视频中，目标绑定、跨轮保持与恢复是否同时成立。
Editing examples
32
+
TOPIC
III / TIME & INTERACTION / MULTI-TURN & AGENTS
Editing agents
条件准备、工具执行与失败返修，分别影响系统效果和成本。
· tool preparation
Qwen3-VL-8B agent +检索 reference、grounding、结构化计划 +Wan2.2-TI2V-5B 负责像素生成 +Qwen3.5-4B 提供编辑条件
从自然语言请求到可用的生成条件
MMLVE · shot coordination
按 shot 分配任务 +Global Memory Card 维持目标外观 +检查失败并修订执行 prompt +底层 editor 为外部模型
从单次调用到多镜头执行与返修
固定底层 editor 和调用预算，再比较有无 reference 检索、分镜分配、反馈返 +修。
End-to-end cost
记录工具准确率、调用次数、等待时间、服务拒绝和最终失败率。
INSIGHT
需要分别记录 tool accuracy、edit success、重试次数与端到端成本。
· MMLVE-Agent
33
+
TOPIC
IV / DATA & EVALUATION / DATA & SUPERVISION
Where do target videos come from?
真实源上的编辑、分别生成的配对、teacher 估计与模拟器重放提供不同监督。
MECHANISM / METHOD
SOURCE EVIDENCE
INFERENCE / LEARNING
Señorita · InsViE
real source video
→
expert edit / propagation
源保持明确，编辑失败和筛选偏差可能进入 target。
Goku movement
matched keyframes
→
two generations
两段运动与背景可能同时变化，需检查非目标错配。
Online / simulation
PropFly · PhysEditWorld
teacher estimates / replay
→
condition change / physics
前者受 teacher 能力约束，后者受变量与模拟域覆盖约束。
Pair validity
检查目标变化以外的身份、背景、运动是否错配，避免把随机差异学成编辑。
Filtering bias
只保留高分样本会改变任务分布；需要记录难例被丢弃的原因与比例。
IMPLICATION
配对有效性决定监督质量；样本数和任务标签不能单独说明这一点。
Señorita-2M · InsViE-1M · Goku · PropFly · PhysEditWorld
34
+
METHOD
IV / DATA & EVALUATION / DATA & SUPERVISION
Goku
将 movement 与 composition 的数据构造，连接到统一编辑器训练。
goku.png
配对关键帧经两次 生成 source / target。源视频也是合成 +的，需要筛查非目标运动差异。
Composite target
2–5 个专家操作串联构造 target；前序误差与操作顺序可能进入 +监督。
统一编辑器学习直接执行，推理无需重复专家链。
Data
Goku-2M; 10 edit classes
Benchmark
Goku-Bench; 1K cases
Editor
Wan2.2-5B + learned mask branch
IMPLICATION
需要检查双路 的非目标差异，以及专家执行顺序留下的监督偏差。
Goku
35
+
TOPIC
IV / DATA & EVALUATION / DATA & SUPERVISION
Supervision beyond video pairs
操作级区域、物理状态和生成来源，使监督可以被定位与复核。
CoinVE 保存 2–5 条操作 +逐指令区域与 composite mask +可定位遗漏、错误主体与
Instruction ↔ subject ↔ region
State & provenance
PhysEditWorld 保存 +Game2World 用 去 HUD +分别追踪物理状态和清理对下游的影响
Initial state / intervention / editor provenance
Training utility
Operation labels 可定位错绑和遗漏；state variables 使物理干预可以匹配重 +放。
Downstream evidence
的去 HUD 应评价对 world-model 训练的作用，不能直接视为 +通用编辑成功。
IMPLICATION
操作结构、物理变量与生成来源，支持更精细的训练筛选和失败分析。
CoinVE · PhysEditWorld · Game2World
36
+
TOPIC
IV / DATA & EVALUATION / BENCHMARKS & EVALUATION
What counts as a successful edit?
分开观察目标完成、非目标保持与输出质量，才能解释失败来自哪里。
01
对象、属性、动作与事件顺序 +定位到正确区域和时间 +组合任务看每项与全部完成
检查每个操作，并保留 all-operation success。
02
身份、背景与有效历史修改 +允许反事实的必要后果改变 +检查跨帧、跨镜头、跨轮漂移
改变物理交互时，必要后果也属于目标变化。
03
Quality & coherence
画质、闪烁、运动与接触 +长视频关注累积错误 +音视频任务还需要同步证据
AV editing 还要检查音频内容与声画同步。
IMPLICATION
很高，既可能说明保留成功，也可能意味着根本没有编辑。
VEFX-Bench · CoVEBench · JAVEdit · 本地 benchmark 汇总（2026-09-26）
37
+
TOPIC
IV / DATA & EVALUATION / BENCHMARKS & EVALUATION
Benchmark landscape
按任务与可见证据选择 benchmark，再讨论模型分数。
Research questionResourcesEvidence to inspect
Visual editingFiVE / IVEBench / VEFX指令、保持、质量与输入条件
Compositional binding / CoinVE-Bench每项操作、全部完成与主体绑定
Motion & structureGoku-Bench / MotionV2V事件、轨迹、相机与非目标变化
Physics & world / What-If World无关扰动稳定性 / 成对物理干预
Comparable interfaces
Text-only、edited keyframe、mask、tracks 的目标信息量不同，分别列出。
Failure granularity
均值之外保留 missed edit、wrong entity、collateral change、temporal drift +等失败类型。
IMPLICATION
最后一类包含生成与物理预测诊断，不能直接混入 instruction-edit 排行榜。
FiVE · VEFX-Bench · · CoinVE · Goku · et al.
38
+
TOPIC
IV / DATA & EVALUATION / BENCHMARKS & EVALUATION
Can the judge detect an unedited output?
No-edit：直接把 source 作为 output，检验指令完成指标的判别力。
VEFX Motion33
IF · 1–4 · Reward-4B
高于该协议的被测编辑方法
Omni Temporal-200
Overall · full video · Qwen3-VL-32B
高于该协议已有结果的编辑方法
IVEBench Short64
IS · 1–5 · Qwen2.5-VL-72B
该设置能压低无编辑输出得分
PROTOCOL SNAPSHOT 2026-09-26 / NO-EDIT EXCLUDED FROM EDITOR RANKING
IMPLICATION
三个分数属于不同量表；共同诊断的是评判者是否把保守输出误认成成功编辑。
本地 benchmark 汇总（2026-09-26）
39
+
TOPIC
IV / DATA & EVALUATION / BENCHMARKS & EVALUATION
Evaluation protocol
抽帧位置、上下文与 rubric 决定 judge 实际看到了哪些证据。
01
12 frames / full video +时间位置、音轨、分辨率
→
02
Judge & rubric
模型版本、prompt +完成 / 保持 / 质量的标准
→
03
No-edit + 人工分歧 +失败率 + 有效样本分母
Observed sensitivity
本地 12 帧与全视频设置会改变分数与排序，变化方向还依赖 judge 容量。
Comparable reporting
固定 cases 与输入接口，保留失败案例，再报告分项和区间。
IMPLICATION
“更长输入”提供更多证据，但准确性仍需人工或可验证任务来校准。
本地 benchmark 汇总（2026-09-26） · CoVEBench · MMLVE-Agent
40
+
TOPIC
OPEN PROBLEMS / 1 OF 5
Action & object interaction
动作编辑还需要绑定已有物体、建立接触，并让其他参与者与物体状态作出响应。
Open questionWhat remains difficultDiagnostic case
把动作绑定到原视频中的物体， +确认身份、初始位置与可用状态叼起原地面上的骨头，避免新增替身 +检查物体轨迹、数量与身份
手与物体的接触、姿态与状态变化 +能否在同一时间线上成立？抓稳、倾倒、放回，逐段检查 +穿模、滑动、开合与液体变化
多个人或物体能否协同响应， +让动作、接触与回应按顺序发生？双人递杯：递出、接住、松手 +检查角色、交接时刻与物体连续性
用新动作替换原事件， +并同步改变必要的交互后果把原投球改为把球放进箱中 +旧投球消失，原球进入原箱
Existing objects
源视频可提供物体身份、几何与初始状态；如何把这些证据转成接触和动作约 +束，仍需专门设计。
Compositional control
多角色、多物体、多指令需要分别绑定。应同时检查每项交互和全部交互的完 +成情况。
INSIGHT
交互编辑需同时验证对象绑定、接触时序与物体状态，并允许必要的轨迹和交互后果随目标动作改变。
DynaEdit · MotionV2V · VOID · CoinVE · World in World
41
+
CASE
OPEN PROBLEMS / ACTION & OBJECT INTERACTION
Dog–object interaction
任务：让狗叼起原视频地面上的骨头。检查同一物体的接触、拾取与后续运动。
Source / 原骨头在地面
Archived target / 白色骨头在嘴里
/ 骨头仍在地面
Still frames · 2.40 s
原骨头黄褐色；target 中的骨头变成白色，形状也 +不同。
Target 从首帧便叼着骨头，未展示从地面拾起的 +过程。
输出中的骨头仍在地面，狗的毛色发生额外变化。
DIAGNOSIS
原物体交互要求同一对象从地面进入嘴中，并保持接触与运动连续；只出现叼骨头的画面还不足以判成功。
Editing examples ·
42
+
TOPIC
OPEN PROBLEMS / 2 OF 5
History & multi-turn editing
需要保存可追溯的视觉证据，也需要知道每轮指令改变了什么、哪些修改仍然有效。
Open questionWhat remains difficultDiagnostic case
原视频、历史输出与指令保存什么？ +压缩后如何保留下一轮需要的依据？仅上一轮 / 全历史 / 结构化记录 +比较长序列的身份与修改保持
新指令如何覆盖旧指令， +并确定每个对象当前有效的修改？蓝杯改红，再改绿，背景不变 +检查指代、冲突与当前有效颜色
撤销某步时保留其他有效修改， +同时处理依赖该步骤的后续操作撤销杯子颜色，保留新背景 +撤销递杯后，接杯动作应如何处理？
长遮挡、跨镜头与多轮重编码后， +如何找回同一对象与历史约束？物体离场后再出现，继续前轮编辑 +按轮记录身份、纹理与状态漂移
Visual history + edit state
历史画面提供视觉证据；有效指令、对象绑定与版本关系记录编辑状态。 + 只保存部分计算历史。
Memory budget
固定显存与读取预算，比较整段保留、关键帧检索与结构化记录，检查压缩是 +否丢掉关键事件。
INSIGHT
多轮编辑需要同时知道“曾经发生什么”和“现在应保留什么”，让覆盖、撤销与对象重现使用一致的历史状态。
Memory-V2V · MLV-Edit · JoyAI-Video-Edit · CoinVE · Editing examples
43
+
TOPIC
OPEN PROBLEMS / 3 OF 5
World models × video editing
从原视频恢复可干预的状态，再将预测的交互后果落实到生成结果中。
Open questionWhat remains difficultDiagnostic case
视频能约束哪些几何、接触与速度？ +被遮挡状态的不确定性如何保留？同一视频比较 2D tracks 与 3D state +检查遮挡后原物体定位与交互
“叼起”如何对应对象、接触与状态？ +动作条件能否控制同一实例？指定原骨头、接触时刻与离地目标 +比较纯文本与显式状态约束
改动一个动作后，哪些后果必须变？ +如何保留不受影响的内容？改变递杯方式，接收方同步响应 +检查接触后运动及无关区域保持
预测状态怎样变成编辑视频？ +生成偏差能否反馈并修正后续轨迹？遮挡后继续搬动物体并改变视角 +比较一次生成与状态反馈修正
可见物体位置来自观测；遮挡几何、质量与摩擦可能只能估计。错误状态会让 +后续预测偏离原场景。
编辑可用于构造共享初始状态的反事实训练对；需验证干预与后果，并检查编 +辑伪影是否成为学习捷径。
INSIGHT
World model 与编辑结合，需要从原视频恢复可干预状态，并让预测后果与生成视频中的对象、接触和事件一致。
World in World · NeoVerse · ActionSplice · VOID · PhysEditWorld
44
+
TOPIC
OPEN PROBLEMS / 4 OF 5
Editing agents & world-model planning
把目标拆成可执行条件，检查生成结果，再决定修复、回滚或结束。
Open questionWhat remains difficultDiagnostic case
对象、镜头与操作依赖怎样落到 +底层 editor 能执行的控制条件？把叼骨头分为定位、接触、拾取 +检查计划与实际输入条件是否对应
该用 mask、轨迹、参考图还是状态？ +工具失误怎样传到最终视频？固定 editor 比较不同条件接口 +分别记录定位错、控制错与生成错
怎样发现错物体、伪接触与历史遗忘？ +返修能否保留已经成功的部分？加入“新骨头替身”与 no-edit 负例 +检验判错后的局部返修及副作用
何时重试、回滚或让用户确认？ +更多调用是否真的提升完成率？相同调用预算比较固定流程与 agent +记录任务完成、延迟与停止原因
Agent 提出对象和事件干预，world model 预测后果，editor 生成，再核对结 +果状态；每一环都可能传播错误。
工具输出、对象 ID、有效修改与版本依赖需要对齐。撤销递杯后，应重新检查 +后续接杯与放杯的前提。
INSIGHT
Agent 的价值要体现为可执行的计划、可定位的失败和可恢复的编辑，并在相同预算下提高真实任务完成率。
Aurora · MMLVE-Agent · Memory-V2V · World in World · CoVEBench
45
+
TOPIC
OPEN PROBLEMS / 5 OF 5
Benchmark design & judge reliability
评测需要覆盖交互过程与历史依赖，并证明评判者能识别具体失败。
Open questionWhat remains difficultDiagnostic case
接触、原物体复用、事件替换与 +多轮依赖是否真正进入测试？按任务、对象关系与时长分层 +加入依赖历史的 session tasks
能否区分完成、未编辑、物体替身 +以及把新动作追加在原动作之后？no-edit、错物体、错顺序负对照 +人工校准，并报告误判率
稀疏抽帧可能遗漏接触瞬间； +上下文与 judge 也会影响评分固定样本比较抽帧与全视频 +加入事件时间戳，检查短时失败
输入条件、额外生成与失败过滤 +怎样影响模型之间的比较？按文本 / mask / tracks 与预算分组 +保留拒绝、重试成本及置信区间
开放式编辑可以有多个合理结果。用对象、事件及有效历史约束定义可接受结 +果，单一 target 相似度并不充分。
检查训练与测试的来源重叠、配对偏差和合成偏差；记录困难案例被过滤的原 +因及任务分布变化。
INSIGHT
Benchmark 应能定位错物体、接触失败、旧动作残留与历史遗忘，并说明判分依据、输入条件和完成任务的成本。
CoVEBench · IVEBench · VEFX-Bench · 本地 benchmark 汇总（2026-09-26） · MotionV2V
46
+ +

Contents

Introduction

Video Editing Frontiers
What changes when we edit a video?
Change, preserve, condition

Models & Priors

Training-free editing
FlowEdit
DynaEdit
DynaEdit: candidate selection & noise correlation
Image-to-video transfer
Learning an instruction editor
SAMA
Bernini
Bernini: how the planner and renderer are trained
TIDE
Discrete editing & joint audio–visual editing

Tasks & Control

Local & compositional control
PropFly
CoinVE
Motion, action & camera control
MotionV2V
Action substitution
VOID
PhysEditWorld
World models × video editing
World in World
Scene editing & rollout intervention

Time & Interaction

Long video & streaming
JoyAI-Video-Edit
SVEET
MSEditor
MMLVE-Agent
Multi-turn editing
Apply, hold, undo
Editing agents

Data & Evaluation

Where do target videos come from?
Goku
Supervision beyond video pairs
What counts as a successful edit?
Benchmark landscape
Can the judge detect an unedited output?
Evaluation protocol

Open problems

Action & object interaction
Dog–object interaction
History & multi-turn editing
World models × video editing
Editing agents & world-model planning
Benchmark design & judge reliability
 +

 +Paper figure + + \ No newline at end of file diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/presentation.js b/discussions/2026-09-28-peiyuan-video-editing-frontiers/presentation.js new file mode 100644 index 0000000..2b25843 --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/presentation.js @@ -0,0 +1,124 @@ +(() => { + const slides = [...document.querySelectorAll('.slide')]; + const meta = window.DECK_META; + const dialogs = [...document.querySelectorAll('dialog')]; + const popover = document.getElementById('termPopover'); + let termAnchor = null, pinned = false, termTimer, suppressFocus = false; + function closeTerm(restoreFocus = false) { + clearTimeout(termTimer); + const previous = termAnchor; + previous?.setAttribute('aria-expanded','false'); + previous?.removeAttribute('aria-controls'); + termAnchor = null; pinned = false; popover.hidden = true; + if (restoreFocus && previous) { suppressFocus = true; previous.focus(); suppressFocus = false; } + } + function placeTerm() { + if (!termAnchor || popover.hidden) return; + popover.style.maxHeight=`${innerHeight-24}px`; + const r=termAnchor.getBoundingClientRect(), p=popover.getBoundingClientRect(),pad=12; + let left=r.left < innerWidth/2 ? r.right+18 : r.left-p.width-18; + let top=Math.max(pad,Math.min(innerHeight-p.height-pad,r.top-30)); + if (leftinnerWidth-pad) { + left=Math.max(pad,Math.min(innerWidth-p.width-pad,r.left)); + const below=innerHeight-r.bottom-2*pad,above=r.top-2*pad; + if(below>=p.height)top=r.bottom+pad; + else if(above>=p.height)top=r.top-p.height-pad; + else if(below>=above){popover.style.maxHeight=`${Math.max(100,below)}px`;top=r.bottom+pad;} + else{popover.style.maxHeight=`${Math.max(100,above)}px`;top=pad;} + } + popover.style.left=`${left}px`;popover.style.top=`${top}px`; + } + function pinState(value) { + pinned=value;popover.classList.toggle('pinned',pinned); + document.getElementById('pinTerm').textContent=pinned?'Pinned':'Pin'; + document.getElementById('pinTerm').setAttribute('aria-pressed',String(pinned)); + document.getElementById('termMode').textContent=pinned?'PINNED · ESC TO CLOSE':'TECHNICAL NOTE'; + } + function showTerm(el,force=false) { + clearTimeout(termTimer); + if(suppressFocus || (pinned&&!force))return; + const d=window.DECK_DETAILS[el.dataset.detail]; + if(!d)return; + termAnchor?.setAttribute('aria-expanded','false');termAnchor?.removeAttribute('aria-controls'); + termAnchor=el;el.setAttribute('aria-expanded','true');el.setAttribute('aria-controls','termPopover'); + document.getElementById('termTitle').innerHTML=d.titleHtml; + const body=document.getElementById('termBody');body.replaceChildren(); + d.blocks.forEach(b=>{const h=document.createElement('h3'),p=document.createElement('p');h.textContent=b.label;p.innerHTML=b.html;body.append(h,p)}); + const sources=document.createElement('div');sources.className='term-sources'; + d.refs.forEach(r=>{const a=document.createElement('a');a.href=r.url;a.textContent=r.title+' ↗';a.target='_blank';a.rel='noopener noreferrer';sources.append(a)}); + body.append(sources);popover.hidden=false;popover.scrollTop=0;pinState(false);placeTerm(); + } + function hideTermSoon(){clearTimeout(termTimer);termTimer=setTimeout(()=>{if(!pinned&&!popover.matches(':hover')&&!popover.contains(document.activeElement)&&!termAnchor?.matches(':hover'))closeTerm()},180)} + document.querySelectorAll('.term-trigger').forEach(el=>{ + el.addEventListener('pointerenter',e=>{if(e.pointerType==='mouse')showTerm(el)}); + el.addEventListener('pointerleave',hideTermSoon); + el.addEventListener('focus',()=>showTerm(el));el.addEventListener('blur',hideTermSoon); + el.addEventListener('click',e=>{if(el.tagName==='A')return;e.stopPropagation();if(pinned&&termAnchor===el){closeTerm();return;}showTerm(el,true);pinState(true)}); + el.addEventListener('keydown',e=>{if(e.key==='ArrowDown'){e.preventDefault();showTerm(el,true);pinState(true);document.getElementById('pinTerm').focus()}}); + }); + popover.addEventListener('pointerenter',()=>clearTimeout(termTimer));popover.addEventListener('pointerleave',hideTermSoon); + popover.addEventListener('focusout',hideTermSoon); + document.getElementById('pinTerm').onclick=()=>pinState(!pinned); + document.getElementById('closeTerm').onclick=()=>closeTerm(true); + document.addEventListener('pointerdown',e=>{if(!popover.contains(e.target)&&!e.target.closest('.term-trigger'))closeTerm()}); + document.querySelectorAll('[data-notes]').forEach(el=>el.onclick=()=>toggle('notes')); + document.querySelectorAll('[data-image]').forEach(el=>el.onclick=()=>{document.querySelector('#zoom img').src=el.dataset.image;toggle('zoom')}); + let index = Math.max(0, meta.findIndex(s => s.id === location.hash.slice(1))); + const videos = () => [...slides[index].querySelectorAll('video')]; + function resize() { + document.documentElement.style.setProperty('--scale', Math.min(innerWidth / 1280, innerHeight / 720)); + placeTerm(); + } + function updateNotes() { + document.getElementById('noteTitle').innerHTML = `${index + 1}. ${meta[index].titleHtml}`; + document.getElementById('noteBody').innerHTML = window.DECK_NOTES[index]; + } + function go(n) { + closeTerm(); + index = Math.max(0, Math.min(slides.length - 1, n)); + slides.forEach((s, i) => { s.classList.toggle('active', i === index); s.inert = i !== index; if (i !== index) s.querySelectorAll('video').forEach(v => v.pause()); }); + history.replaceState(null, '', `#${meta[index].id}`); + document.getElementById('counter').textContent = `${index + 1} / ${slides.length}`; + document.getElementById('mediaButton').hidden = videos().length === 0; + document.getElementById('mediaButton').textContent = 'Play'; + document.getElementById('prev').disabled = index === 0; + document.getElementById('next').disabled = index === slides.length - 1; + updateNotes(); + } + function toggle(id) { closeTerm(); const d = document.getElementById(id); if (d.open) d.close(); else { dialogs.forEach(x => x.close()); d.showModal(); } } + async function play() { + const vs = videos(); + if (vs.some(v => !v.paused)) { vs.forEach(v => v.pause()); document.getElementById('mediaButton').textContent = 'Play'; } + else { vs.forEach(v => { v.currentTime = 0; }); await Promise.all(vs.map(v => v.play().catch(() => {}))); document.getElementById('mediaButton').textContent = 'Pause'; } + } + document.getElementById('prev').onclick = () => go(index - 1); + document.getElementById('next').onclick = () => go(index + 1); + document.getElementById('tocButton').onclick = () => toggle('toc'); + document.getElementById('notesButton').onclick = () => toggle('notes'); + document.getElementById('mediaButton').onclick = play; + document.getElementById('fullButton').onclick = () => document.fullscreenElement ? document.exitFullscreen() : document.documentElement.requestFullscreen(); + document.querySelectorAll('[data-go]').forEach(b => b.onclick = () => { go(Number(b.dataset.go)); document.getElementById('toc').close(); }); + dialogs.forEach(d => { d.querySelector('.close').onclick = () => d.close(); d.addEventListener('click', e => { if (e.target === d) { const r = d.getBoundingClientRect(); if (e.clientX < r.left || e.clientX > r.right || e.clientY < r.top || e.clientY > r.bottom) d.close(); } }); }); + document.querySelectorAll('.figure').forEach(img => { + const zoom = () => { document.querySelector('#zoom img').src = img.src; toggle('zoom'); }; + img.onclick = zoom; img.onkeydown = e => { if (e.key === 'Enter') zoom(); }; + }); + addEventListener('keydown', e => { + if(e.key==='Escape'&&!popover.hidden){e.preventDefault();closeTerm(true);return;} + if (dialogs.some(d => d.open)) return; + if (['INPUT','TEXTAREA','SELECT','VIDEO'].includes(document.activeElement?.tagName)||popover.contains(document.activeElement)) return; + if(document.activeElement?.tagName==='BUTTON'&&e.key===' ')return; + if (['ArrowRight','PageDown',' '].includes(e.key)) { e.preventDefault(); go(index + 1); } + if (['ArrowLeft','PageUp'].includes(e.key)) { e.preventDefault(); go(index - 1); } + if (e.key === 'Home') go(0); if (e.key === 'End') go(slides.length - 1); + if (e.key.toLowerCase() === 'o') toggle('toc'); + if (e.key.toLowerCase() === 'n') toggle('notes'); + if (e.key.toLowerCase() === 'f') document.getElementById('fullButton').click(); + if (e.key.toLowerCase() === 'p') play(); + }); + let timer; + addEventListener('pointermove', () => { document.body.classList.add('controls-visible'); clearTimeout(timer); timer = setTimeout(() => document.body.classList.remove('controls-visible'), 1800); }); + addEventListener('resize', resize); addEventListener('hashchange', () => { const i = meta.findIndex(s => s.id === location.hash.slice(1)); if (i >= 0) go(i); }); + // Programmatic navigation also supports repeatable offline review. + window.goToSlide = go; resize(); go(index); +})(); diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/references.html b/discussions/2026-09-28-peiyuan-video-editing-frontiers/references.html new file mode 100644 index 0000000..4394ed7 --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/references.html @@ -0,0 +1,195 @@ + + + + + + + Video Editing Frontiers — References · Video Editing Frontiers · CausalVideoGroup + + + +
+ CausalVideoGroup + +
+
+

2026-09-28 · Peiyuan Zhu

Video Editing Frontiers — References

Video Editing Frontiers

+

Key reading

+

1. FlowEdit (ICCV 2025)

+
    +
  • Link: FlowEdit (ICCV 2025)
    • +
  • Role: Inference-time transport
    • +
  • Why important: Editing follows a difference between target and source velocity fields.
    • +
  • Main limitation: The original work addresses images; a video adaptation changes the backbone and must verify temporal behavior.
    • +
  • Discussion question: How does source preservation interact with changes in motion?
    • +
+

2. DynaEdit

+
    +
  • Link: DynaEdit
    • +
  • Role: Dynamic rewriting
    • +
  • Why important: Its source-query and guidance design motivates a separate discussion of preserving appearance while changing dynamics.
    • +
  • Main limitation: Sampling controls do not by themselves establish correct object contact or event replacement.
    • +
  • Discussion question: Which failures come from the control signal, and which come from the backbone?
    • +
+

3. Bernini

+
    +
  • Link: Bernini
    • +
  • Role: Semantic planning and rendering
    • +
  • Why important: A semantic planner and a renderer expose different roles for edit intent and visual detail.
    • +
  • Main limitation: The deck follows the existing v1 method audit; it does not claim a completed v2 difference audit.
    • +
  • Discussion question: Which planner supervision improves action semantics without unwanted source changes?
    • +
+

4. World in World

+
    +
  • Link: World in World
    • +
  • Role: Source-grounded world-model editing
    • +
  • Why important: Source observations and correspondence-based evidence support controlled re-observation and editing.
    • +
  • Main limitation: Camera-rendering evidence should not be treated as proof of arbitrary physical interaction editing.
    • +
  • Discussion question: How should hidden state and contact uncertainty enter the editing interface?
    • +
+

5. MMLVE-Agent

+
    +
  • Link: MMLVE-Agent
    • +
  • Role: Editing agents
    • +
  • Why important: Shot assignment, global visual references, and feedback separate orchestration from pixel generation.
    • +
  • Main limitation: External editor capability, service failures, and retry costs influence the end-to-end result.
    • +
  • Discussion question: Does planning improve success over a fixed workflow at the same call budget?
    • +
+

6. CoVEBench

+
    +
  • Link: CoVEBench
    • +
  • Role: Compositional evaluation
    • +
  • Why important: Fine-grained checks help separate individual instruction requirements.
    • +
  • Main limitation: A compositional instruction is different from a sequence of requests with history and selective undo.
    • +
  • Discussion question: Which event and session annotations reveal failures hidden by aggregate quality?
    • +
+

Paper and model index

+ +

Example and asset notes

+
+
+ + + diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/references.md b/discussions/2026-09-28-peiyuan-video-editing-frontiers/references.md new file mode 100644 index 0000000..5ddbfbf --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/references.md @@ -0,0 +1,177 @@ +# Video Editing Frontiers — References + +## Key reading + +## 1. FlowEdit (ICCV 2025) + +- Link: [FlowEdit (ICCV 2025)](https://arxiv.org/pdf/2412.08629) +- Role: Inference-time transport +- Why important: Editing follows a difference between target and source velocity fields. +- Main limitation: The original work addresses images; a video adaptation changes the backbone and must verify temporal behavior. +- Discussion question: How does source preservation interact with changes in motion? + +## 2. DynaEdit + +- Link: [DynaEdit](https://arxiv.org/pdf/2603.17989) +- Role: Dynamic rewriting +- Why important: Its source-query and guidance design motivates a separate discussion of preserving appearance while changing dynamics. +- Main limitation: Sampling controls do not by themselves establish correct object contact or event replacement. +- Discussion question: Which failures come from the control signal, and which come from the backbone? + +## 3. Bernini + +- Link: [Bernini](https://arxiv.org/pdf/2605.22344) +- Role: Semantic planning and rendering +- Why important: A semantic planner and a renderer expose different roles for edit intent and visual detail. +- Main limitation: The deck follows the existing v1 method audit; it does not claim a completed v2 difference audit. +- Discussion question: Which planner supervision improves action semantics without unwanted source changes? + +## 4. World in World + +- Link: [World in World](https://arxiv.org/pdf/2609.11548) +- Role: Source-grounded world-model editing +- Why important: Source observations and correspondence-based evidence support controlled re-observation and editing. +- Main limitation: Camera-rendering evidence should not be treated as proof of arbitrary physical interaction editing. +- Discussion question: How should hidden state and contact uncertainty enter the editing interface? + +## 5. MMLVE-Agent + +- Link: [MMLVE-Agent](https://arxiv.org/pdf/2608.26809) +- Role: Editing agents +- Why important: Shot assignment, global visual references, and feedback separate orchestration from pixel generation. +- Main limitation: External editor capability, service failures, and retry costs influence the end-to-end result. +- Discussion question: Does planning improve success over a fixed workflow at the same call budget? + +## 6. CoVEBench + +- Link: [CoVEBench](https://arxiv.org/pdf/2606.08415) +- Role: Compositional evaluation +- Why important: Fine-grained checks help separate individual instruction requirements. +- Main limitation: A compositional instruction is different from a sequence of requests with history and selective undo. +- Discussion question: Which event and session annotations reveal failures hidden by aggregate quality? + +## Paper and model index + +- [World in World](https://arxiv.org/pdf/2609.11548) +- [NeoVerse](https://arxiv.org/pdf/2601.00393) +- [ActionSplice](https://arxiv.org/pdf/2609.08230) +- [PhysEditWorld](https://arxiv.org/pdf/2606.26694) +- [GaussianDWM++](https://arxiv.org/pdf/2608.16234) +- [Twin Rollouts](https://arxiv.org/pdf/2608.08982) +- [Game2World](https://arxiv.org/pdf/2608.24680) +- [CRONOS](https://arxiv.org/pdf/2605.23699) +- [What-If World](https://arxiv.org/pdf/2605.27589) +- [Qwen-Video-Edit](https://arxiv.org/pdf/2608.14790) +- [GRNEdit](https://arxiv.org/pdf/2608.16328) +- [MSEditor](https://arxiv.org/pdf/2608.17559) +- [MMLVE-Agent](https://arxiv.org/pdf/2608.26809) +- [EditVid](https://arxiv.org/pdf/2609.04190) +- [SVEET](https://arxiv.org/pdf/2609.24788) +- [VideoX-Qwen](https://arxiv.org/pdf/2609.26015) +- [Vidu S2](https://arxiv.org/pdf/2609.11638) +- [JAVEdit](https://arxiv.org/pdf/2606.03168) +- [MDN-Control](https://arxiv.org/pdf/2609.16475) +- [CoinVE](https://arxiv.org/pdf/2608.17566) +- [AnyV2V](https://arxiv.org/pdf/2403.14468) +- [DreamMotion](https://arxiv.org/pdf/2403.12002) +- [FastVideoEdit](https://arxiv.org/pdf/2403.06269) +- [COVE](https://arxiv.org/pdf/2406.08850) +- [VideoDirector](https://arxiv.org/pdf/2411.17592) +- [V2Edit](https://arxiv.org/pdf/2503.10634) +- [DFVEdit](https://arxiv.org/pdf/2506.20967) +- [VINO](https://arxiv.org/pdf/2506.12520) +- [FlowAnchor](https://arxiv.org/pdf/2604.22586) +- [EquiEdit](https://arxiv.org/pdf/2607.05056) +- [EffiVED](https://arxiv.org/pdf/2403.11568) +- [EVE](https://arxiv.org/pdf/2403.09334) +- [Movie Gen](https://arxiv.org/pdf/2410.13720) +- [VACE](https://arxiv.org/pdf/2503.07598) +- [UNIC](https://arxiv.org/pdf/2506.04216) +- [OmniV2V](https://arxiv.org/pdf/2506.01801) +- [EditVerse](https://arxiv.org/pdf/2509.20360) +- [Unpaired Clips](https://arxiv.org/pdf/2510.14648) +- [Scaling / Editto](https://arxiv.org/pdf/2510.15742) +- [UniVideo](https://arxiv.org/pdf/2510.08377) +- [EasyV2V](https://arxiv.org/pdf/2512.16920) +- [SAMA](https://arxiv.org/pdf/2603.19228) +- [Omni-Video 2](https://arxiv.org/pdf/2602.08820) +- [Kiwi-Edit](https://arxiv.org/pdf/2603.02175) +- [ViFeEdit](https://arxiv.org/pdf/2603.15478) +- [OmniWeaving](https://arxiv.org/pdf/2603.24458) +- [Mamoda2.5](https://arxiv.org/pdf/2605.02641) +- [Lance](https://arxiv.org/pdf/2605.18678) +- [Bernini](https://arxiv.org/pdf/2605.22344) +- [RVEDiT](https://arxiv.org/pdf/2605.24674) +- [LoomVideo](https://arxiv.org/pdf/2606.06042) +- [TIDE](https://arxiv.org/pdf/2606.08260) +- [I2VEdit](https://arxiv.org/pdf/2405.16537) +- [VIVID](https://arxiv.org/pdf/2411.15260) +- [VideoPainter](https://arxiv.org/pdf/2503.05639) +- [VideoGrain](https://arxiv.org/pdf/2502.17258) +- [Shape-for-Motion](https://arxiv.org/pdf/2506.22432) +- [PropFly](https://arxiv.org/pdf/2602.20583) +- [NOVA](https://arxiv.org/pdf/2603.02802) +- [MiVE](https://arxiv.org/pdf/2605.14664) +- [GIVE](https://arxiv.org/pdf/2606.24225) +- [ReBind](https://arxiv.org/pdf/2607.14681) +- [Streaming Video Diffusion](https://arxiv.org/pdf/2405.19726) +- [Memory-V2V](https://arxiv.org/pdf/2601.16296) +- [MLV-Edit](https://arxiv.org/pdf/2602.02123) +- [LIVEditor-14B](https://arxiv.org/pdf/2605.04569) +- [StreamEdit](https://arxiv.org/pdf/2605.21466) +- [LiveEdit](https://arxiv.org/pdf/2606.26740) +- [FateZero](https://arxiv.org/pdf/2303.09535) +- [Text2Video-Zero](https://arxiv.org/pdf/2303.13439) +- [TokenFlow](https://arxiv.org/pdf/2307.10373) +- [RAVE](https://arxiv.org/pdf/2312.04524) +- [ReCo](https://arxiv.org/pdf/2512.17650) +- [EditBoard](https://arxiv.org/pdf/2409.09668) +- [Señorita-2M](https://arxiv.org/pdf/2502.06734) +- [InsViE-1M](https://arxiv.org/pdf/2503.20287) +- [FiVE](https://arxiv.org/pdf/2503.13684) +- [CoVEBench](https://arxiv.org/pdf/2606.08415) +- [VEFX-Bench](https://arxiv.org/pdf/2604.16272) +- [Goku](https://arxiv.org/pdf/2606.30599) +- [Aurora](https://arxiv.org/pdf/2605.18748) +- [DynaEdit](https://arxiv.org/pdf/2603.17989) +- [MotionV2V](https://arxiv.org/pdf/2511.20640) +- [VOID](https://arxiv.org/pdf/2604.02296) +- [JoyAI-Video-Edit](https://arxiv.org/pdf/2608.03974) +- [FlowEdit (ICCV 2025)](https://arxiv.org/pdf/2412.08629) +- [WorldMirror](https://arxiv.org/pdf/2510.10726) +- [Wan](https://arxiv.org/pdf/2503.20314) +- [Wan2.2](https://github.com/Wan-Video/Wan2.2) +- [SD3](https://arxiv.org/pdf/2403.03206) +- [FLUX](https://github.com/black-forest-labs/flux) +- [FLUX.2 Klein](https://bfl.ai/models/flux-2-klein) +- [SigLIP](https://arxiv.org/pdf/2303.15343) +- [Gemma](https://arxiv.org/pdf/2503.19786) +- [LTX-2.3](https://ltx.io/blog/ltx-2-3-release) +- [SAM2](https://arxiv.org/pdf/2408.00714) +- [SAM3](https://arxiv.org/pdf/2511.16719) +- [VGGT](https://arxiv.org/pdf/2503.11651) +- [LingBot-World](https://arxiv.org/pdf/2601.20540) +- [Self Forcing](https://arxiv.org/pdf/2506.08009) +- [LongLive](https://arxiv.org/pdf/2509.22622) +- [Qwen-Image-Edit](https://arxiv.org/pdf/2508.02324) +- [Qwen2.5-VL](https://arxiv.org/pdf/2502.13923) +- [Qwen3-VL](https://arxiv.org/pdf/2511.21631) +- [Qwen3.5-4B](https://qwen.ai/blog?id=qwen3.5) +- [T5](https://arxiv.org/pdf/1910.10683) +- [CLIP](https://arxiv.org/pdf/2103.00020) +- [DINO](https://arxiv.org/pdf/2104.14294) +- [IVEBench](https://arxiv.org/pdf/2510.11647) +- [TAPNext](https://arxiv.org/pdf/2504.05579) +- [CogVideoX](https://arxiv.org/pdf/2408.06072) +- [ControlNet](https://arxiv.org/pdf/2302.05543) +- [LingBot-World-2](https://arxiv.org/pdf/2607.07534) +- [ReCamMaster](https://arxiv.org/pdf/2503.11647) +- [LucyEdit](https://d2drjpuinn46lb.cloudfront.net/Lucy_Edit__High_Fidelity_Text_Guided_Video_Editing.pdf) +- [HappyHorse 1.0](https://www.alibabacloud.com/help/en/model-studio/happyhorse-1-0-video-edit) +- [DDIM](https://arxiv.org/pdf/2010.02502) +- [Gemini Omni Flash](https://ai.google.dev/gemini-api/docs/models/gemini-omni-flash) + +## Example and asset notes + +- [Examples and evaluation protocols](evidence.html) +- [Figure, formula, and video sources](sources.html) diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/sources.html b/discussions/2026-09-28-peiyuan-video-editing-frontiers/sources.html new file mode 100644 index 0000000..c0cebc8 --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/sources.html @@ -0,0 +1,10 @@ + +Figure, formula & video sources · CausalVideoGroup +
CausalVideoGroup
+

2026-09-28 · Peiyuan Zhu

Figure, formula & video sources

+

Paper figures

+

Figures retain their original proportions. The presentation can enlarge the complete paper figures; the Bernini and DynaEdit slide views use crops of those originals.

+

Formula components

Eleven SVG components typeset the FlowEdit ODE, DynaEdit guidance, and Bernini training objectives with MathJax 3.2.2. The public presentation uses the pre-rendered assets and does not load a MathJax CDN at runtime.

+

Example videos

Nine videos accompany the action-substitution, clothing-edit, and dog–bone comparisons. They are the presentation examples supplied by Peiyuan Zhu. The dog clips are 832 × 480, 25 fps, 81 frames; their static poster images show 2.40 s. Original video files are retained for playback.

+

Example observations and limits · Open presentation

+ diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/style.css b/discussions/2026-09-28-peiyuan-video-editing-frontiers/style.css new file mode 100644 index 0000000..7e8ad0c --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/style.css @@ -0,0 +1,24 @@ +:root{color-scheme:light;--bg:#fbfaf7;--text:#17212b;--muted:#5e6b78;--line:#c9d0d5;--accent:#1e5a96;--navy:#102b4e} +*{box-sizing:border-box}html,body{margin:0;width:100%;height:100%;overflow:hidden}body{background:#e5e8eb;color:var(--text);font-family:-apple-system,BlinkMacSystemFont,"PingFang SC","Helvetica Neue",Arial,sans-serif}button{font:inherit;color:inherit;cursor:pointer}a{color:var(--accent)}button:focus-visible,a:focus-visible{outline:2px solid var(--accent);outline-offset:3px} +#deck{position:absolute;width:1280px;height:720px;left:50%;top:50%;transform:translate(-50%,-50%) scale(var(--scale,1));transform-origin:center}.slide{position:absolute;inset:0;background:var(--bg);display:none;overflow:hidden}.slide.active{display:block}.text,.rule,.figure,.media,table,.panel{position:absolute}.text{white-space:pre;letter-spacing:0;font-family:"PingFang SC","Helvetica Neue",Arial,sans-serif}.panel{border:0;pointer-events:none}.figure{object-fit:contain;cursor:zoom-in}.figure:focus-visible{outline:2px solid var(--accent)}.media{object-fit:contain;background:#102039} +table{border-collapse:collapse;table-layout:fixed;font-size:17px;background:var(--bg)}th,td{border:0.4px solid var(--line);padding:7px 12px;text-align:left;vertical-align:middle;white-space:pre-line;line-height:1.35;overflow-wrap:normal}th{font-size:17px;font-weight:600;background:#e8eef3;height:42px}td{color:var(--muted)}td.primary{color:var(--navy);font-weight:600} +.term-trigger{display:inline;border:0;border-bottom:1px dotted #7c94ab;padding:0;margin:0;background:none;font:inherit;color:inherit;letter-spacing:inherit;text-align:inherit;line-height:inherit;white-space:inherit;vertical-align:baseline;cursor:help}.term-trigger:hover,.term-trigger[aria-expanded=true]{color:var(--accent);background:#dfeaf380;border-bottom-style:solid}.details-link,.paper-link{position:absolute;bottom:10px;right:136px;background:none;border:0;padding:2px 0;color:var(--accent);font-size:10px;letter-spacing:.02em}.paper-link{right:200px}.details-link:hover,.paper-link:hover{text-decoration:underline} +.controls{position:fixed;bottom:5px;right:12px;z-index:20;display:flex;align-items:center;gap:1px;padding:2px 4px;background:#fbfaf7f5;border:1px solid var(--line);border-radius:3px;font-size:11px;opacity:0;transition:opacity .2s}.controls:hover,.controls:focus-within,body.controls-visible .controls{opacity:1}.controls button{border:0;background:transparent;padding:5px 8px;border-radius:0}.controls button:hover{background:#dfeaf3}.controls button:disabled{opacity:.3}.controls [hidden]{display:none}#counter{padding:0 6px;color:var(--muted);font-variant-numeric:tabular-nums} +dialog{background:var(--bg);color:var(--text);border:1px solid var(--line);border-top:3px solid var(--navy);border-radius:0;box-shadow:0 18px 55px #142d4430;max-width:min(1040px,94vw);max-height:88vh;padding:28px 36px;line-height:1.65}dialog::backdrop{background:#182b425c}.close{position:absolute;right:11px;top:5px;background:none;border:0;font-size:25px;color:var(--muted)}dialog h2{margin:0 0 18px;font-size:24px;color:var(--navy)}dialog h3{margin:17px 0 7px;color:var(--navy)}#toc{width:1040px}.toc-list{columns:2;column-gap:40px}.toc-list section{break-inside:avoid}.toc-list h3{font-size:12px;text-transform:uppercase;letter-spacing:.06em;color:#a33b34;break-after:avoid}.toc-list button{display:flex;justify-content:space-between;gap:12px;width:100%;text-align:left;border:0;border-bottom:1px solid var(--line);padding:7px 0;background:none;font-size:13px;break-inside:avoid}.toc-list button:hover{color:var(--accent)}.toc-list button span{color:var(--muted);font-variant-numeric:tabular-nums}#notes{width:850px;font-size:16px}.note-detail{margin:20px 0;padding:6px 18px 14px;border-left:2px solid var(--accent);background:#e8eef360}.note-detail p{margin:9px 0}.note-detail a{font-size:13px}#zoom{width:96vw;height:92vh;max-width:96vw;max-height:92vh;padding:32px;background:#fff}#zoom img{width:100%;height:100%;object-fit:contain} +.term-popover{position:fixed;z-index:50;width:470px;max-width:calc(100vw - 24px);max-height:calc(100dvh - 24px);overflow:auto;background:#fffefb;border:1px solid var(--line);border-top:3px solid var(--accent);border-radius:3px;padding:15px 21px 20px;box-shadow:0 14px 44px #142d443b;font-size:15px;line-height:1.7;color:#405160;overscroll-behavior:contain}.term-popover[hidden]{display:none}.term-tools{display:flex;align-items:center;gap:6px;font-size:10px;color:var(--muted)}.term-tools span{margin-right:auto;letter-spacing:.05em}.term-tools button{background:none;border:0;padding:3px 6px}.term-tools button:hover{background:#e8eef3}#closeTerm{font-size:20px;line-height:1}.term-popover h2{font-size:21px;line-height:1.3;color:var(--navy);margin:10px 0 15px}.term-popover h3{font-size:11px;letter-spacing:.04em;color:var(--accent);margin:14px 0 4px}.term-popover p{margin:0}.term-popover .term-sources{margin-top:15px;padding-top:10px;border-top:1px solid var(--line);font-size:12px}.term-sources a{margin-right:12px}.term-popover.pinned{border-top-color:#a33b34}.term-popover.pinned #pinTerm{color:#a33b34;background:#f2e9e5} +@media(max-width:600px){.term-popover{font-size:14px;padding:13px 16px}.term-popover h2{font-size:19px}dialog{padding:26px 20px}.toc-list{columns:1}.controls{right:5px;gap:0}.controls button{padding:5px 6px}} +@media(prefers-reduced-motion:reduce){*{transition:none!important;scroll-behavior:auto!important}} +@page{size:1280px 720px;margin:0}@media print{html,body{width:1280px;height:auto;overflow:visible;background:var(--bg);print-color-adjust:exact;-webkit-print-color-adjust:exact}#deck{position:static;transform:none;width:1280px;height:auto}.slide{display:block!important;position:relative;width:1280px;height:720px;break-after:page;page-break-after:always}.slide:last-child{break-after:auto}.controls,dialog,.term-popover,.details-link,.paper-link{display:none!important}.term-trigger{border:0;background:none;color:inherit}.figure{cursor:default}video{object-fit:contain}} +/* Paper links and technical explanations are distinct, composable affordances. */ +.paper-ref{font:inherit;color:inherit;text-decoration:underline;text-decoration-color:#8193a666;text-underline-offset:3px;text-decoration-thickness:1px;cursor:pointer;border:0} +.paper-ref:hover,.paper-ref[aria-expanded=true]{color:var(--accent);text-decoration-color:var(--accent);background:#dfeaf350} +.term-trigger.paper-ref{border:0;cursor:pointer} +.role-title .paper-ref{text-decoration-color:#8193a650;text-underline-offset:6px} +.figure-frame{position:absolute;overflow:hidden} +.figure-frame .figure{max-width:none;max-height:none} +.equation.term-trigger{position:absolute;display:block;border:0;padding:0;background:none;line-height:0;white-space:normal} +.equation img{display:block;width:100%;height:100%;object-fit:contain;pointer-events:none} +.equation.term-trigger:hover,.equation.term-trigger[aria-expanded=true]{background:#dce6ef88;outline:1px solid #93a8bc55;outline-offset:5px} +.toc-row{display:flex;gap:12px;align-items:center;justify-content:space-between;border-bottom:1px solid var(--line);font-size:13px;break-inside:avoid;padding:2px 0} +.toc-row>span{flex:1}.toc-list .toc-row button{width:auto;flex:none;border:0;padding:6px 2px 6px 10px;color:var(--muted);font-variant-numeric:tabular-nums} +.term-popover h2 .paper-ref{color:inherit} diff --git a/discussions/2026-09-28-peiyuan-video-editing-frontiers/summary.html b/discussions/2026-09-28-peiyuan-video-editing-frontiers/summary.html new file mode 100644 index 0000000..6663c3a --- /dev/null +++ b/discussions/2026-09-28-peiyuan-video-editing-frontiers/summary.html @@ -0,0 +1,27 @@ + +Video Editing Frontiers — Topic Summary · CausalVideoGroup +
CausalVideoGroup
+

2026-09-28 · Peiyuan Zhu

Video Editing Frontiers — Topic Summary

+

Successful editing requires the requested change, preservation of unrelated content, and a coherent event over time.

+

Method landscape

+

The presentation compares inference-time transport, learned instruction editors, local and compositional controls, motion and action editing, counterfactual changes, long and multi-shot video, multi-turn history, and tool-driven systems. Dedicated pages cover FlowEdit (ICCV 2025), DynaEdit, Bernini, MotionV2V, and other representative methods.

+

Mechanisms and evidence

+

FlowEdit supplies an image-editing transport formulation; video adaptation requires a video backbone. DynaEdit targets dynamic rewriting through its source queries and guidance. Bernini separates semantic planning from video rendering. These routes use different conditioning signals and training assumptions, so their capabilities need task-specific evidence.

+

World-model approaches introduce visual state, geometry, and interventions. World in World and NeoVerse provide different uses of source evidence. Moving from camera or appearance changes to existing-object interaction requires additional state recovery and contact constraints. ActionSplice intervenes during generation; transferring that mechanism to arbitrary recorded video needs source alignment.

+

Aurora and MMLVE-Agent motivate tool preparation, shot coordination, memory, and feedback. Their end-to-end gains should be compared with fixed workflows and repeated sampling under matched budgets.

+

Research questions

+
Central claim: Editing success must be checked at the level of objects, events, and valid edit history, alongside visual quality.
+
Hidden assumption: A desired action visible in the output may involve a new object, omit the required transition, or retain the event that should have been replaced.
+
Failure boundary: Existing-object contact, action substitution, occlusion, history-dependent instructions, and judge protocol changes expose distinct weaknesses.
+
Connection: Causal video editing links a requested intervention to its necessary consequences while retaining unrelated source evidence.
+
Concrete experiment: On the same source cases, compare outputs with no edit, an object substitute, an appended action, and a completed interaction; calibrate the judge against explicit object and event annotations.
+
Discussion seed: Which intermediate representation makes object interaction both controllable and verifiable?
+

Open questions

    +
  1. How can an editor operate on the same object already present in the source, with continuous contact and state change?
    2. +
  2. Which history should be stored to support instruction updates, selective undo, and reappearance after occlusion?
    3. +
  3. How can world models recover uncertain source state, propagate intervention consequences, and correct discrepancies in rendered video?
    4. +
  4. When do planning, tool choice, and local repair improve an editing agent under a fixed compute budget?
    5. +
  5. How should benchmarks expose wrong objects, missing contact, retained actions, and forgotten history without rewarding no-edit outputs?
    6. +

These are research questions and candidate experiments. The related methods supply specific mechanisms; the proposed combined system has not been established by the examples.

+

Examples and protocol notes

The deck includes action substitution, clothing edits across turns, and a dog–bone interaction comparison. They illustrate failure types without estimating a model-wide rate. The evaluation discussion uses a frozen 2026-09-26 snapshot. Read the evidence notes.

+ diff --git a/discussions/index.html b/discussions/index.html index 83bab80..0be588e 100644 --- a/discussions/index.html +++ b/discussions/index.html @@ -2,7 +2,8 @@
CausalVideoGroup

Weekly archive

Topic discussions

Background prepared before each meeting and technical conclusions recorded afterward, organized by date, leader, and topic.

-

2026-08-24 · Yuxin Wang

Physics-Consistent Video Generation: Who Teaches the Generator Physics?

A design-space view of how simulators, explicit controls, training signals, latent dynamics, and inference-time alignment can make generated videos obey real-world physical laws, and what current benchmarks can actually establish.

before-meeting video-generation physics-consistency world-model controllable-generation benchmark evaluation

Read discussion →
+

2026-09-28 · Peiyuan Zhu

Video Editing Frontiers

A survey of video editing mechanisms and controllable changes, with method deep dives, interactive examples, and open problems in object interaction, editing history, world models, agents, and evaluation.

before-meeting video-editing action-editing world-model agents benchmark

Read discussion →
+

2026-08-24 · Yuxin Wang

Physics-Consistent Video Generation: Who Teaches the Generator Physics?

A design-space view of how simulators, explicit controls, training signals, latent dynamics, and inference-time alignment can make generated videos obey real-world physical laws, and what current benchmarks can actually establish.

before-meeting video-generation physics-consistency world-model controllable-generation benchmark evaluation

Read discussion →

2026-08-10 · Yichang Jian

From Video Scores to World Consistency

A data-backed discussion of how video-generation evaluation moves from perceptual proxy scores toward geometric, revisit, and out-of-sight world consistency.

before-meeting video-generation world-model benchmark evaluation

Read discussion →

2026-07-20 · Jinyuan Hu

Multi-Shot Video Generation: From Joint Denoising to Online Directing

A technical map of multi-shot video generation, organized by how shots are generated and how history enters the model: holistic joint denoising, shot-level autoregression, causal streaming, and system-level directing.

before-meeting video-generation multi-shot-video long-video-generation causal-generation agentic-video

Read discussion →

2026-07-13 · Yifan Shen

Forcing Series: The Evolution of AR Video Distillation

A technical map of methods that distill bidirectional video diffusion models into few-step autoregressive generators that produce frames in temporal order and support streaming output.

before-meeting video-generation autoregressive-video video-distillation streaming-generation

Read discussion →
diff --git a/index.html b/index.html index 78c23cb..98e4234 100644 --- a/index.html +++ b/index.html @@ -30,9 +30,9 @@

Video generation from a causal perspective.

Weekly reading group

Topic discussions

Each entry prepares the necessary background before the meeting, then records the assumptions, failure cases, and experiments identified during discussion.

-

2026-08-24 · Yuxin Wang

Physics-Consistent Video Generation: Who Teaches the Generator Physics?

A design-space view of how simulators, explicit controls, training signals, latent dynamics, and inference-time alignment can make generated videos obey real-world physical laws, and what current benchmarks can actually establish.

before-meeting video-generation physics-consistency world-model controllable-generation benchmark evaluation

Read discussion →
-

2026-08-10 · Yichang Jian

From Video Scores to World Consistency

A data-backed discussion of how video-generation evaluation moves from perceptual proxy scores toward geometric, revisit, and out-of-sight world consistency.

before-meeting video-generation world-model benchmark evaluation

Read discussion →
-

2026-07-20 · Jinyuan Hu

Multi-Shot Video Generation: From Joint Denoising to Online Directing

A technical map of multi-shot video generation, organized by how shots are generated and how history enters the model: holistic joint denoising, shot-level autoregression, causal streaming, and system-level directing.

before-meeting video-generation multi-shot-video long-video-generation causal-generation agentic-video

Read discussion →
+

2026-09-28 · Peiyuan Zhu

Video Editing Frontiers

A survey of video editing mechanisms and controllable changes, with method deep dives, interactive examples, and open problems in object interaction, editing history, world models, agents, and evaluation.

before-meeting video-editing action-editing world-model agents benchmark

Read discussion →
+

2026-08-24 · Yuxin Wang

Physics-Consistent Video Generation: Who Teaches the Generator Physics?

A design-space view of how simulators, explicit controls, training signals, latent dynamics, and inference-time alignment can make generated videos obey real-world physical laws, and what current benchmarks can actually establish.

before-meeting video-generation physics-consistency world-model controllable-generation benchmark evaluation

Read discussion →
+

2026-08-10 · Yichang Jian

From Video Scores to World Consistency

A data-backed discussion of how video-generation evaluation moves from perceptual proxy scores toward geometric, revisit, and out-of-sight world consistency.

before-meeting video-generation world-model benchmark evaluation

Read discussion →
Meeting guidelinesHow to contribute
diff --git a/people/index.html b/people/index.html index 919a29f..159cba3 100644 --- a/people/index.html +++ b/people/index.html @@ -3,7 +3,7 @@

Group

People

Group leadership and Topic Discussion Leaders contributing to the research archive.

Group Leader · guangyi

Guangyi Chen

Email

Discussion Leaders

-

Discussion Leader · peiyuan

Peiyuan Zhu

Email

No published projects or discussions yet.

+
diff --git a/sitemap.xml b/sitemap.xml index 35e66d3..edd6d05 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -8,10 +8,14 @@ https://causalvideogroup.github.io/discussions/ https://causalvideogroup.github.io/people/ https://causalvideogroup.github.io/tags/ + https://causalvideogroup.github.io/discussions/2026-09-28-peiyuan-video-editing-frontiers/ https://causalvideogroup.github.io/discussions/2026-08-24-yuxin-physics-consistent-video-generation/ https://causalvideogroup.github.io/discussions/2026-08-10-yichang-video-generation-benchmark/ https://causalvideogroup.github.io/discussions/2026-07-20-jinyuan-multi-shot-video-generation/ https://causalvideogroup.github.io/discussions/2026-07-13-yifan-forcing-ar-video-distillation/ + https://causalvideogroup.github.io/discussions/2026-09-28-peiyuan-video-editing-frontiers/summary.html + https://causalvideogroup.github.io/discussions/2026-09-28-peiyuan-video-editing-frontiers/references.html + https://causalvideogroup.github.io/discussions/2026-09-28-peiyuan-video-editing-frontiers/meeting-note.html https://causalvideogroup.github.io/discussions/2026-08-24-yuxin-physics-consistent-video-generation/summary.html https://causalvideogroup.github.io/discussions/2026-08-24-yuxin-physics-consistent-video-generation/references.html https://causalvideogroup.github.io/discussions/2026-08-24-yuxin-physics-consistent-video-generation/meeting-note.html diff --git a/tags/index.html b/tags/index.html index b624d60..265f139 100644 --- a/tags/index.html +++ b/tags/index.html @@ -2,9 +2,11 @@
CausalVideoGroup

Index

Research tags

Browse projects and discussions by research theme.

-
+
+ + - + @@ -13,7 +15,8 @@ + -
+