A structure-focused, appearance-constrained physics injection framework that transfers simulation-derived physical structure into pretrained video diffusion models — while preserving their real-video generative prior.
Recent video diffusion models generate increasingly realistic videos, yet visual plausibility alone does not imply physical understanding. Physical laws govern evolving 3D scene states, while videos observe only their 2D projections. We introduce Phys4D, a structure-focused, appearance-constrained physics injection framework that transfers simulation-derived physical structure into pretrained video diffusion models while preserving their real-video generative prior. Phys4D operationalizes physics-aware video generation through a prior-preserving RGB-D-motion interface. Our key principle is simulation teaches physics, not appearance: simulation provides dense structural supervision, including depth, motion, and masks, while adaptation remains anchored to the pretrained real-video prior rather than drifting toward simulator-specific visual statistics. To this end, Phys4D exposes geometry and motion with lightweight prediction heads, injects local physical structure through gated geometry-motion supervision, and aligns sequence-level evolution using a simulation-grounded trajectory objective. Supported by a scalable coupled-physics simulation system, Phys4D improves observable physical behavior, geometry-motion consistency, and trajectory-level evolution across multiple video generation backbones, while maintaining visual realism and fidelity. We release the code, datasets, introduction slides, and video demos on this project page.
Simulated videos are procedurally rendered and visually mismatched with internet-scale real video. Directly fitting a generator to them transfers simulator-specific visual bias and degrades the pretrained prior. Phys4D therefore uses simulation selectively: prioritize physical structure while constraining appearance drift.
Structure-transfer formulation. We formulate physical knowledge injection for video generation as a structure-transfer problem, where simulation provides geometry, motion, and trajectory supervision while the pretrained real-video generative prior is preserved.
The Phys4D framework. A structure-focused physics injection framework that constructs a prior-preserving RGB-D-motion interface, injects local physical structure through gated geometry-motion supervision, and aligns global scene evolution through a unified trajectory-level policy objective.
Scalable coupled-physics simulator. We build a unified simulation system spanning primitive and coupled physical regimes, and show that Phys4D improves fine-grained physical behavior, geometry-motion consistency, and trajectory-level evolution across multiple video backbones.
Open release. To facilitate reproducibility and broader use, we release our code, simulation datasets, introduction slides, and video demos on this project page.
Phys4D injects simulation-derived physical structure into a pretrained video diffusion model through a prior-preserving RGB-D-motion interface, in three stages.
Two lightweight auxiliary heads are attached to the pretrained DiT backbone: a motion head predicting inter-frame optical flow and a depth head predicting per-frame depth. The heads are supervised by pseudo-labels from off-the-shelf estimators on curated internet videos and self-generated videos.
Simulation supervision is restricted along three axes — when, where, and what to learn. A LoRA-style residual physics adapter is the only trainable pathway through which simulation modifies the generator; a noise gate activates it at high-noise structure-forming steps, and a dynamic-region gate localizes it to regions where physical changes occur.
Adjacent-frame losses cannot tell whether the completed evolution follows the intended physical trajectory. Stage III lifts generated RGB-D-motion into 4D point trajectories and aligns them with simulator trajectories from paired rollouts, using a symmetric 4D Chamfer reward under a direct policy optimization objective.
α(σ) is the noise gate (when), Ĝℓ the learned dynamic-region gate (where), and Aℓ the lightweight physics adapter — the only pathway simulation may use.
A unified metric over lifted spatiotemporal points, combining 3D position and normalized time so that trajectories are compared jointly in space and time.
Symmetric 4D Chamfer distance between generated and simulated dynamic point sets from the same paired rollout — a simulation-grounded alignment reward, not a universal physical-law reward.
Real videos provide rich appearance priors but rarely expose dense 4D physical states. We use simulation not as a target visual domain, but as an executable physical teacher that converts solvers and constraints into video-compatible structural supervision.
Built on Isaac Sim and GarmentLab, integrating rigid-body dynamics, PBD, FEM, and task-specific solvers into a unified scene-level framework. Domain randomization over mass, friction, restitution, stiffness, gravity, and scale exposes regimes governed by shared physical laws while preventing memorization of fixed motion templates.
Physics-IQ scenarios under the matched switch-frame I2V protocol. Each clip shows the text prompt, the WAN2.2-5B baseline, and the same backbone with Phys4D.
A grabber arm lowers a tennis ball onto cardstock propped on a clockwise-rotating platform. Phys4D keeps contact and support consistent as the platform turns.
An orange ball rolls out of a black pipe across a coffee table. The baseline distorts object scale and count; Phys4D maintains correct object identity and a coherent rolling trajectory.
A knife slices through a halved tangerine on a glass cutting board — a contact and separation event requiring consistent local geometry through the cut.
Grapefruit juice pours from a glass dispenser into a glass holding water. Phys4D produces flow consistent with gravity and container geometry rather than emitting from the wrong side.
A blue block on a clockwise turntable casts a long shadow on the wall behind it — shadow geometry must track the block’s rotating 3D pose.
A grabber releases a paper towel onto a shallow dish of blue liquid — a coupled deformable–fluid interaction with wicking and sagging.
A folded paper burns on a glass cutting board while white smoke rises — a thermodynamic state change with directional buoyant motion.
A grey tennis ball exits a black tube and rolls rightward across a wood surface, requiring a stable rolling trajectory and consistent depth ordering with the lampshade.
A 30 lb kettlebell and a sheet of green paper are lowered onto two pillows — mass-dependent deformation that appearance-driven models tend to under-express.
We evaluate Phys4D from four complementary angles: observable video-level physics, 4D interface consistency, preservation of the real-video prior, and which design choices matter.
Does Phys4D improve observable physical plausibility in generated videos? Phys4D improves observable physical plausibility across three backbones and three independent physics diagnostics. Physics-IQ is reported under the matched switch-frame I2V protocol.
| Backbone | Params | Method | Physics-IQ MSE ↓ | Physics-IQ Score ↑ | VBench2.0-Physics ↑ | PhyGenBench ↑ |
|---|---|---|---|---|---|---|
| CogVideoX | 5B | Base | 0.013 ±0.002 | 18.8 ±1.4 | 0.4849 ±0.014 | 0.45 ±0.02 |
| CogVideoX | 5B | + Phys4D | 0.008 ±0.001 | 32.8 ±1.9 | 0.5954 ±0.018 | 0.58 ±0.03 |
| WAN2.2 | 5B | Base | 0.016 ±0.002 | 16.8 ±1.3 | 0.4928 ±0.015 | 0.45 ±0.02 |
| WAN2.2 | 5B | + Phys4D | 0.013 ±0.001 | 30.9 ±1.8 | 0.6247 ±0.017 | 0.57 ±0.03 |
| Open-Sora V1.2 | 1.1B | Base | 0.021 ±0.003 | 14.5 ±1.2 | 0.4623 ±0.016 | 0.44 ±0.02 |
| Open-Sora V1.2 | 1.1B | + Phys4D | 0.014 ±0.002 | 24.7 ±1.6 | 0.5899 ±0.019 | 0.57 ±0.03 |
Consistent gains across all three backbones suggest that simulation-derived physical structure transfers to generated videos beyond simulator-specific metrics.
Does Phys4D improve 4D world-level geometry–motion–trajectory consistency? Held-out simulated scenes with ground-truth depth, flow, and object trajectories, covering per-frame geometry, local geometry–motion consistency, global trajectory evolution, and novel-time continuity.
| Backbone | Method | AbsRel ↓ | Warp L1 ↓ | Flow EPE ↓ | 4D Chamfer ↓ | Mean Drift ↓ | Fail Rate ↓ | Novel Depth ↓ | Novel Warp ↓ |
|---|---|---|---|---|---|---|---|---|---|
| WAN2.2 | Base + OTF | 0.3929 | 0.7990 | 1.2516 | 0.5058 | 0.5369 | 12.38% | 0.5841 | 1.1076 |
| CogVideoX | Base + OTF | 0.3483 | 0.7054 | 1.2343 | 0.4923 | 0.5239 | 11.34% | 0.5432 | 1.1254 |
| Open-Sora V1.2 | Base + OTF | 0.4216 | 0.7436 | 1.2718 | 0.5286 | 0.5637 | 13.54% | 0.6534 | 1.1573 |
| Phys4D + WAN2.2 | RGB + OTF | 0.3347 | 0.6698 | 1.0824 | 0.4873 | 0.5126 | 10.91% | 0.5527 | 1.0954 |
| Phys4D + WAN2.2 | Ours | 0.2386 | 0.4558 | 0.4478 | 0.4159 | 0.4312 | 8.36% | 0.4746 | 1.0289 |
OTF denotes an off-the-shelf diagnostic pipeline that derives depth and motion cues from pretrained estimators (Depth Anything V2 and SEA-RAFT). Phys4D is evaluated through its own learned RGB-D-motion interface. All methods use the same held-out simulation scenes and camera settings.
Does Phys4D preserve the pretrained real-video generative prior? Evaluated with VBench 1.0 under the same prompts, sampling settings, and evaluation pipeline. Visual Avg. is the mean of Aesthetic Quality, Imaging Quality, and Color.
| Backbone | Params | Method | Aesthetic Quality ↑ | Imaging Quality ↑ | Color ↑ | Visual Avg. ↑ | Appearance Style ↑ | Human Action ↑ |
|---|---|---|---|---|---|---|---|---|
| CogVideoX | 5B | Base | 0.6124 | 0.6931 | 0.9018 | 0.7358 | 0.2107 | 0.7452 |
| CogVideoX | 5B | + Phys4D | 0.6097 | 0.6886 | 0.8995 | 0.7326 | 0.2184 | 0.7728 |
| Open-Sora V1.2 | 1.1B | Base | 0.5948 | 0.6715 | 0.8842 | 0.7168 | 0.1972 | 0.6872 |
| Open-Sora V1.2 | 1.1B | + Phys4D | 0.5917 | 0.6782 | 0.8821 | 0.7173 | 0.2011 | 0.7023 |
| WAN2.2 | 5B | Base | 0.6286 | 0.7164 | 0.9112 | 0.7521 | 0.2139 | 0.7433 |
| WAN2.2 | 5B | + Phys4D | 0.6351 | 0.7119 | 0.9087 | 0.7519 | 0.2162 | 0.7712 |
Phys4D keeps Visual Avg. nearly unchanged across all three backbones while consistently maintaining or improving Appearance Style and Human Action — open-domain visual quality is preserved, and simulation transfers mainly as physical structure rather than simulator appearance.
Which design choices are responsible for physical knowledge injection? All scores are Physics-IQ on WAN2.2-5B under the same matched switch-frame I2V protocol. Rows are grouped by ablation type and should be compared within each group.
Takeaway: local SFT initializes global trajectory alignment — Stage III is most effective when it starts from Stage II.
Takeaway: gains are not explained by auxiliary heads alone — removing depth and flow still improves over the base model.
Takeaway: physics is best injected at structure-forming timesteps; high-noise adaptation consistently beats full-interval or low-noise updates.
Takeaway: geometry, motion, and warp coupling are complementary — Phys4D relies on coupled geometry–motion learning rather than a single loss term.