Wan2.2-5B trunk + both action→motion encoders: ground truth vs. rollout

mot_df_vt_wan5b_vmx — the full stack. A pretrained Wan2.2-Fun-5B-Control DiT is shared by both Mixture-of-Transformers experts (per-modality LoRA on a frozen trunk), each branch is conditioned by its own frozen Stage-1 action→motion encoder — the tactile one predicting gel flow, the visual one predicting camera-image flow — and the tactile stream is predicted as motion + residual: the previous gel latent is transported by the action-derived flow and diffusion generates only the correction.

359.6M trainable / 5.3B total Wan2.2 VAE latents, C=48, 16×16 epoch 76 (best ep 66, val 0.3363) vmx + amx, both frozen tactile target: motion+residual

Where prediction starts: each row is an (episode, start) pair on the 6 fps grid. Frames [start, start+8) — 2 latent frames — are given as clean context; prediction begins at fps6 frame start+8 (decoded frame 5 of the short clips). Short cells predict ~1.3 s; long cells continue autoregressively to 16 s with the model's own output as context.

Ground truth vs prediction, frame by frame

Architecture in one paragraph

Both experts run in lockstep: each cross-attends its own action context, then their tokens are concatenated (256 video + 512 tactile = 768) for a single joint self-attention, then a per-expert FFN. The joint mask is temporally causal across modalities — camera frame t attends tactile frames ≤ t and vice versa. Each modality carries its own noise scheduler with independently drawn per-frame noise levels, so the two branches sit at unrelated points of their diffusion schedules. On the tactile side a frozen action-motion encoder reads the inverse-action displacement map on the gel plane plus a calibrated physics flow prior and injects a zero-initialised motion feature into the future tactile tokens; a frozen latent-flow probe scores the predicted clean latents against measured AllTracker optical flow.

The action, projected into pixel space

This is the half of the action the video expert cross-attends — action[0:18]. Both sensors' gel triads are projected into the middle camera and carried as absolute image coordinates: 2 sensors × 3 points × (u, v, z), with u,v normalised to [0,1] and z in metres. The tactile expert never sees this; it gets the body-frame SE(3) step instead.

Rendered at the 6 fps action grid over the 16 s long-rollout span; the short-window rows are the first 16 frames of these. The triads landing on the physical sensors in every frame is also the calibration check — a forward-project/triangulate round trip would return ~0 residual even with wrong extrinsics, so only the image can falsify it.

Ten additional held-out rollouts

Beyond the 6 shared rows, ten further held-out windows were evaluated at both horizons — 0510_ep005 @ 14, 42 and 0510_ep006 @ 100, 250, 300, 350, 500, 550, 620, 680. (ep005 holds only 153 frames at 6 fps, so a full 96-frame rollout only fits from early starts; hence eight of the ten come from the longer ep006.) They are selectable in the strip viewer above.

These ten are a broader sample of the same held-out episodes, and they land close to the original six (short view 23.81 vs 24.36; long view 19.59 vs 19.87) — the headline numbers are not an artifact of three lucky start offsets. Long-horizon tactile is ~3 dB better here (29.35 vs 26.43), but that is a method difference, not a data one: these long rollouts use keep-2 block-autoregressive sampling — committing both predicted latent frames per window instead of re-predicting every frame — which we measured earlier to be worth about +1 dB tactile at half the inference cost, and which also halves the number of closed-loop handoffs that the motion+residual target accumulates drift across. Treat the long-cell comparison against the original six as method-confounded.

Metrics

Decoded-pixel PSNR (mean ± sem over n = 6 rollouts per cell), GT vs. rollout, through the Wan2.2 VAE — tactile streams via the tactile LoRA, the camera view via the base VAE. tac is the mean of the two sensors.

Against the 240M same-target baseline

mot_df_vt_tacresidual is the closest comparison: the same motion+residual tactile target and the same 12 rollout rows, but a from-scratch 240M trunk and no visual motion encoder. So the delta below bundles two changes — the pretrained Wan trunk and the vmx injection.

Δ is this run − the 240M baseline, in dB. Read: the camera view reaches 24.36 dB on held-out short windows — the best view number of any model trained on this data (previous best 24.06). Tactile short-horizon is a touch below the 240M twin, and long-horizon tactile stays in the mid-20s: that is the known cost of the motion+residual target, whose sequential warp reconstruction accumulates drift over a 16 s closed loop — a property of the target, not of the motion encoders. Absolute-target runs score ~31 dB there.

Drift: how error grows over the 16 s rollout

Long rollouts are autoregressive — the model's own prediction becomes the next step's context, so error compounds. Per-latent-frame MSE, averaged over the 6 rollouts in each split, shaded ±1 sd. The first 2 frames (shaded) are ground-truth context, not predictions. Frame 0 is exactly zero; frame 1 is small but non-zero because the continuous GT reference is stitched across per-window VAE encodes while the rollout is seeded from a single encode — the two are both ground truth and differ slightly past frame 0. That offset is a property of the reference, not rollout error.

Short window — 2.7 s, 2 latent frames given, 2 predicted

Left half of each clip is ground truth, right half is the model's rollout.

Long rollout — 16 s autoregressive, predictions fed back

Browse every rollout

How it was run

RUN=.../runs_mot/mot_df_vt_wan5b_vmx
CKPT=$RUN/checkpoints/last.ckpt        # epoch 76 (best ep 66, val 0.3363)

# short window (one 16-frame window; 2 latent frames of context)
python -m vm_diffusion.scripts.infer_windows --ckpt $CKPT --config $RUN/config.yaml \
    --windows motherboard_0510_episode_005:0 ... --out_dir $RUN/eval/short_test

# long rollout (sliding window, keep-1, driven by the episode's real action sequence)
python -m vm_diffusion.scripts.long_rollout_mot --config $RUN/config.yaml --ckpt $CKPT \
    --episode motherboard_0510_episode_005.pt --start 0 --seconds 16 --fps 6

# decode both through the Wan2.2 VAE (tactile LoRA for tl/tr, base VAE for view)
python decode_rollout_wan22.py --rollout-dir <cell dir> --fps 6

Caveats

• Six rollouts per cell (two episodes × three start offsets). Treat the sem as indicative, not as a significance test.
• This is a held-out-episode split, not a generalization split. The test episodes are the same board and the same session as many training episodes, so these numbers measure "unseen trajectory, seen conditions".
• PSNR rewards blur. A model that predicts a smooth mean-ish gel image scores well. Watch the clips, not only the table — the tactile texture is where the difference shows.
• The checkpoint is last.ckpt, not best-*. Validation loss for this model is ~56% the frozen-flow-probe term, which oscillates and does not track denoising quality, so selecting on it would pick a checkpoint for the wrong reason. A fixed step count is also the fairer comparison against the baseline.
• Long-rollout GT reference is stitched across per-window VAE encodes, so it differs slightly from any single encode past frame 0. The rollout seed itself is exact.
• The projected action is not always in frame. Across these rows the gel centre is in-frame 100% of the time for most, but as low as 88.5% (train row 0). When a sensor leaves the frame its action image is blank, so the video expert receives no positional signal for that step — while the tactile expert's body-frame SE(3) half is unaffected. The two halves of the action degrade independently.

Diffusion forcing (per-frame independent noise levels), pyramid sampling with 100 denoise steps, 36-D causally-masked action conditioning, 3D axial RoPE. Latents are Wan2.2 VAE, 48 channels on a 16×16 grid at 6 fps.