Relay Forcing

Checkpoints of Relay Forcing: Real-Time Streaming Video Generation at 1.58 NFE. Every model whose SANA-WM-Bench result appears in the main paper is included, together with the video decoder used for the LTX-2.3-22B models and the training data of the main models. The repository holds 2.12 TB in total.

Main models

Folder Model Used in
wan2.2-ti2v-5b/relay-forcing Relay Forcing, 1.58 NFE Table 1, Table 2, Figure 4, Figure 5(a), Figure 6(a), Table 4 (Full head), Table 5 (Relay Forcing), Table 6 (5-8), Figure 7 (5 x 8)
ltx-2.3-22b/relay-forcing-15s Relay Forcing, 1.58 NFE, 15 s model Table 1, Figure 4, Figure 6(a)
ltx-2.3-22b/relay-forcing-60s Relay Forcing, 1.58 NFE, 60 s model Table 2, Figure 5(b), Figure 6(a)
ltx-2.3-22b/causal-vae-decoder Latent-causal grouped VAE decoder used to decode both LTX-2.3-22B models Table 1, Table 2, Figure 5(b), Figure 6(a)

Ablation models

All ablations use Wan2.2-TI2V-5B.

Folder Model Used in
wan2.2-ti2v-5b/ablations/dmd-4step/relay-homogeneous-critic 4-step vanilla DMD student, relay student with homogeneous critic Figure 6(b) (Relay, Homogeneous), Table 3 (Relay, Rehearsal), Table 5 (4-step)
wan2.2-ti2v-5b/ablations/dmd-4step/relay-offline-critic 4-step vanilla DMD student, relay student with offline critic Figure 6(b) (Relay, Offline)
wan2.2-ti2v-5b/ablations/dmd-4step/causal-homogeneous-critic 4-step vanilla DMD student, block causal student with homogeneous critic Figure 6(b) (Block causal, Homogeneous), Table 3 (Block causal, Rehearsal)
wan2.2-ti2v-5b/ablations/dmd-4step/causal-offline-critic 4-step vanilla DMD student, block causal student with offline critic Figure 6(b) (Block causal, Offline)
wan2.2-ti2v-5b/ablations/dmd-4step/relay-gt-history 4-step vanilla DMD student from a relay teacher trained on ground truth history Table 3 (Relay, Ground truth)
wan2.2-ti2v-5b/ablations/dmd-4step/relay-resampling 4-step vanilla DMD student from a relay teacher trained with Resampling Forcing Table 3 (Relay, Resampling)
wan2.2-ti2v-5b/ablations/head-components/last-block-only Trajectory head reading only the last relay DiT block Table 4 (Last block features only)
wan2.2-ti2v-5b/ablations/head-components/no-step-attention Trajectory head without step attention Table 4 (No step attention)
wan2.2-ti2v-5b/ablations/head-components/no-two-tap-conv Trajectory head without dynamic two-tap convolution Table 4 (No two-tap convolution)
wan2.2-ti2v-5b/ablations/head-components/parallel-linear Parallel linear layers on the final feature Table 4 (Parallel linear layers)
wan2.2-ti2v-5b/ablations/unrolling/1step 1-step student with DMD and GAN Table 5 (1-step)
wan2.2-ti2v-5b/ablations/unrolling/1step-head34 1-step student with a 34-layer single-step head, DMD and GAN Table 5 (1-step + head)
wan2.2-ti2v-5b/ablations/unrolling/2step 2-step student with DMD and GAN Table 5 (2-step)
wan2.2-ti2v-5b/ablations/unrolling/4step-gan 4-step student with DMD and GAN Table 5 (4-step + GAN)
wan2.2-ti2v-5b/ablations/gan-steps/none Trajectory head trained with DMD only Table 6 (None)
wan2.2-ti2v-5b/ablations/gan-steps/steps1-8 Trajectory head with GAN on steps 1 to 8 Table 6 (1-8)
wan2.2-ti2v-5b/ablations/gan-steps/steps1-4 Trajectory head with GAN on steps 1 to 4 Table 6 (1-4)
wan2.2-ti2v-5b/ablations/head-scaling/l3-s4 Trajectory head with 3 layers and 4 unrolled steps Figure 7 (3 x 4)
wan2.2-ti2v-5b/ablations/head-scaling/l3-s8 Trajectory head with 3 layers and 8 unrolled steps Figure 7 (3 x 8)
wan2.2-ti2v-5b/ablations/head-scaling/l3-s16 Trajectory head with 3 layers and 16 unrolled steps Figure 7 (3 x 16)
wan2.2-ti2v-5b/ablations/head-scaling/l5-s4 Trajectory head with 5 layers and 4 unrolled steps Figure 7 (5 x 4)
wan2.2-ti2v-5b/ablations/head-scaling/l5-s16 Trajectory head with 5 layers and 16 unrolled steps Figure 7 (5 x 16)
wan2.2-ti2v-5b/ablations/head-scaling/l8-s4 Trajectory head with 8 layers and 4 unrolled steps Figure 7 (8 x 4)
wan2.2-ti2v-5b/ablations/head-scaling/l8-s8 Trajectory head with 8 layers and 8 unrolled steps Figure 7 (8 x 8)
wan2.2-ti2v-5b/ablations/head-scaling/l8-s16 Trajectory head with 8 layers and 16 unrolled steps Figure 7 (8 x 16)

Figure 6(b), Table 3 and the 4-step row of Table 5 report the mean over three training seeds. Each of those folders holds one of the three seeds.

Files

Every folder contains the following.

  • model-0000x-of-0000N.safetensors and model.safetensors.index.json hold the EMA weights, stored as trained. All reported results are generated with these EMA weights. Where the model has a trajectory head, the head is part of the same state dict.
  • config.yaml is the training configuration. Base model files are given relative to their Hugging Face repositories. Initialization sources, which are intermediate checkpoints, are set to null.

The three main models also contain training_state/. It is the PyTorch Distributed Checkpoint of the final training step, with the student, its EMA, the fake score model, the optimizer states, the data loader and random number generator states, and the per-rank streaming chain states. Resuming requires the same number of ranks and data loader workers per rank. These are 16 ranks with 2 workers for Wan2.2-TI2V-5B, 32 ranks with 2 workers for the 15 s LTX-2.3-22B model, and 32 ranks with no worker processes for the 60 s LTX-2.3-22B model.

manifest.json lists every file with its size and SHA-256.

Training data

data/ holds the training windows of the main models as preencoded latents.

Folder Used by Content Size
data/wan2.2-ti2v-5b-229f every wan2.2-ti2v-5b/ model 10,080 windows of 229 frames at 960x544 and 24 fps. Wan2.2 VAE video latents, umT5 text features and per-frame cameras 126.2 GB in 26 shards
data/ltx-2.3-22b-231f ltx-2.3-22b/relay-forcing-15s 10,080 windows of 231 frames at 960x544 and 24 fps. LTX-2.3 video latents in 33-frame groups, first-image latents, LTX-2.3 text features and per-frame cameras 126.7 GB in 26 shards
data/ltx-2.3-22b-957f ltx-2.3-22b/relay-forcing-60s 2,476 windows of 957 frames at 960x544 and 24 fps. LTX-2.3 video latents in 33-frame groups, first-image latents, LTX-2.3 text features and per-frame cameras 68.1 GB in 14 shards

Each folder holds uncompressed tar shards and files.jsonl, which lists every member with its size, SHA-256 and shard. Extract all shards of a folder into that folder.

cd data/ltx-2.3-22b-231f && for shard in shard-*.tar; do tar -xf "$shard"; done

manifests/ then holds the training manifests, whose paths are relative to the manifest itself. The config.yaml of each model points data.args.index_file to these manifests relative to the repository root. The windows are encoded from SolarWM-Data, and their use is subject to the terms of SolarWM-Data and of its sources.

Loading

import json
from pathlib import Path

from huggingface_hub import snapshot_download
from safetensors.torch import load_file

folder = "wan2.2-ti2v-5b/relay-forcing"
root = Path(snapshot_download("oliveryanzuolu/Relay-Forcing", allow_patterns=[f"{folder}/model*", f"{folder}/config.yaml"])) / folder
index = json.loads((root / "model.safetensors.index.json").read_text())
state_dict = {}
for shard in sorted(set(index["weight_map"].values())):
    state_dict.update(load_file(root / shard))

The keys match the module declared under models.backbone in config.yaml, or under models.video_decoder for the decoder. The training and inference code will be released separately.

Base models

  • Wan2.2-TI2V-5B models start from the camera-controlled SolarWM-5B and use the VAE and text encoder of Wan2.2-TI2V-5B.
  • LTX-2.3-22B models start from LTX-2.3 and use its text and image encoders with Gemma 3 12B. Videos are decoded with the VAE of LTX-2.5 and the decoder weights in ltx-2.3-22b/causal-vae-decoder.
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for oliveryanzuolu/Relay-Forcing

Finetuned
(128)
this model