World models (systems that synthesize realistic video sequences from an initial image and a set of actions) are becoming central to embodied AI, simulation, and robotics research. The core challenge is scaling these systems to generate minute-long, high-resolution video without requiring prohibitively large clusters for both training and inference. Most competitive open-source baselines either require multi-GPU inference or sacrifice resolution to stay within compute budgets.

NVIDIA’s SANA-WM directly targets these bottlenecks. Built on the SANA-Video codebase and available through the NVlabs/Sana GitHub repository, it is a 2.6B-parameter Diffusion Transformer (DiT) trained natively for one-minute generation at 720p with metric-scale 6-DoF camera control. It supports three single-GPU inference variants: a bidirectional generator for high-quality offline synthesis, a chunk-causal autoregressive generator for sequential rollout, and a few-step distilled autoregressive generator for faster deployment. The distilled variant denoises a 60-second 720p clip in 34 seconds on a single RTX 5090 with NVFP4 quantization.

The Architecture: Four Core Design Decisions

1. Hybrid Linear Attention with Gated DeltaNet (GDN)

Standard softmax attention has memory and compute complexity that grows quadratically with sequence length — a serious problem when generating 961 latent frames for a 60-second video at 720p. SANA-Video, the predecessor, used cumulative ReLU-based linear attention, which maintains a constant-size recurrent state. However, this has no decay mechanism: all past frames accumulate with equal weight, causing drift over minute-scale sequences.

SANA-WM replaces most attention blocks with frame-wise Gated DeltaNet (GDN). Unlike token-wise GDN used in language models, SANA-WM’s frame-wise variant processes one entire latent frame per recurrent step. The GDN update rule incorporates a decay gate γ (which down-weights stale past frames) and a delta-rule correction (which updates only the residual between the target value and the current state prediction), keeping the recurrent state at a constant D×D size regardless of video length.

To stabilize training, the research team introduces an algebraic key-scaling approach: keys are scaled by 1/√(D·S), where D is the head dimension and S is the number of spatial tokens per frame. This ensures the spectral norm of the transition matrix remains bounded and eliminates the NaN divergence events observed with standard L2 key normalization (1/√D) or no scaling at all, both of which triggered NaN events at steps 16 and 1, respectively.

The final backbone interleaves 15 frame-wise GDN blocks with 5 softmax attention blocks (at layers 3, 7, 11, 15, and 19) across 20 total transformer blocks. The softmax blocks provide exact long-range recall where GDN’s recurrence alone is insufficient.

2. Dual-Branch Camera Control

Camera-controlled world modeling requires the model to faithfully follow a continuous 6-DoF trajectory, not just align with a text description of motion. SANA-WM uses two complementary branches that operate at different temporal rates:

• Coarse branch (UCPE attention): Operates at the latent-frame rate. For each latent token, it computes a ray-local camera basis from the camera-to-world pose and intrinsics, then applies a Unified Camera Positional Encoding (UCPE) to the geometric channels of each attention head. This captures global trajectory structure across the full sequence.
• Fine branch (Plücker mixing): Addresses a compression mismatch. Each latent token summarizes eight raw frames, each with its own distinct camera pose. The fine branch computes pixel-wise Plücker raymaps (a 6D representation: ray direction d and moment o×d) from all eight raw frames within one VAE temporal stride, packs them into a 48-channel tensor, and injects this embedding after each self-attention output via a zero-initialized projection. This restores intra-stride camera motion that the coarse branch cannot see at latent-frame resolution.

Ablations on OmniWorld show that neither branch alone matches the dual approach: UCPE-only achieves a Camera Motion Consistency (CamMC) of 0.2453, while UCPE + Plücker mixing reaches 0.2047.

3. Two-Stage Generation Pipeline

Stage-1 SANA-WM outputs, while spatiotemporally consistent, can contain structural artifacts over long sequences. A second-stage refiner, initialized from the 17B LTX-2 model with rank-384 LoRA adapters fine-tuned on paired synthetic and real video data, corrects these artifacts. It uses truncated-σ flow matching: stage-1 latents are perturbed with a large starting noise (σ_start = 0.9), and the refiner learns to map this noisy input toward the high-fidelity target. Only three Euler denoising steps are needed at inference. The refiner reduces long-horizon visual drift (ΔIQ) from 3.79 to 1.17 on the Simple-Trajectory split, and from 3.09 to 0.31 on the Hard-Trajectory split.

4. Robust Data Annotation Pipeline

Training camera-controlled video generation requires metric-scale 6-DoF pose annotations, the information not available in standard video datasets. The research team modified VIPE (a camera-pose annotation engine) by replacing its depth backend with Pi3X (for long-sequence-consistent depth) fused with MoGe-2 (for accurate per-frame metric scale). They also extended the bundle adjustment stage to treat focal lengths and principal points as per-frame variables rather than shared global intrinsics, enabling more robust annotation on internet video with varying focal lengths.

The resulting pipeline processes seven training corpus entries drawn from multiple open-source sources: SpatialVID-HQ (real, 10s clips), DL3DV real clips (10s), DL3DV GS Refined synthetic clips (60s, rendered via 3D Gaussian Splatting), OmniWorld (synthetic, 60s), Sekai Game (synthetic, 60s), Sekai Walking-HQ (real, 60s), and MiraData (real, 60s). This yields a total of 212,975 clips with metric-scale pose annotations. The LTX2-VAE used for compression is 2.0× smaller than ST-DC-AE and 8.0× smaller than Wan2.1-VAE, which directly improves training and inference efficiency.

For DL3DV, which contains static 3D scene captures rather than native one-minute videos, the research team fit one FCGS 3D Gaussian Splatting reconstruction per scene, designed diverse one-minute camera paths, rendered long videos with known intrinsics and extrinsics, and then refined the rendered outputs with DiFix3D to reduce splatting artifacts.

Training Strategy and Infrastructure

SANA-WM’s compute involves two phases on 64 H100 GPUs. First, before DiT training, the team adapts the LTX2 VAE to the SANA-Video SFT training data in approximately 50K steps, taking roughly 3.5 days. The main DiT training then follows a four-stage progressive schedule lasting approximately 15 days:

• Stage 1 (~2.75 days): Adapt the pre-trained SANA-Video model to the frame-wise GDN architecture on short (5s) video clips. This replaces cumulative linear attention with the recurrent GDN blocks on a cheaper, short-horizon training regime where failure modes are easier to diagnose.
• Stage 2 (~2 days): Introduce hybrid attention by replacing every fourth GDN block with a standard softmax attention block on the same short-clip setting, improving the efficiency–quality trade-off.
• Stage 3 (~8 days): Extend training to 961-frame (60-second) sequences and incorporate Dual-Branch Camera Control. Context-Parallel (CP=2) sharding distributes the latent sequence across GPUs using prefix-sum composition of GDN transition matrices — a mathematically exact parallelization strategy requiring minimal communication overhead.
• Stage 4 (~2.5 days): Fine-tune a chunk-causal variant for autoregressive rollout, then apply self-forcing distillation to reduce sampling to four denoising steps. Attention-sink tokens and local temporal windows are added to the softmax attention layers to keep memory and per-chunk latency constant during long rollouts.

Custom fused Triton kernels for GDN scan and gate operations contribute approximately 1.5× to 2× efficiency gains throughout training.

Benchmark Results

The research team introduces a purpose-built 60-second world-model benchmark with 80 initial scenes generated by Nano Banana Pro across four scene categories game, indoor, outdoor-city, and outdoor-nature (20 per category). Each paired with Simple and Hard camera trajectory splits. The main evaluation uses each model’s multi-step, undistilled autoregressive setting.

On this benchmark, SANA-WM with the second-stage refiner achieves the following across both splits:

• Camera accuracy (Simple / Hard): Rotation error (RotErr) of 4.50° / 8.34°; Translation error (TransErr) of 1.39 / 1.39; CamMC of 1.41 / 1.44 — the best among all compared methods, including LingBot-World (14B+14B parameters, 8 GPUs) and HY-WorldPlay (8B parameters, 8 GPUs).
• Visual quality: 80.62 / 81.89 VBench Overall on Simple / Hard splits, comparable to LingBot-World (81.82 / 81.89) while generating 720p outputs on a single GPU per clip.
• Throughput: 22.0 videos/hour on 8 H100s with the full pipeline (refiner included), compared to 0.6 videos/hour for LingBot-World — a 36× throughput advantage.
• Memory: The full pipeline fits in 74.7 GB, within the 80 GB H100 budget. Stage-1-only inference fits in 51.1 GB.
• Temporal stability: After refinement, ΔIQ (imaging quality degradation from first to last 10-second window) drops to 1.17 on Simple and 0.31 on Hard, compared to 23.59 and 25.88 for HY-WorldPlay.

Marktechpost’s Visual Explainer

# Clone the repo
git clone https://github.com/NVlabs/Sana.git
cd Sana && ./environment_setup.sh sana

Key Takeaways

• NVIDIA’s SANA-WM generates 60-second, 720p, camera-controlled videos on a single GPU — trained in ~18.5 days on 64 H100s with only 212,975 public video clips.
• A hybrid Gated DeltaNet + softmax attention backbone keeps the recurrent state at a constant D×D size regardless of video length, solving the memory explosion that makes minute-scale generation impractical with standard softmax attention.
• Dual-branch camera control — UCPE at the latent-frame rate and Plücker mixing at the raw-frame rate — brings CamMC down to 0.2047, the best among all compared methods including models 5× larger.
• A second-stage refiner initialized from 17B LTX-2 with rank-384 LoRA cuts long-horizon visual drift (ΔIQ) from 3.09 to 0.31 on Hard trajectories using just 3 Euler denoising steps.
• At 22.0 videos/hour on 8 H100s, SANA-WM + refiner delivers 36× higher throughput than LingBot-World (14B+14B, 8 GPUs) at comparable VBench visual quality scores.

Check out the Paper, GitHub Repo and Project Page. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us