World2Act
Latent Action Post-Training from World Model Dynamics
TL;DR
World2Act aligns WM video dynamics and robot actions in a shared latent space, then trains a lightweight residual policy on top of a frozen VLA. Latent supervision improves control without decoding WM rollouts into pixels or updating the WM and VLA backbones.
Abstract
World Models (WMs) offer a promising mechanism for post-training Vision-Language-Action (VLA) policies by providing dynamics priors that improve generalization under task and scene variation. However, most WM-based post-training methods rely on pixel-space supervision, making policies sensitive to visual artifacts introduced by imperfect WM rollouts. We present World2Act, a latent-space post-training framework that transfers WM dynamics to the VLA policy without pixel-space supervision. World2Act operates in two stages: (i) it induces a shared video–action latent space by contrastively aligning WM-dynamics latents with action embeddings, and (ii) it post-trains the VLA by guiding policy action representations toward WM-imagined dynamics rather than decoded pixels. Built on GR00T-N1.6, World2Act delivers absolute success-rate gains of up to +2.5 percentage points on simulation benchmarks (RoboCasa, LIBERO, Bridge-SIMPLER) and +5.0 percentage points on a real robot over finetuned VLA baselines. Notably, it outperforms pixel-space WM supervision by up to +6.0 percentage points, including on LIBERO where pixel supervision degrades the baseline, suggesting that latent WM dynamics offer a more stable WM-based post-training alternative to pixel-space transfer. Code and checkpoints will be released to support future research on WM-based post-training.
Motivation
Existing world-model post-training still depends heavily on pixel-space supervision. World2Act instead transfers dynamics in latent space.
Transfer WM dynamics without pixel-space supervision
Pixel-space methods derive pseudo-actions or rewards from imagined videos, which can turn visual artifacts into noisy supervision. World2Act aligns action representations directly with WM latent dynamics. On RoboCasa, increasing the number of post-training trajectories from 0 to 1,000 raises success from 70.1% to 72.6%.
Method
The method is a two-stage pipeline: latent alignment first, then residual-policy post-training using transferred world-model dynamics.
Main pipeline
Stage 1 — Latent alignment: Video and action adapters map Cosmos-Predict2 WM latents and expert action chunks into a shared 32-dimensional space. An action decoder reconstructs robot actions; bidirectional InfoNCE aligns temporally corresponding video and action chunks.
Stage 2 — Residual post-training: We freeze the WM, GR00T-N1.6-ft, and Stage-1 modules, then train a lightweight residual policy to correct VLA action latents toward WM targets. The frozen decoder maps corrected latents back to executable action chunks. Training uses neither environment rewards nor success labels.
Experimental Results
World2Act improves GR00T-N1.6-ft across RoboCasa, LIBERO, Bridge-SIMPLER, and five real-world robot tasks. Gains below are absolute percentage-point changes over the finetuned VLA baseline.
RoboCasa
+2.5 points over GR00T-N1.6-ft (70.1%).
LIBERO
+1.1 points over GR00T-N1.6-ft (97.0%).
Bridge-SIMPLER
+1.4 points over GR00T-N1.6-ft (57.7%).
Real-world success
+5.0 points over GR00T-N1.6-ft (18.0%), across five tasks.
RoboCasa
Starting from the same GR00T-N1.6-ft checkpoint, World2Act improves success from 70.1% to 72.6%, compared with 70.5% for DreamGen. Each post-training method uses 1,000 imagined trajectories and 25,000 optimizer updates per simulator.
LIBERO
World2Act improves average success from 97.0% to 98.1%, while DreamGen reduces it to 92.1%. This is a +6.0 percentage-point advantage over DreamGen on the same backbone. World2Act finishes within 0.4 percentage points of Cosmos Policy (98.5%).
Bridge-SIMPLER
Across seven WidowX manipulation tasks, World2Act reaches 59.1% average success, compared with 57.7% for GR00T-N1.6-ft and 58.4% for DreamGen. The benchmark extends SIMPLER with open-drawer, close-drawer, and eggplant-in-sink tasks.
Real-world evaluation
On a Franka Research 3, World2Act achieves 23% average success versus 18% for GR00T-N1.6-ft and 17% for DreamGen. Evaluation covers five tasks with 20 trials per task: cup-to-plate placement, bowl pickup, closing a microwave, turning on a stove, and pushing a cube. Post-training uses 20 real demonstrations per task and 100 WM-generated latent trajectories.
Latent versus pixel supervision
With the same GR00T-N1.6-ft initialization, World2Act reaches 72.6% ± 1.1% success, compared with 71.0% for VLA-RFT, 70.5% for DreamGen, and 69.8% for Ctrl-World. Values are means over five independent runs; the table reports standard deviations. A residual behavior-cloning variant reaches 70.4%.
Cross-task generalization
RoboCasa is split into 12 seen and 12 unseen tasks; evaluation uses only the unseen split. Even when WM pretraining is restricted to seen tasks, World2Act outperforms DreamGen by 1.36 percentage points. Unseen-task success increases with the number of seen training tasks.
Qualitative Results
Explore WM imagination, real-robot execution, and action-latent alignment. Select a figure for a closer look, or play the individual demonstrations below.
World Model and Real-World Execution
Real-robot demonstrations
Watch individual real-world task executions.
Summary: World2Act
Bridging the gap between WM and VLA via latent alignment
Key Contributions
Pixel-Space Supervision Analysis: We identify sensitivity to visual artifacts in imagined rollouts as a limitation of pixel-space WM post-training.
Shared Video–Action Space: Contrastive alignment connects temporally matched WM video-dynamics latents and robot action chunks.
Efficient Residual Post-Training: A lightweight residual policy refines action latents while the WM, VLA, adapters, and action decoder remain frozen.
Simulation and Real Robots: World2Act improves GR00T-N1.6-ft on RoboCasa, LIBERO, Bridge-SIMPLER, and five real-world tasks.