Meta tags:
description=A comprehensive roadmap on video generation evolving toward world models, introducing a four-generation taxonomy and future directions across robotics, autonomous driving, and embodied AI.;
author=Jingtong Yue, Ziqi Huang, Zhaoxi Chen, Xintao Wang, Pengfei Wan, Ziwei Liu;

Headings (most frequently used words):

world, generation, and, model, video, models, as, in, with, taxonomy, simulation, of, the, visual, roadmap, definition, navigation, real, foundation, knowledge, simulating, artificial, intelligence, abstract, introduction, problem, methods, hierarchical, future, everything, everywhere, anytime, citation, acknowledgements, references, motivation, position, scope, four, generations, from, to, mode, faithfulness, accurate, interactiveness, controllability, interactive, dynamics, planning, time, complex, prediction, stochasticity, multi, scale, low, probability, insight, beyond, two, complementary, development, directions, applications, societal, impact, conclusion, outlook, centric, spatial, dynamic, aware, general, scene, robotics, autonomous, driving, gaming,

Text of the page (most frequently used words):
and (148), world (92), generation (74), the (72), #models (65), video (63), for (37), model (32), with (25), physical (23), planning (22), #control (21), real (19), 2025 (18), 2024 (18), simulation (18), from (18), this (17), driving (16), that (16), these (16), they (16), can (15), figure (15), such (14), diffusion (13), physics (13), text (13), like (13), reasoning (12), motion (12), time (11), autonomous (11), modeling (11), systems (11), visual (10), general (10), spatial (10), temporal (10), across (10), dynamics (10), faithfulness (10), navigation (9), scale (9), foundation (9), dynamic (9), based (9), how (9), interaction (9), robotics (9), long (9), interactiveness (9), controllable (8), multi (8), learning (8), multimodal (8), consistent (8), generate (8), simulate (8), overview (8), while (8), are (7), through (7), knowledge (7), both (7), conditions (7), realism (7), vision (7), signals (7), interactive (6), controllability (6), simulators (6), understanding (6), actions (6), scene (6), virtual (6), evolving (6), embodied (6), agents (6), their (6), environments (6), gaming (6), goal (6), scenes (6), causal (6), methods (6), engines (5), high (5), data (5), fidelity (5), via (5), generative (5), aware (5), camera (5), grounded (5), open (5), intelligence (5), roadmap (5), into (5), two (5), physically (5), internal (5), between (5), term (5), domains (5), laws (5), multiple (5), but (5), taxonomy (5), also (5), stochasticity (5), stage (5), sequences (5), realistic (5), maps (5), short (5), along (5), which (5), capabilities (5), yue (4), genie (4), cosmos (4), prediction (4), process (4), aether (4), language (4), self (4), fields (4), simulating (4), artificial (4), paper (4), scientific (4), act (4), within (4), worlds (4), capable (4), under (4), future (4), predict (4), driven (4), digital (4), decision (4), scales (4), evolution (4), not (4), each (4), three (4), integrate (4), intrinsic (4), external (4), conditioned (4), depth (4), perception (4), improve (4), geometry (4), coherence (4), trajectories (4), limited (4), inputs (4), consistency (4), clips (4), axis (4), four (4), generations (4), outputs (4), mental (4), jingtong (3), all (3), infinite (3), game (3), transformer (3), large (3), condition (3), drive (3), agent (3), precise (3), simulator (3), 2023 (3), autoregressive (3), framework (3), wan (3), sora (3), arxiv (3), any (3), environment (3), rare (3), outcomes (3), different (3), providing (3), could (3), learn (3), define (3), reality (3), parallel (3), together (3), oriented (3), bridge (3), complementary (3), yet (3), diverse (3), evolve (3), structure (3), local (3), global (3), coherent (3), plan (3), static (3), collectively (3), where (3), key (3), processes (3), events (3), low (3), probability (3), true (3), represent (3), interactions (3), complex (3), behavior (3), making (3), incorporate (3), enable (3), adapt (3), first (3), sketches (3), toward (3), support (3), visually (3), produce (3), application (3), highlights (3), capability (3), semantic (3), captures (3), parker (2), holder (2), new (2), decart (2), miragelsd (2), zero (2), oasis (2), hunyuan (2), gamecraft (2), xuanchi (2), dreams (2), synthetic (2), vista (2), versatile (2), drivingworld (2), gaia (2), view (2), videoagent (2), policy (2), transfer1 (2), conditional (2), geometric (2), unified (2), pandora (2), towards (2), natural (2), gen3c (2), cameractrl (2), wisa (2), phyt2v (2), dit (2), advancing (2), efficient (2), point (2), sketchvideo (2), sketch (2), adding (2), image (2), magi (2), hunyuanvideo (2), drivedreamer (2), peekaboo (2), trailblazer (2), trajectory (2), physgen (2), diffusion4d (2), ltx (2), latent (2), would (2), article (2), author (2), ziqi (2), huang (2), zhaoxi (2), chen (2), xintao (2), wang (2), pengfei (2), ziwei (2), liu (2), precision (2), creative (2), will (2), perceive (2), reason (2), influence (2), everyday (2), climate (2), biology (2), example (2), humans (2), applications (2), impact (2), major (2), accurate (2), become (2), testing (2), reinforcement (2), space (2), representations (2), imagine (2), forming (2), third (2), countless (2), paradigm (2), active (2), broad (2), molecular (2), training (2), serving (2), enabling (2), nwm (2), unify (2), probabilistic (2), exhibit (2), audio (2), purpose (2), futures (2), simulated (2), comprehensive (2), progress (2), ended (2), playable (2), reactive (2), transition (2), outline (2), hierarchical (2), insight (2), macroscopic (2), mesoscopic (2), millisecond (2), level (2), microscopic (2), fourth (2), stochastic (2), only (2), about (2), what (2), deterministic (2), random (2), capturing (2), accidents (2), weather (2), biological (2), variations (2), spatiotemporal (2), marks (2), state (2), over (2), maintaining (2), changes (2), among (2), allowing (2), users (2), provides (2), works (2), rendering (2), instructions (2), show (2), early (2), instruction (2), synthesis (2), supporting (2), commands (2), embed (2), causality (2), prior (2), information (2), approaches (2), demonstrate (2), generalize (2), cross (2), attention (2), bridging (2), controlnet (2), illustrates (2), strategies (2), injection (2), represents (2), now (2), horizon (2), responsiveness (2), interactivity (2), allows (2), subject (2), remain (2), spatially (2), priors (2), object (2), basic (2), however (2), maintain (2), alignment (2), presents (2), evolves (2), dimensions (2), one (2), main (2), others (2), continuous (2), corresponding (2), focuses (2), gradual (2), intelligent (2), appealing (2), task (2), videos (2), make (2), cup (2), water (2), mode (2), temporally (2), because (2), genuine (2), definition (2), images (2), may (2), specific (2), objective (2), during (2), well (2), characteristics (2), simulates (2), intention (2), university (2), rights, reserved, bar, frontier, latency, valevski, universe, jiaqi, hybrid, history, scalable, shenyuan, generalizable, xiaotao, constructingworld, gpt, russell, mengjiao, memory, augmented, yanjiang, action, joint, denoising, alhaija, adaptive, jiannan, states, informed, hao, exploration, controlled, jing, assistant, qiyao, llm, guided, iterative, refinement, weikang, pseudo, gaussian, dense, tracking, zekai, shader, fenglin, editing, lvmin, hansi, weijie, systematic, brooks, xiaofeng, jain, masked, duo, kurt, shaowei, rigid, body, hanwen, fast, haoxin, videocrafter1, quality, zangwei, democratizing, production, haochen, realtime, references, thank, jiaming, song, discussions, valuable, feedback, acknowledgements, yue2025video, title, journal, preprint, 2511, 08585, year, you, find, work, useful, please, consider, citing, citation, concludes, converging, synergistic, paradigms, reshape, ultimately, authors, foresee, laying, next, era, conclusion, outlook, profoundly, science, industry, life, failure, cases, ecological, astronomy, might, microbial, growth, atmospheric, shifts, animal, habitats, helping, anticipate, extinction, risks, disasters, laboratories, transform, experiment, design, age, societal, paths, direction, pursues, perfect, pass, turing, test, instruments, hypothesis, silico, experimentation, research, develops, focus, less, pixel, more, predictive, let, optimize, policies, before, execution, lines, unite, agency, sees, spawn, realities, same, initial, mere, observers, creators, development, directions, promise, structures, chemistry, generating, testbeds, drug, protein, medicine, further, everything, everywhere, anytime, linking, coordinating, sensory, streams, essence, cousins, endless, just, renderers, reasoners, own, universes, demonstrates, maturing, interacting, imagined, rendered, animation, grow, layout, sections, deepens, abilities, beyond, planetary, cosmological, daily, human, detail, introduces, likely, happen, black, swan, anomalies, preserving, plausibility, operate, leap, forecasting, horizons, response, factors, extended, even, dependencies, resulting, obeys, newtonian, fluid, principles, modify, objects, observe, instant, responses, convergence, jepa, ideal, sandbox, treat, engine, manipulate, characters, controller, prototypes, create, form, gamengen, road, safe, layouts, bounding, boxes, later, help, visualize, observations, robot, plans, goals, pad, unisim, follow, turn, left, zoom, marking, passive, person, views, enhance, clouds, das, use, sparse, cues, combine, backbones, modules, render, visuals, techniques, transformers, inject, efficiently, shown, five, representative, subfigures, correspond, method, modal, concatenation, addition, respectively, decisive, shift, flexible, modalities, semantically, richer, conditioning, input, elementary, localized, single, introduce, additional, guide, enforce, dependent, offering, flexibility, vae, frames, continuity, appearance, tasks, imitation, still, lack, videocrafter, centric, establishes, foundations, chrono, logical, horizontal, categorizing, existing, vertical, additionally, colors, used, indicate, genera, tions, chronological, develop, rather, than, sequentially, continues, improving, hierarchy, discussed, below, grouped, illustrate, matures, secondary, category, viewed, progression, axes, dominant, ability, emphasizes, remains, superficial, weak, absent, rely, heavily, good, showing, doing, second, rise, begin, modes, longer, improved, centered, stability, them, simple, executing, sequence, pick, pour, forms, pure, emerges, defining, infinitely, extending, stimuli, handle, entity, given, command, coffee, autonomously, involving, heating, grinding, beans, pouring, logic, becomes, frame, integrates, common, uncertainties, phenomena, extremes, distributions, achieving, behaves, imagining, plausible, content, independent, interface, guides, differs, concept, theoretical, tool, assess, whether, truly, exhibits, achieves, adds, generates, must, take, audios, combinations, including, leveraged, intermediate, other, generated, depending, downstream, defined, cousin, without, replicating, instances, effectively, mirrors, mdp, pomdp, performing, subjective, inference, problem, survey, structured, emphasize, visualization, scope, distinct, guidance, distinguish, perspectives, objectively, cognition, environmental, embodies, essential, position, count, was, derived, searches, conducted, using, fixed, keyword, combination, google, scholar, annual, papers, articles, counts, aim, various, stands, out, inherently, advances, allow, reflects, motivation, emphasized, our, range, foundational, shot, generalization, core, introduction, merely, creating, constructing, obey, conceptualize, modern, fusion, components, implicit, renderer, visualizes, simulations, tracing, propose, finally, representing, emergence, abstract, repo, project, lead, carnegie, mellon, nanyang, technological, kling, team, kuaishou, technology,

Text of the page (random words):
te a broad range of environments across multiple spatial and temporal scales the figure highlights four foundational characteristics real time responsiveness stochasticity multi scale planning and intrinsic physical faithfulness these collectively support the long term goal of zero shot generalization 1 1 motivation world models aim to simulate the physical and causal structure of reality driving progress in robotics autonomous driving and embodied ai among various approaches video generation stands out because vision inherently captures both spatial and temporal information advances in diffusion and autoregressive models now allow high fidelity long term video synthesis that reflects real world physics making video generation a natural foundation for world modeling figure 2 overview of annual papers and articles paper counts in video generation world model fields the article count was derived from searches conducted using the fixed keyword combination video generation and world model from google scholar and arxiv 1 2 position we distinguish between two complementary perspectives the physical world model which objectively simulates external dynamics and the mental world model which captures internal cognition intention and reasoning the physical axis represents laws of motion and environmental evolution while the mental axis embodies perception and intention understanding both is essential for bridging simulation and intelligent behavior figure 3 the characteristics of physical world model and mental world model this figure highlights the distinct inputs internal processes and outputs as well as the interaction through perception planning and guidance between the physical world model and the mental world model 1 3 scope this survey focuses on how video generation evolves toward comprehensive world models capable of learning structured world knowledge for perception reasoning and planning we emphasize multimodal systems that integrate text vision and control signals to simulate real world dynamics and generalize across domains such as robotics gaming and scientific visualization ️ 2 problem definition and taxonomy we define a physical world model as a digital cousin of the real world a simulator that captures objective world knowledge without replicating specific instances it maps multimodal inputs text images audio actions trajectories to temporally coherent video outputs effectively serving as a stochastic generative process the model mirrors an mdp pomdp learning objective world dynamics during training while performing subjective inference during generation figure 4 overview of the world model defined in this paper the world model must take inputs such as text images videos audios or their combinations it may also incorporate external conditions for interaction including spatial conditions and navigation conditions a video generation model is leveraged to process the intermediate state representations to produce video outputs while other task specific outputs may also be generated depending on the downstream application 2 1 taxonomy of four generations from video generation to world model generation 1 faithfulness generates short realistic clips with limited control or planning generation 2 interactiveness adds controllability spatiotemporal reasoning and semantic consistency generation 3 planning achieves real time self evolving and physics driven long horizon simulation generation 4 stochasticity models low probability events and multi scale dynamics capturing both deterministic and random physical processes 2 2 definition of navigation mode we define the navigation mode as a temporally evolving content independent and spatially reasoning control interface that guides video generation it differs from spatial conditions like depth or sketch maps because it allows genuine interaction and dynamic reasoning this concept provides a theoretical tool to assess whether a video model truly exhibits planning and interactivity 3 roadmap the roadmap of video generation toward world modeling can be viewed as a continuous progression along three capability axes faithfulness interactiveness and planning each generation focuses on one dominant ability while advancing the others in parallel together these dimensions represent the gradual evolution from visual realism to physically grounded simulation and intelligent reasoning 1 generation 1 faithfulness the first generation emphasizes visual and temporal realism models in this stage can generate short visually appealing clips that maintain basic text video consistency and object motion coherence however their world remains superficial physical laws 3d geometry and task understanding are weak or absent these systems rely heavily on static spatial conditions like sketches or depth maps and show limited controllability which are good at showing but not at doing 2 generation 2 interactiveness the second generation marks the rise of controllability and semantic understanding models begin to support navigation modes such as trajectories actions or text instructions allowing humans or agents to influence the simulated world they generate longer coherent videos with consistent physics and improved subject centered control temporal stability and semantic alignment make them capable of simple goal driven planning e g executing a short sequence like pick up the cup and pour water this stage forms the bridge between pure generation and real time simulation 3 generation 3 planning planning emerges as a defining capability generation 3 world models simulate complex real time and self evolving dynamics grounded in intrinsic physical knowledge they can produce infinitely extending sequences adapt to external stimuli and handle multi entity interactions with causal coherence for example given the command make a cup of coffee the model can autonomously generate long sequences involving heating water grinding beans and pouring all while maintaining spatial consistency and temporal logic interaction becomes local and precise with frame level control and physical reasoning 4 generation 4 stochasticity the fourth generation integrates probabilistic reasoning and multi scale modeling enabling simulation of both common and rare events these models can represent uncertainties and low probability phenomena like accidents weather extremes or biological variations within realistic distributions they unify microscopic millisecond mesoscopic everyday and macroscopic long term time scales achieving global multimodal interaction across language vision and control in this stage the world model behaves as a general purpose simulator capable of imagining countless plausible futures with physical and causal fidelity figure 5 overview of the capabilities of world model across 4 generations this figure presents the three main capabilities of world models along with their corresponding secondary capabilities under each category 4 methods a hierarchical taxonomy world modeling evolves along three parallel dimensions faithfulness interactiveness and planning which develop together rather than sequentially each generation highlights one main capability but continues improving the others forming a continuous hierarchy the methods discussed below are grouped by generation and application domains general scenes robotics autonomous driving and gaming which collectively illustrate how controllable video generation matures into true world simulation figure 6 chronological overview of methods from video generation to world models the figure presents a chrono logical overview along the horizontal axis categorizing existing methods by four application domains along the vertical axis general scenes robotics autonomous driving and gaming additionally different colors are used to indicate different genera tions of world models 4 1 generation 1 faithfulness accurate simulation of the real world in generation 1 the goal is to produce short yet realistic video clips that maintain visual coherence and text video alignment this stage establishes the foundations of world modeling through video foundation models spatial world models and navigation world models 4 1 1 foundation models with visual centric world knowledge early diffusion transformer and vae based models such as ltx video open sora and videocrafter generate visually consistent frames with basic temporal continuity they learn general motion and appearance priors from large scale data supporting tasks like motion imitation and scene rendering however they still lack true 3d reasoning physical causality and goal oriented planning 4 1 2 spatial world models spatially conditioned models introduce additional inputs such as depth maps sketches motion fields or 3d geometry to guide the generation process works such as diffusion4d or physgen embed geometric or physical priors to enforce spatial consistency and realistic object interactions while these signals improve controllability they remain scene dependent and static offering limited flexibility for dynamic interaction 4 1 3 navigation world models navigation world models incorporate temporal control signals like trajectories camera motion or text commands into the input space they enable motion aware video generation and elementary interactivity methods like trailblazer peekaboo and drivedreamer demonstrate how adding trajectories or actions allows localized control of subject or camera motion yet they remain limited to short sequences and single agent scenes 4 2 generation 2 interactiveness controllability and interactive dynamics generation 2 represents a decisive shift toward dynamic flexible control models now integrate multiple modalities such as geometry physics actions and language to support long horizon interaction and semantically consistent behavior faithfulness and planning also improve through richer conditioning and real time responsiveness figure 7 overview of condition injection strategies this figure illustrates five representative strategies for condition injection subfigures a through e correspond to the controlnet based method multi modal transformer cross attention concatenation and addition respectively 4 2 1 foundation models with dynamic aware world knowledge these models such as sora hunyuanvideo and magi 1 combine diffusion backbones with autoregressive or causal modules to improve temporal coherence they learn not only to render visuals but also to predict and adapt to changes in the environment techniques like controlnet cross attention and multimodal transformers inject control signals efficiently bridging foundation video models and interactive world simulators as shown in figure 7 4 2 2 video generation as world model in general scene general scene approaches demonstrate how controllable video generation can generalize across environments geometry conditioned models e g sketchvideo use sparse sketches or motion cues for precise visual control 3d prior models e g das gs dit enhance realism and structure with point clouds and multi view information physical prior models e g phyt2v wisa embed physical laws to improve realism and causality camera motion models e g cameractrl ii gen3c simulate agent like first person views instruction based models e g pandora aether follow text commands such as turn left or zoom in marking the transition from passive generation to active world simulation 4 2 3 video generation as world model in robotics in robotics video based world models help agents visualize and plan their actions within physical environments systems like cosmos transfer1 unisim pad and videoagent integrate multimodal observations vision depth text to predict outcomes of robot motion these models can generate and adapt visual plans conditioned on control signals or goals providing a bridge between perception and decision making in embodied learning 4 2 4 video generation as world model in autonomous driving autonomous driving world models simulate road scenes for safe policy learning and planning methods such as cosmos drive dreams gaia 2 drivingworld and vista generate realistic driving sequences conditioned on layouts bounding boxes and navigation maps later systems incorporate trajectory based and instruction based control to enable reactive and goal oriented driving video synthesis supporting data generation for planning and reinforcement learning 4 2 5 video generation as world model in gaming gaming provides an ideal sandbox for interactive simulation works like hunyuan gamecraft genie 2 oasis and gamengen treat video generation as a controllable game rendering engine where users manipulate characters or scenes through controller signals or language instructions these prototypes show how generative models can create dynamic playable virtual worlds an early form of open ended digital simulation 4 3 generation 3 planning real time and complex prediction generation 3 marks a major leap from controllable video generation to true simulation and forecasting these models can plan and evolve the state of the world over long horizons while maintaining physical realism they integrate intrinsic physics causal reasoning and self evolving world dynamics to simulate how a scene changes in response to both internal and external factors models such as v jepa 2 miragelsd nwm and genie 3 represent this stage they predict extended or even infinite sequences by reasoning over interactions among multiple agents complex environments and temporal dependencies the resulting systems exhibit physically intrinsic faithfulness their behavior obeys newtonian and fluid dynamic principles and real time local interactiveness allowing users or agents to modify objects and observe instant consistent responses this generation signals the convergence of video simulation and decision making providing a framework for embodied planning and scientific modeling 4 4 generation 4 stochasticity multi scale and low probability simulation the fourth generation introduces stochastic world modeling where systems reason not only about what is likely but also about what could happen such models can generate both deterministic and random outcomes capturing rare black swan events like accidents weather anomalies or biological variations while preserving physical plausibility they operate across multiple spatiotemporal scales microscopic millisecond level detail e g molecular motion mesoscopic daily human scale dynamics and macroscopic planetary or cosmological processes generation 4 systems such as nwm and aether unify these scales under probabilistic planning linking local physics with global evolution they exhibit global and multimodal interactiveness coordinating multiple agents and sensory streams vision text audio control in essence these are general purpose simulation engines digital cousins of the real world that can imagine endless physically coherent futures 4 5 insight beyond generation sections 4 1 to 4 4 collectively outline a h...