Meta tags:
description=A video model conditioned on any motion.;
keywords=Video Generation, Motion;

Headings (most frequently used words):

control, motion, with, object, camera, track, video, step, model, prompts, transfer, and, effect, of, prompting, controlling, generation, trajectories, train, conditioned, prompt, the, emergent, physics, geometry, overview, interacting, images, from, depth, geometric, primitives, by, composition, drag, based, image, editing, magnification, failures, text, sparsity, comparisons, uncurated, samples, cherry, picking, related, work, references, bibtex, dolly, zooms, arcing, circles, mouse, controlled, how, to, read, these, visualizations, conditioning, enables, prediction,

Text of the page (most frequently used words):
the (119), motion (67), and (61), video (49), that (29), this (28), with (27), for (27), model (25), #control (22), can (21), camera (19), #generation (16), image (16), 2024 (15), more (15), see (15), are (14), also (14), our (14), tracks (14), example (13), prompts (13), arxiv (12), mouse (12), these (11), which (10), effect (10), object (10), not (9), trajectories (9), videos (9), models (9), below (9), from (9), show (9), prompt (9), editing (8), point (8), work (8), based (7), text (7), there (7), above (7), examples (7), motions (7), transfer (7), controllable (6), sparse (6), siggraph (6), diffusion (6), drag (6), trajectory (6), how (6), such (6), track (6), magnification (5), tracking (5), sand (5), conditioned (5), prior (5), have (5), samples (5), inputs (5), here (5), but (5), desired (5), use (5), monkey (5), get (5), first (5), conditioning (5), user (5), you (4), website (4), source (4), doersch (4), sun (4), wang (4), prompting (4), cvpr (4), world (4), any (4), using (4), 2023 (4), way (4), resulting (4), far (4), right (4), causal (4), move (4), even (4), because (4), uncurated (4), note (4), density (4), achieve (4), emergent (4), out (4), find (4), result (4), when (4), frame (4), similar (4), drags (4), interacting (4), generated (4), just (3), geng (3), zhang (3), carl (3), chen (3), controlling (3), subtle (3), eccv (3), interactive (3), perception (3), depth (3), flexible (3), method (3), synthesis (3), representation (3), fine (3), grained (3), very (3), capabilities (3), will (3), while (3), shown (3), been (3), cherry (3), input (3), scene (3), does (3), sometimes (3), tree (3), failures (3), behavior (3), might (3), cow (3), magnified (3), they (3), images (3), visualize (3), some (3), primitive (3), real (3), time (3), train (3), points (3), one (3), temporally (3), potential (3), gallery (3), back (2), page (2), please (2), code (2), want (2), your (2), daniel (2), herrmann (2), charles (2), hur (2), junhwa (2), cole (2), forrester (2), serena (2), pfaff (2), tobias (2), lopez (2), guevara (2), tatiana (2), aytar (2), yusuf (2), rubinstein (2), michael (2), oliver (2), owens (2), andrew (2), deqing (2), bibtex (2), manifold (2), via (2), 2022 (2), harley (2), particle (2), occlusions (2), range (2), estimation (2), photo (2), iclr (2), manipulation (2), shi (2), generative (2), monocular (2), consistent (2), animation (2), efficient (2), draganything (2), conductor (2), large (2), subset (2), propose (2), condition (2), project (2), led (2), related (2), good (2), left (2), cursor (2), gets (2), generating (2), all (2), picked (2), four (2), exactly (2), set (2), however (2), those (2), comparisons (2), moving (2), arcing (2), seen (2), too (2), often (2), underlying (2), much (2), simple (2), part (2), specific (2), has (2), pine (2), limitations (2), lion (2), should (2), happens (2), face (2), cases (2), enough (2), pin (2), background (2), running (2), then (2), enables (2), powerful (2), works (2), adding (2), static (2), interesting (2), only (2), where (2), take (2), still (2), create (2), simultaneous (2), what (2), geometric (2), placing (2), over (2), roughly (2), results (2), neither (2), nor (2), circles (2), dolly (2), zooms (2), capability (2), single (2), prediction (2), fact (2), lead (2), about (2), hair (2), construct (2), physics (2), indicate (2), step (2), university (2), means, free, borrow, ask, link, footer, remember, remove, analytics, included, header, licensed, under, creative, commons, attribution, sharealike, international, license, article, geng2024motionprompting, author, title, journal, preprint, 2412, 02700, year, hao, 2018, rotstein, pathways, wadhwa, 2013, phase, processing, 2012, eulerian, revealing, changes, liu, 2005, neurips, tap, vid, benchmark, revisited, through, teller, ijcv, 2008, long, alzayer, magic, fixup, streamlining, watching, dynamic, guidance, differentiable, estimators, mou, dragondiffusion, enabling, style, dragdiffusion, harnessing, pan, gan, johansson, psychophysics, 1973, visual, biological, its, analysis, piccinelli, unidepth, universal, metric, lei, animateanything, zhou, trackgo, tora, oriented, transformer, i2v, explicit, modeling, videocomposer, compositional, controllability, niu, mofa, field, adaptions, frozen, anything, entity, precision, yin, dragnuwa, integrating, motionctrl, unified, controller, references, body, cite, recent, xiao, focuses, attention, built, adam, who, introduced, aside, upper, corner, begins, before, anticipates, full, entire, randomly, sampled, cherrypick, implicit, bias, chose, found, performs, relatively, well, animals, ended, focusing, kinds, picking, between, visualized, red, dot, participants, human, study, were, knob, allow, users, tune, themselves, intended, low, underspecified, higher, precisely, determined, expense, giving, constraints, strict, happy, middle, ground, keeping, dynamics, sparsity, throughout, webpage, describe, factor, influence, possible, try, keep, gradually, complex, most, empirically, significant, unintended, consequences, illuminate, spontaneously, conjuring, pawn, thin, air, clearly, unphysical, apparently, know, suggest, used, probe, chess, surprising, undesired, effects, constructed, underspecifies, arc, attributes, itself, looking, down, into, water, other, constructing, carefully, inadverdently, horns, inherent, ambiguity, van, gogh, snow, another, application, task, involves, taking, new, easier, algorithm, smoothing, magnifying, feeding, breathing, line, identical, insight, benefit, inherit, edit, masks, exhibit, pausing, removes, percept, perceived, playing, gestalt, common, fate, occurs, classic, surprisingly, applying, extremely, domain, apply, order, need, quite, dense, 1500, visible, brick, wall, bird, eye, trees, chewing, banana, hard, puppeteer, dog, cat, spinning, earth, skull, compose, two, together, technically, approximation, extreme, forth, composed, head, turns, composition, bottom, row, funny, happen, don, spherical, reinterpret, manipulating, sphere, approximated, than, alone, again, primitives, controlled, doing, changing, focal, length, never, posed, data, simply, falls, training, arcs, beyond, design, estimator, cloud, projecting, provided, sequence, cameras, defining, thing, effectively, interact, briefly, asking, questions, theory, signal, enable, suggests, scaling, improving, may, eventually, systems, able, make, predictions, answer, counterfactuals, behave, pull, shake, branches, phenomena, smoke, particularly, exciting, shows, general, querying, addition, believe, future, faster, illustrate, place, grid, wherever, being, dragged, concurrent, due, flexibility, multiple, times, release, actually, same, llm, tease, different, depending, elicit, incredibly, allows, encode, thousands, objects, global, overview, except, section, trails, their, colored, help, distinguish, them, created, converting, case, hand, dragging, pass, takes, minutes, sample, each, read, visualizations, geometry, alphaxiv, slides, 200, paper, 2025, oral, brown, michigan, google, deepmind,

Text of the page (random words):
motion prompting controlling video generation with motion trajectories motion prompting controlling video generation with motion trajectories daniel geng 1 2 charles herrmann 1 junhwa hur 1 forrester cole 1 serena zhang 1 tobias pfaff 1 tatiana lopez guevara 1 carl doersch 1 yusuf aytar 1 michael rubinstein 1 chen sun 1 3 oliver wang 1 andrew owens 2 deqing sun 1 1 google deepmind 2 university of michigan 3 brown university project lead cvpr 2025 oral paper arxiv slides 200 mb alphaxiv bibtex interacting gallery camera control gallery motion transfer gallery step 1 train a track conditioned video model step 2 prompt the model with motion prompts object control see more emergent physics see more control with geometry see more camera control see more object camera control see more motion transfer see more how to read these visualizations we visualize tracks and first frame inputs on the left and the generated video on the right the tracks have trails to indicate their trajectory and are colored to help distinguish them some motion prompts are created by converting user mouse inputs in which case we visualize the mouse motions and drags by placing a cursor where the mouse is and a hand if the user is dragging please note that this does not indicate that the videos are generated real time in fact it takes about 12 minutes to sample each video also because these videos are generated in one pass the motion is not causal see here for an example of this all videos on this page are cherry picked from four samples except for those in the uncurated section in which we show uncurated samples generated by our model overview we first train a video generation model conditioned on any motion to do this we condition our model on point trajectories 19 20 21 an incredibly flexible representation of motion this allows us to encode the motion of just single points or thousands of points the motion of specific objects or of a global scene and even occlusions and temporally sparse motion to actually use this model we construct motion prompts in much the same way we might prompt an llm we can use motion prompts to tease out different capabilities from our video model depending on how we do this we can elicit a large range of behavior such as object control emergent physics camera control simultaneous object and camera control drag based image editing motion magnification and motion transfer interacting with images we illustrate one simple way to construct a motion prompt above we can take user mouse motions and drags and place a grid of tracks wherever the mouse is being dragged the result is similar to prior and concurrent work on sparse trajectory control 1 2 3 4 5 6 7 8 9 10 11 26 but due to the flexibility of our motion representation we can also drag multiple times release the mouse or pin the background still with static tracks we also find that some inputs result in emergent phenomena such as with the smoke pine tree sand hair or cow below this is particularly exciting because it shows the potential for video models to be general world models and the potential for motion to be a way of interacting with and querying these world models in addition while the results below are neither real time nor causal see above we believe that they show the potential of future video generation models as they get faster more efficient and more powerful track conditioning enables prediction one interesting thing to note because our method works for temporally sparse track conditioning we can effectively do prediction for example in the above samples we interact with an image briefly roughly asking questions such as what happens if i shake the branches of this tree or how will the hair behave if i pull on it in fact in theory any temporally sparse conditioning signal should enable this capability this suggests that scaling and improving these models may eventually lead to systems able to make predictions and answer counterfactuals about the world see more examples camera control from depth beyond single drags we can also design motion prompts to achieve camera control we do this by first running a monocular depth estimator 12 to get a point cloud and then projecting points on to a user provided sequence of cameras defining the desired camera trajectory by doing this we can move the camera in arcs or circles with user mouse control or even get dolly zooms by changing the camera focal length note that we never train our model on posed data this camera control capability simply falls out of training a model to be conditioned on tracks dolly zooms arcing circles mouse controlled see more examples object control with geometric primitives we can also reinterpret mouse motions as manipulating a geometric primitive such as a sphere by placing these tracks over an object that can be roughly approximated by the primitive we can get a motion prompt with more fine grained control over the object than with sparse mouse tracks alone again the results below are neither real time nor causal see above in the bottom row we show a funny example of what might happen if you don t use a spherical primitive object and camera control by composition we can also create motion prompts that result in simultaneous object and camera motion to do this we compose a camera control motion prompt with an object control motion prompt by adding the two tracks together technically this is an approximation but is good enough for camera trajectories that are not too extreme below we show examples of back and forth camera motion composed with head turns motion transfer some motions are hard to create in these cases we can transfer a desired motion from a source video to a first frame for example below we puppeteer a monkey and a skull with a moving face and transfer the spinning of the earth to a cat and a dog surprisingly we find that our model can transfer motion even when applying extremely out of domain motions to images for example we can take the motion of a monkey chewing on a banana and apply it to a bird s eye photo of trees and a brick wall in order to do this we find that we need quite dense tracks we use 1500 but only visualize a subset below so that the source video is still visible the resulting videos exhibit an interesting effect in which pausing the video on any frame removes the percept of the source video that is the monkey can only be perceived when the video is playing where a gestalt common fate effect occurs this is related to classic work on motion perception 13 and a similar effect can be seen here see more examples drag based image editing there is a line of work that enables drag based editing of images 14 15 16 17 18 25 we can also achieve a similar effect by conditioning our model on these drags the set up is identical to the interacting with an image examples above similar to the insight from 18 25 the benefit of using a video model is that we inherit a powerful video prior prior works also propose edit masks which we can achieve by adding static tracks to the conditioning as shown on the far right motion magnification another application our model is motion magnification 22 23 24 this task involves taking a video with subtle motions and generating a new video in which these motions have been magnified so that they are easier to see we do this by running a tracking algorithm on an input video smoothing and magnifying the resulting tracks and then feeding the first frame of the input video and magnified tracks to our model below we show examples in which subtle breathing motion has been magnified failures sometimes motion prompts will have surprising and undesired effects this often happens when the constructed motion prompt underspecifies the desired behavior for example with the snow monkey we want to arc the camera so that it is above the scene however the model attributes part of that motion to the monkey itself resulting in it looking face down into the water in other cases we get failures when not constructing the motion prompts carefully enough in the cow example we inadverdently pin the horns of the cow to the background and sometimes there is just inherent ambiguity in motion as with the van gogh example failures can also illuminate limitations of the underlying video model in the chess example we see the model spontaneously conjuring a pawn from thin air a clearly unphysical behavior and in the lion example we see that the model apparently does not know how a lion should move these examples suggest that motion prompts might be used to probe the capabilities and limitations of video models effect of text prompts throughout this webpage we use text prompts that describe the image but not the desired motion so as to factor out the influence of text on the motion as much as possible we also try to keep the text prompts simple here we show the effect of gradually more complex text prompts for the most part empirically we find that the text prompt does not have a significant effect on the motion sometimes prompting for a specific result has unintended consequences such as the pine tree example on the far right effect of track sparsity track density is a knob that we allow users to tune themselves here we show the effect of track density on video generation for a motion prompt with the intended effect of arcing the camera above the scene as can be seen for low density tracks the motion is underspecified for higher density tracks this motion is more precisely determined but at the expense of giving the model constraints that are too strict often there is a happy middle ground in which we can achieve the desired motion while keeping emergent dynamics from the underlying video prior comparisons here we show comparisons between our method image conductor and draganything with the input motion visualized as a moving red dot these are exactly the videos that participants in our human study were shown uncurated samples and cherry picking all videos shown so far have been cherry picked from four samples below we show the uncurated randomly sampled videos these are exactly the set of videos from which we cherrypick note however that there is also implicit bias in how we chose the inputs for example we found that our video model performs relatively well on animals and as such we ended up focusing more on those kinds of inputs aside the sand video on the far right is a good example of how our model is not causal the sand in the upper left corner begins to move before the mouse cursor even gets there it anticipates that the mouse will move the sand there because the model gets the full motion while generating the entire video related work this project is built on prior work led by adam harley who re introduced point tracking 19 20 and also work on tracking led by carl doersch 21 there is also a very large body of work on motion conditioned video generation a subset of which we cite below 1 2 3 4 5 6 7 8 9 10 11 26 very very recent work by xiao et al trajectory attention also propose a way to condition video generation models on trajectories and focuses on the resulting fine grained camera control and video editing capabilities references 1 wang et al motionctrl a unified and flexible motion controller for video generation siggraph 2024 2 yin et al dragnuwa fine grained control in video generation by integrating text image and trajectory arxiv 2023 3 chen et al motion conditioned diffusion model for controllable video synthesis arxiv 2023 4 li et al image conductor precision control for interactive video synthesis arxiv 2024 5 wu et al draganything motion control for anything using entity representation eccv 2024 6 niu et al mofa video controllable image animation via generative motion field adaptions in frozen image to video diffusion model eccv 2024 7 wang et al videocomposer compositional video synthesis with motion controllability arxiv 2023 8 shi et al motion i2v consistent and controllable image to video generation with explicit motion modeling siggraph 2024 9 zhang et al tora trajectory oriented diffusion transformer for video generation arxiv 2024 10 zhou et al trackgo a flexible and efficient method for controllable video generation arxiv 2024 11 lei et al animateanything consistent and controllable animation for video generation arxiv 2024 12 piccinelli et al unidepth universal monocular metric depth estimation cvpr 2024 13 johansson et al visual perception of biological motion and a model for its analysis perception psychophysics 1973 14 pan et al drag your gan interactive point based manipulation on the generative image manifold siggraph 2023 15 shi et al dragdiffusion harnessing diffusion models for interactive point based image editing cvpr 2024 16 mou et al dragondiffusion enabling drag style manipulation on diffusion models iclr 2024 17 geng et al motion guidance diffusion based image editing with differentiable motion estimators iclr 2024 18 alzayer et al magic fixup streamlining photo editing by watching dynamic videos arxiv 2024 19 sand and teller particle video long range motion estimation using point trajectories ijcv 2008 20 harley et al particle video revisited tracking through occlusions using point trajectories eccv 2022 21 doersch et al tap vid a benchmark for tracking any point in a video neurips 2022 22 liu et al motion magnification siggraph 2005 23 wu et al eulerian video magnification for revealing subtle changes in the world siggraph 2012 24 wadhwa et al phase based video motion processing siggraph 2013 25 rotstein et al pathways on the image manifold image editing via video generation arxiv 2024 26 hao et al controllable video generation with sparse trajectories cvpr 2018 bibtex article geng2024motionprompting author geng daniel and herrmann charles and hur junhwa and cole forrester and zhang serena and pfaff tobias and lopez guevara tatiana and doersch carl and aytar yusuf and rubinstein michael and sun chen and wang oliver and owens andrew and sun deqing title motion prompting controlling video generation with motion trajectories journal arxiv preprint arxiv 2412 02700 year 2024 this website is licensed under a creative commons attribution sharealike 4 0 international license this means you are free to borrow the source code of this website we just ask that you link back to this page in the footer please remember to remove the analytics code included in the header of the website which you do not want on your website