Meta tags:
description=Project webpage for fairy-video2video;
keywords=Diffusion Models, video-to-video, generative AI;

Headings (most frequently used words):

video, results, fairy, fast, parallelized, instruction, guided, to, synthesis, abstract, method, qualitative, user, study, bibtex, character, object, swap, stylization, long, generation,

Text of the page (most frequently used words):
the (34), and (20), video (15), attention (14), frame (14), frames (12), fairy (11), features (10), this (9), turn (9), into (9), with (8), anchor (8), style (7), cross (7), temporal (6), that (5), videos (5), diffusion (5), page (4), from (4), instruction (4), synthesis (4), figure (4), our (4), consistency (4), model (4), which (3), fast (3), parallelized (3), guided (3), models (3), paper (3), user (3), study (3), set (3), without (3), based (3), generated (3), results (3), sculpture (3), source (3), feature (3), semantic (3), image (3), correspondence (3), was (2), project (2), you (2), are (2), website (2), just (2), under (2), bichen (2), chuang (2), ching (2), yao (2), wang (2), xiaoyan (2), jia (2), yichen (2), krishnakumar (2), kapil (2), xiao (2), tong (2), liang (2), feng (2), licheng (2), vajda (2), peter (2), arxiv (2), scale (2), evaluation (2), 1000 (2), samples (2), generation (2), methods (2), for (2), qualitative (2), please (2), pencil (2), sketch (2), van (2), gogh (2), knight (2), low (2), poly (2), art (2), long (2), memory (2), particular (2), can (2), within (2), seconds (2), via (2), second (2), make (2), lion (2), left (2), right (2), given (2), extract (2), respect (2), cached (2), anc (2), propagated (2), each (2), tokens (2), enhancing (2), current (2), target (2), estimation (2), propagates (2), between (2), similar (2), regions (2), through (2), across (2), editing (2), approach (2), superior (2), strategy (2), built, using, adopted, free, borrow, ask, link, back, footer, licensed, creative, commons, attribution, sharealike, international, license, nerfies, academic, template, article, wu2023fairy, title, author, journal, preprint, 2312, 13834, year, 2023, bibtex, comparison, baselines, significantly, surpassed, baseline, demonstrating, its, effectivity, conduct, large, consists, best, knowledge, largest, literature, far, compare, extensively, existing, tokenflow, renderer, gen, comparsion, supplimentary, look, able, arbitrary, issue, due, proposed, a100, gpus, tokyo, minecraft, ukiyo, cubism, monet, stylization, vintage, car, baby, cub, bronze, statue, marble, roman, wood, yeti, metal, character, object, swap, section, showcase, process, all, downsampling, interpolation, resize, output, longer, side, 512, keep, aspect, ratio, same, see, more, first, row, contains, scroll, illustration, blocks, cache, input, perform, ensure, sample, extracted, define, global, successive, when, generating, new, replace, self, layer, take, exhibit, analogous, content, thereby, analysis, gives, rise, central, component, visualization, score, shows, query, point, performs, accurate, any, finetuning, bridge, showing, temporally, tracks, intermediate, inside, map, interpreted, similarity, metric, assessing, throughout, various, where, one, region, will, assign, higher, other, shown, following, fig, consequently, representations, refined, weighted, sum, effectively, minimizing, disparity, translates, improved, examines, tracking, propagation, paradigm, context, method, introduce, minimalist, yet, robust, adaptation, them, applications, centers, concept, mechanism, implicitly, ensuring, coherence, high, fidelity, not, only, addresses, limitations, previous, including, processing, speed, also, improves, unique, data, augmentation, renders, equivariant, affine, transformations, both, images, remarkably, efficient, generates, 120, 512x384, duration, fps, outpacing, prior, works, least, 44x, comprehensive, involving, confirms, delivers, quality, decisively, outperforming, established, abstract, supplementary, genai, meta, scrollable, table, arrow, buttons,

Text of the page (random words):
fairy fast parallelized instruction guided video to video synthesis scrollable table with arrow buttons fairy fast parallelized instruction guided video to video synthesis bichen wu ching yao chuang xiaoyan wang yichen jia kapil krishnakumar tong xiao feng liang licheng yu peter vajda genai meta paper supplementary videos abstract in this paper we introduce fairy a minimalist yet robust adaptation of image editing diffusion models enhancing them for video editing applications our approach centers on the concept of anchor based cross frame attention a mechanism that implicitly propagates diffusion features across frames ensuring superior temporal coherence and high fidelity synthesis fairy not only addresses limitations of previous models including memory and processing speed it also improves temporal consistency through a unique data augmentation strategy this strategy renders the model equivariant to affine transformations in both source and target images remarkably efficient fairy generates 120 frame 512x384 videos 4 second duration at 30 fps in just 14 seconds outpacing prior works by at least 44x a comprehensive user study involving 1000 generated samples confirms that our approach delivers superior quality decisively outperforming established methods method fairy re examines the tracking and propagation paradigm under the context of diffusion model features in particular we bridge cross frame attention with correspondence estimation showing that it temporally tracks and propagates intermediate features inside a diffusion model the cross frame attention map can be interpreted as a similarity metric assessing the correspondence between tokens throughout various frames where features from one semantic region will assign higher attention to similar semantic regions in other frames as shown in the following figure fig 3 consequently the current feature representations are refined and propagated through a weighted sum of similar regions across frames via attention effectively minimizing feature disparity between frames which translates to improved temporal consistency figure 3 visualization of attention score the left image shows the query point p within the current frame and the right image is the target frame cross frame attention performs accurate temporal correspondence estimation without any finetuning the analysis gives rise to our anchor based model the central component of fairy to ensure temporal consistency we sample k anchor frames from which we extract diffusion features and the extracted features define a set global features to be propagated to successive frames when generating each new frame we replace the self attention layer with cross frame attention with respect to the cached features of anchor frames with cross frame attention the tokens in each frame take the features in anchor frames that exhibit analogous semantic content thereby enhancing consistency figure 4 illustration of attention blocks a given a set of anchor frames we extract and cache the attention feature k_ anc and v_ anc b given an input frame we perform cross frame attention with respect to the cached features of anchor frames qualitative results in this section we showcase videos generated by fairy we process all frames from the source video without temporal downsampling or frame interpolation we resize the output video s longer side to 512 and keep the aspect ratio the same please scroll left or right to see more results the first row contains source videos 1 character object swap turn into a metal knight sculpture turn into a yeti turn into a wood sculpture turn into a marble roman sculpture turn into a bronze statue turn into a baby lion cub turn into lion turn into a vintage car 2 stylization in van gogh style in low poly art style in monet style in cubism style as pencil sketch in ukiyo e style make it minecraft make it tokyo 3 long video generation results fairy is able to scale to arbitrary long video without memory issue due to the proposed anchor based attention in particular a 27 second video can be generated within 71 89 seconds via 6 a100 gpus in low poly art style turn into a knight in van gogh style as pencil sketch user study in this paper we conduct a large scale user study on an evaluation set consists of 1000 video instruction samples to our best knowledge this is the largest evaluation in the video to video generation literature so far we compare extensively with existing methods tokenflow renderer and gen 1 please look at supplimentary videos for qualitative comparsion figure 7 a b comparison with baselines fairy significantly surpassed baseline models demonstrating its effectivity bibtex article wu2023fairy title fairy fast parallelized instruction guided video to video synthesis author wu bichen and chuang ching yao and wang xiaoyan and jia yichen and krishnakumar kapil and xiao tong and liang feng and yu licheng and vajda peter journal arxiv preprint arxiv 2312 13834 year 2023 this page was built using the academic project page template which was adopted from the nerfies project page you are free to borrow the of this website we just ask that you link back to this page in the footer this website is licensed under a creative commons attribution sharealike 4 0 international license