Meta tags:

Headings (most frequently used words):

64x64, video, kth, bair, stochastic, moving, mnist, prediction, generation, interpolation, mcvd, masked, conditional, diffusionfor, and, spatin, architecture, vikram, voleti, alexia, jolicoeur, martineau, christopher, pal, code, paper, blog, summary, abstract, cityscapes, 128x128,

Text of the page (most frequently used words):
frames (30), the (29), and (25), past (22), future (17), video (15), prediction (11), 64x64 (9), interpolation (9), #generation (8), current (8), are (8), masked (7), for (7), with (6), model (6), our (6), conditional (6), diffusion (6), predicted (5), this (5), image (4), column (4), left (4), number (4), when (4), conditioning (4), using (4), from (4), autoregressive (4), pred (4), quality (4), mcvd (4), only (4), interp (3), stochastic (3), moving (3), mnist (3), bair (3), kth (3), real (3), frame (3), previous (3), randomly (3), which (3), approach (3), high (3), into (3), row (3), autoregressively (3), cityscapes (3), that (3), videos (3), gpus (3), models (3), training (3), tasks (3), unconditional (3), all (3), both (3), generate (3), known (3), architecture (2), right (2), two (2), during (2), what (2), may (2), change (2), would (2), large (2), conditioned (2), show (2), then (2), arrow (2), brightness (2), fake (2), samples (2), but (2), data (2), beyond (2), can (2), predict (2), voleti (2), vikram (2), jolicoeur (2), martineau (2), alexia (2), pal (2), christopher (2), sota (2), capable (2), general (2), purpose (2), score (2), based (2), train (2), manner (2), novel (2), allows (2), diverse (2), recurrent (2), convolutional (2), blocks (2), paper (2), uses (2), spatin, smmnist, digits, overlap, will, have, guess, those, numbers, were, before, overlapping, they, fixed, used, match, baselines, start, generates, many, steps, given, validation, set, top, below, skip, ground, truth, shown, rows, notice, initial, advancing, passing, under, car, far, 3rd, 4th, initially, small, barely, visible, second, background, has, advanced, foreground, note, some, contain, changes, explain, definitively, overrepresented, more, parameters, needed, fix, problem, achieve, 128x128, first, use, next, last, new, free, running, 2022, year, neurips, advances, neural, information, processing, systems, booktitle, https, arxiv, org, abs, 2205, 09853, url, title, author, voleti2022mcvd, inproceedings, challenging, task, state, art, generative, tends, poor, generalization, difficult, furthermore, existing, frameworks, typically, not, simultaneously, handling, other, related, such, work, devise, framework, called, these, synthesis, probabilistic, denoising, where, independently, mask, straightforward, setup, single, executing, broad, range, specifically, neither, nor, experiments, types, built, simple, non, architectures, generating, arbitrary, lengths, block, wise, yields, results, across, standard, benchmarks, computation, times, measured, days, abstract, scales, well, channels, could, scaled, much, further, than, trains, produces, conditions, through, concatenation, space, time, adaptive, normalization, instead, complex, transformer, net, handle, four, cases, present, reconstruction, unknown, masks, injects, gaussian, noise, denoises, them, loss, function, forward, backward, summary, blog, code,

Text of the page (random words):
masked conditional video diffusion for prediction generation and interpolation mcvd masked conditional video diffusion for prediction generation and interpolation vikram voleti alexia jolicoeur martineau christopher pal code paper blog summary general purpose model for video generation forward backward prediction and interpolation uses a score based diffusion loss function to generate novel frames injects gaussian noise into the current frames and denoises them conditional on past and or future frames randomly masks past and or future frames during training which allows the model to handle the four cases unconditional generation both past and future are unknown future prediction only the past is known past reconstruction only the future is known interpolation both past and present are known uses a 2d convolutional u net instead of a complex 3d or recurrent or transformer architecture conditions on past and future frames through concatenation or space time adaptive normalization produces high quality and diverse video samples trains with only 1 4 gpus scales well with the number of channels and could be scaled much further than in the paper abstract video prediction is a challenging task the quality of video frames from current state of the art sota generative models tends to be poor and generalization beyond the training data is difficult furthermore existing prediction frameworks are typically not capable of simultaneously handling other video related tasks such as unconditional generation or interpolation in this work we devise a general purpose framework called masked conditional video diffusion mcvd for all of these video synthesis tasks using a probabilistic conditional score based denoising diffusion model conditioned on past and or future frames we train the model in a manner where we randomly and independently mask all the past frames or all the future frames this novel but straightforward setup allows us to train a single model that is capable of executing a broad range of video tasks specifically future past prediction when only future past frames are masked unconditional generation when both past and future frames are masked and interpolation when neither past nor future frames are masked our experiments show that this approach can generate high quality frames for diverse types of videos our mcvd models are built from simple non recurrent 2d convolutional architectures conditioning on blocks of frames and generating blocks of frames we generate videos of arbitrary lengths autoregressively in a block wise manner our approach yields sota results across standard video prediction and interpolation benchmarks with computation times for training models measured in 1 12 days using 4 gpus inproceedings voleti2022mcvd author voleti vikram and jolicoeur martineau alexia and pal christopher title mcvd masked conditional video diffusion for prediction generation and interpolation url https arxiv org abs 2205 09853 booktitle neurips advances in neural information processing systems year 2022 video prediction first we use real past frames to predict current frames then we autoregressively predict the next current frames using the last predicted frames as the new past frames free running left column with frame number real image right column predicted image cityscapes 128x128 past 2 current 5 autoregressive pred 28 note that some cityscapes videos contain brightness changes which may explain the brightness change in our fake samples but it is definitively overrepresented in the fake data more parameters would needed to fix this problem beyond what we can achieve with our 4 gpus our approach generates high quality frames many steps into the future given the two conditioning frames from the cityscapes validation set top left we show 7 predicted future frames in row 2 below then skip to frames 20 28 autoregressively predicted in row 4 ground truth frames are shown in rows 1 and 3 notice the initial large arrow advancing and passing under the car at frame 20 the far left of the 3rd and 4th row the initially small and barely visible second arrow in the background of the conditioning frames has advanced into the foreground kth 64x64 past 10 current 5 autoregressive pred 20 bair 64x64 past 2 current 5 autoregressive pred 28 stochastic moving mnist 64x64 past 5 current 5 autoregressive pred 20 in smmnist when two digits overlap during 5 frames a model conditioning on 5 previous frames will have to guess what those numbers were before overlapping so they may change randomly this would be fixed by using a large number of conditioned previous frames we used 5 to match previous prediction baselines which start from 5 frames video generation kth 64x64 bair 64x64 stochastic moving mnist 64x64 video interpolation left column with frame number real image right column predicted image kth 64x64 past 10 interp 10 future 5 bair 64x64 past 1 interp 5 future 2 stochastic moving mnist 64x64 past 5 interp 5 future 5 spatin architecture