Meta tags:
description=Video Colorization with Pre-trained Text-to-Image Diffusion Models;
keywords=Colorization, Video Colorization, Stable Diffusion;

Headings (most frequently used words):

colorization, video, with, pre, trained, text, to, image, diffusion, models, abstract, results, method, playing, low, quality, webm, version, as, your, browser, does, not, support, 265,

Text of the page (most frequently used words):
the (30), video (15), #colorization (11), and (9), diffusion (8), color (8), sampling (7), latent (7), frames (7), model (7), text (7), your (7), browser (7), does (7), not (7), support (7), 265 (7), pre (6), trained (6), based (5), for (5), frame (5), propagation (5), with (5), image (5), input (4), grayscale (4), adjacent (4), reference (4), this (3), information (3), consistency (3), alternated (3), strategy (3), attention (3), that (3), code (2), during (2), each (2), process (2), step (2), previous (2), bidirectional (2), between (2), stable (2), adapter (2), colors (2), visual (2), quality (2), results (2), colordiffuser (2), proposed (2), temporal (2), across (2), models (2), website, licensed, under, source, project, page, nerfies, creative, commons, attribution, sharealike, international, license, inference, perform, parallel, particular, conditioned, from, essentially, coordinate, reverse, enable, ensure, over, time, extend, mechanism, obtain, conditional, leverages, power, render, space, z_c, according, semantics, mathcal, e_g, i_g, texttt, ref, method, playing, low, webm, version, challenging, task, involves, inferring, plausible, temporally, consistent, paper, present, adaptation, approach, repropose, accept, optional, description, enhance, coherence, maintain, vividness, propose, two, novel, techniques, enables, refine, its, decision, while, captures, spatiotemporal, dependencies, using, next, alternatively, generative, steps, encourages, leading, improved, conduct, extensive, experiments, benchmark, datasets, demonstrate, effectiveness, our, framework, evaluations, show, achieves, state, art, performance, surpassing, existing, methods, terms, fidelity, abstract, coming, soon, arxiv, anonymous, authors,

Text of the page (random words):
video colorization with pre trained text to image diffusion models video colorization with pre trained text to image diffusion models anonymous authors arxiv code coming soon abstract video colorization is a challenging task that involves inferring plausible and temporally consistent colors for grayscale frames in this paper we present colordiffuser an adaptation of a pre trained text to image latent diffusion model for video colorization with the proposed adapter based approach we repropose the pre trained text to image model to accept input grayscale video frames with the optional text description for video colorization to enhance the temporal coherence and maintain the vividness of colorization across frames we propose two novel techniques the color propagation attention and alternated sampling strategy color propagation attention enables the model to refine its colorization decision based on a reference latent frame while alternated sampling strategy captures spatiotemporal dependencies by using the next and previous adjacent latent frames alternatively as reference during the generative diffusion sampling steps this encourages bidirectional color information propagation between adjacent video frames leading to improved color consistency across frames we conduct extensive experiments on benchmark datasets and the results demonstrate the effectiveness of our proposed framework the evaluations show that colordiffuser achieves state of the art performance in video colorization surpassing existing methods in terms of color fidelity temporal consistency and visual quality colorization results playing low quality webm version as your browser does not support h 265 your browser does not support h 265 video your browser does not support h 265 video your browser does not support h 265 video your browser does not support h 265 video your browser does not support h 265 video your browser does not support h 265 video method we extend the pre trained stable diffusion model to a reference based frame colorization model with the adapter based mechanism we obtain a conditional text to image latent diffusion model that leverages the power of the pre trained stable diffusion model to render colors in the latent space z_c according to the visual semantics of the grayscale input g mathcal e_g i_g the text input and the reference color latent z_ texttt ref t during the inference for each frame in the input grayscale video we perform a parallel sampling process each sampling step for a particular frame is conditioned on the latent information from the previous sampling step of an adjacent frame essentially the color propagation attention and the alternated sampling strategy coordinate the reverse diffusion process and enable bidirectional propagation of color information between adjacent frames to ensure consistency in colorization over time this website is licensed under a creative commons attribution sharealike 4 0 international license the source code is based on nerfies project page