Meta tags:
description=Lazy Diffusion Transformer for Interactive Image Editing;
keywords=Transformers, Diffusion Models, Image Inpainting;

Headings (most frequently used words):

the, image, editing, generation, generates, mask, runtime, that, of, lazy, diffusion, transformer, for, interactive, incremental, using, lazydiffusion, model, content, according, to, text, prompt, in, an, area, specified, by, each, update, only, masked, pixels, with, depends, chiefly, on, size, rather, than, abstract, how, does, it, work, results, bibtex, progressive, and, comparison,

Text of the page (most frequently used words):
the (38), image (21), and (16), diffusion (12), that (9), context (9), for (8), inpainting (8), mask (8), #lazydiffusion (7), regenerating (7), transformer (6), editing (6), crop (6), generates (6), our (6), interactive (5), model (5), only (5), entire (5), with (5), two (5), using (5), than (5), lazy (4), when (4), methods (4), generate (4), models (4), canvas (4), masked (4), region (4), based (3), this (3), produce (3), results (3), quality (3), pixart (3), second (3), faster (3), typical (3), runtime (3), decoder (3), encoder (3), global (3), size (3), which (3), user (3), website (2), source (2), code (2), yotam (2), nitzan (2), zongze (2), richard (2), zhang (2), eli (2), shechtman (2), daniel (2), cohen (2), taesung (2), park (2), michaël (2), gharbi (2), arxiv (2), semantic (2), processing (2), objects (2), but (2), not (2), greater (2), contrast (2), fidelity (2), compare (2), stable (2), all (2), each (2), state (2), generation (2), existing (2), smaller (2), small (2), speedup (2), masks (2), resolution (2), computation (2), first (2), tokens (2), full (2), from (2), approach (2), text (2), pixels (2), while (2), project, page, you, want, reuse, their, please, credit, them, appropriately, nerfies, licensed, under, creative, commons, attribution, sharealike, international, license, misc, nitzan2024lazy, title, author, year, 2024, eprint, 2404, 12382, archiveprefix, primaryclass, bibtex, object, closely, related, others, requires, robust, understanding, may, seem, reasonable, isolation, fit, well, within, leverages, compressed, high, comparable, running, ten, times, slower, 512x512, 1024x1024, inapinting, require, relatively, little, reasonably, good, comparison, panel, illustrates, generative, progression, compared, preceding, its, left, progressive, approaches, are, architecture, consistently, especially, ratios, edits, reaching, 10x, similarly, dashed, technically, low, naively, upsamples, match, desired, harming, bottom, reduces, synthesis, strategies, compress, separate, shown, outside, loop, corresponding, transformers, top, maintain, throughout, process, preserve, performing, such, most, discarded, order, fill, hole, convolutional, suffer, same, drawbacks, how, does, work, introduce, novel, partial, updates, efficiently, targets, applications, starting, blank, specifies, sequence, localized, modifications, binary, prompts, generator, operates, phases, processes, current, compact, tailored, conditioned, synthesizes, fashion, contrasts, previous, works, either, regenerate, wasting, time, confine, tight, rectangular, around, ignoring, altogether, scales, typically, introduces, negligible, overhead, demonstrate, competitive, art, terms, providing, interactions, where, represents, abstract, incremental, content, according, prompt, area, specified, update, depends, chiefly, rather, paper, tel, aviv, university, adobe, research,

Text of the page (random words):
lazy diffusion transformer for interactive image editing lazy diffusion transformer for interactive image editing yotam nitzan 1 2 zongze wu 1 richard zhang 1 eli shechtman 1 daniel cohen or 2 taesung park 1 michaël gharbi 1 1 adobe research 2 tel aviv university paper arxiv incremental image generation using lazydiffusion the model generates content according to a text prompt in an area specified by a mask each update generates only the masked pixels with a runtime that depends chiefly on the size of the mask rather than that of the image abstract we introduce a novel diffusion transformer lazydiffusion that generates partial image updates efficiently our approach targets interactive image editing applications in which starting from a blank canvas or an image a user specifies a sequence of localized image modifications using binary masks and text prompts our generator operates in two phases first a context encoder processes the current canvas and user mask to produce a compact global context tailored to the region to generate second conditioned on this context a diffusion based transformer decoder synthesizes the masked pixels in a lazy fashion i e it only generates the masked region this contrasts with previous works that either regenerate the full canvas wasting time and computation or confine processing to a tight rectangular crop around the mask ignoring the global image context altogether our decoder s runtime scales with the mask size which is typically small while our encoder introduces negligible overhead we demonstrate that our approach is competitive with state of the art inpainting methods in terms of quality and fidelity while providing a 10 speedup for typical user interactions where the editing mask represents 10 of the image how does it work our diffusion transformer decoder bottom reduces synthesis computation using two strategies first we compress the image context using a separate encoder not shown outside the diffusion loop second we only generate tokens corresponding to the masked region to generate in contrast typical diffusion transformers top maintain tokens for the entire image throughout the diffusion process to preserve global context when performing inpainting such model generates a full size image most of which is discarded in order to in fill the hole region only existing convolutional diffusion models for inpainting suffer from the same drawbacks runtime we compare lazydiffusion to the two existing inpainting approaches regenerating a smaller crop or the entire image all methods are using a pixart based architecture lazydiffusion is consistently faster than a regenerating the entire image especially for small mask ratios typical to interactive edits reaching a speedup of 10x similarly lazydiffusion is faster than regenerating a crop when the mask is smaller than that for masks greater than that dashed regenerating the crop is technically faster but generates in low resolution and naively upsamples to match the desired resolution harming image quality results progressive editing and generation each panel illustrates a generative progression compared to the preceding state of the canvas to its left comparison we compare lazydiffusion with two models regenerating a 512x512 crop stable diffusion 2 and a pixart inpainting model and two models regenerating the entire 1024x1024 image stable diffusion xl and a second pixart inapinting model when inpainting objects that require relatively little semantic context all methods produce reasonably good results when inpainting an object closely related to others the inpainting model requires robust semantic understanding methods processing only a crop produce objects that may seem reasonable in isolation but do not fit well within the greater context of the image in contrast lazydiffusion leverages the compressed image context to generate high fidelity results comparable in quality to models regenerating the entire image and running up to ten times slower bibtex misc nitzan2024lazy title lazy diffusion transformer for interactive image editing author yotam nitzan and zongze wu and richard zhang and eli shechtman and daniel cohen or and taesung park and michaël gharbi year 2024 eprint 2404 12382 archiveprefix arxiv primaryclass cs cv this website is licensed under a creative commons attribution sharealike 4 0 international license website source code based on the nerfies project page if you want to reuse their source code please credit them appropriately