Meta tags:

Headings (most frequently used words):

-

Text of the page (most frequently used words):
and (48), the (40), for (19), flow (19), training (18), diffusion (15), depth (14), model (13), estimation (12), results (11), our (10), optical (10), table (9), models (9), kitti (8), monocular (8), fine (8), zero (7), shot (7), pre (7), with (6), denoising (6), caption (6), text (6), state (5), art (5), nyu (5), this (5), that (5), data (5), step (5), achieves (4), benchmark (4), sintel (4), are (4), tuning (4), can (4), label (4), inference (4), image (4), refinement (4), test (3), shown (3), method (3), relative (3), error (3), dataset (3), coarse (3), show (3), synthetic (3), performance (3), missing (3), noise (3), distribution (3), one (3), simple (3), ability (3), figure (3), above (3), uncertainty (3), datasets (3), ablation (3), infilling (3), public (2), provide (2), baseline (2), using (2), proposed (2), mixture (2), substantially (2), improve (2), shows (2), benchmarks (2), all (2), outlier (2), rate (2), task (2), absolute (2), 074 (2), indoor (2), autoflow (2), effective (2), estimates (2), adding (2), during (2), given (2), groundtruth (2), gap (2), mitigate (2), values (2), interpolation (2), train (2), neural (2), conditional (2), rgb (2), noisy (2), unrolling (2), surprising (2), effectiveness (2), saurabh (2), saxena (2), charles (2), herrmann (2), junhwa (2), hur (2), abhishek (2), kar (2), mohammad (2), norouzi (2), deqing (2), sun (2), david (2), fleet (2), room (2), below (2), from (2), generation (2), completion (2), point (2), imagen (2), correct (2), their (2), capture (2), multimodal (2), cases (2), samples (2), multimodality (2), unrolled (2), than (2), best (2), published (2), also (2), supervised (2), ddvm (2), vision (2), new, raft, accuracy, over, original, outperforms, even, much, stronger, final, official, tuned, standard, competitive, 055, has, recently, emerged, interestingly, find, trained, alone, tend, very, hallucinate, shapes, addition, flyingthings, kubric, tartanair, remove, induced, bias, toward, polgonal, shaped, regions, significantly, quality, detail, trees, thin, structures, motion, boundaries, tables, improves, both, presence, holes, labels, usual, approach, involving, gaussian, does, not, work, due, first, infill, then, add, map, network, time, stamp, optionally, unroll, stop, gradient, bridge, between, nearest, neighbor, effectively, problem, inproceedings, saxena2023ddvm, title, author, booktitle, thirty, seventh, conference, information, processing, systems, year, 2023, url, https, openreview, net, forum, jdilzsu8wj, bibtex, movie, theatre, meeting, library, living, more, system, bedroom, kitchen, another, interesting, property, arising, iterative, nature, perform, leverage, build, scene, pipeline, combining, existing, conditioned, clouds, some, scenes, generated, respective, prompts, subsampled, 10x, fast, visualization, editor, pretrained, helps, wrong, adds, details, strength, complex, distributions, representing, example, transparent, translucent, reflective, multiple, showing, captures, provides, plausible, when, ambiguities, exist, predictions, modified, sparse, maps, lower, sota, probabilistic, have, transformed, impressive, fidelity, diversity, they, excel, estimating, surprisingly, without, specific, architectures, loss, functions, predominant, these, tasks, compared, conventional, regression, based, methods, enable, monte, carlo, capturing, ambiguity, self, combined, use, real, technical, innovations, handle, incomplete, form, extensive, experiments, focus, quantitative, against, ablations, impute, obtains, about, better, abstract, paper,

Text of the page (random words):
ddvm denoising diffusion vision model the surprising effectiveness of diffusion models for optical flow and monocular depth estimation saurabh saxena charles herrmann junhwa hur abhishek kar mohammad norouzi deqing sun david j fleet paper abstract denoising diffusion probabilistic models have transformed image generation with their impressive fidelity and diversity we show that they also excel in estimating optical flow and monocular depth surprisingly without task specific architectures and loss functions that are predominant for these tasks compared to the point estimates of conventional regression based methods diffusion models also enable monte carlo inference e g capturing uncertainty and ambiguity in flow and depth with self supervised pre training the combined use of synthetic and real data for supervised training and technical innovations infilling and step unrolled denoising diffusion training to handle noisy incomplete training data and a simple form of coarse to fine refinement one can train state of the art diffusion models for depth and optical flow estimation extensive experiments focus on quantitative performance against benchmarks ablations and the model s ability to capture uncertainty and multimodality and impute missing values our model ddvm denoising diffusion vision model obtains a state of the art relative depth error of 0 074 on the indoor nyu benchmark and an fl all outlier rate of 3 26 on the kitti optical flow benchmark about 25 better than the best published method sota results on optical flow and monocular depth estimation our model achieves state of the art results on the public test benchmark for optical flow estimation on kitti and for monocular depth estimation on nyu table 1 flow pre training results table 2 flow fine tuning results zero shot optical flow estimation results on sintel and kitti are shown in table 1 we provide a new raft baseline using our proposed pre training mixture and substantially improve the accuracy over the original our diffusion model outperforms even this much stronger baseline and achieves state of the art zero shot results on sintel final and kitti table 2 shows results on the official test benchmarks for sintel and kitti for a model fine tuned on a standard fine tuning mixture our model achieves an fl all outlier rate of 3 26 on the public kitti test benchmark 25 lower than the best published method table 3 monocular depth estimation results on the task of monocular depth estimation our method achieves a state of the art absolute relative error of 0 074 on the indoor nyu dataset and a competitive absolute relative error of 0 055 on the kitti dataset synthetic training data autoflow af has recently emerged as an effective dataset for optical flow pre training interestingly we find that diffusion models trained with autoflow alone tend to provide very coarse flow estimates and can hallucinate shapes the addition of flyingthings ft kubric ku and tartanair ta remove the af induced bias toward polgonal shaped regions and significantly improve flow quality on fine detail e g trees thin structures and motion boundaries table 4 pre training datasets ablation for zero shot flow estimation table 5 pre training datasets ablation for fine tuning performance on nyu depth v2 tables 4 and 5 show that adding synthetic data during pre training substantially improves results for both zero shot optical flow estimation and fine tuning performance on monocular depth estimation modified diffusion training for sparse label maps denoising diffusion training with infilling and step unrolling given the presence of holes missing labels in groundtruth flow and depth data the usual diffusion model training approach involving adding gaussian noise to the groundtruth label does not work due a training inference distribution gap to mitigate this we first infill missing values using interpolation then we add noise to the label map and train a neural network to model the conditional distribution of the noise given the rgb image s noisy label and time stamp one can optionally unroll the denoising step s during training with stop gradient to bridge the distribution gap between training and inference for y t table 6 ablation for infilling and step unrolled training as the results in table 6 show simple nearest neighbor interpolation and step unrolling effectively mitigate this problem multimodal predictions one strength of diffusion models is their ability to capture complex multimodal distributions this can be effective in representing uncertainty for example in cases of transparent translucent or reflective cases above figure shows multiple samples on the nyu kitti and sintel datasets showing that our model captures multimodality and provides plausible samples when ambiguities exist zero shot coarse to fine refinement for our pretrained model refinement helps correct wrong flow and adds details to correct flow as shown in the figure above zero shot depth completion and text to 3d another interesting property arising the iterative refinement nature of diffusion models is the ability to perform conditional inference zero shot we leverage this to build a simple text to 3d scene generation pipeline by combining our model with existing text to image imagen and text conditioned image completion imagen editor models as shown in the figure above below are 3d point clouds of some scenes generated from the respective text prompts subsampled 10x for fast visualization 3d caption a kitchen 3d caption a bedroom below are more text to rgb d results from our proposed system caption a living room caption a library caption a meeting room caption a movie theatre bibtex inproceedings saxena2023ddvm title the surprising effectiveness of diffusion models for optical flow and monocular depth estimation author saurabh saxena and charles herrmann and junhwa hur and abhishek kar and mohammad norouzi and deqing sun and david j fleet booktitle thirty seventh conference on neural information processing systems year 2023 url https openreview net forum id jdilzsu8wj