Meta tags:
description=CrossMAE: Rethinking Patch Dependence for Masked Autoencoders;

Headings (most frequently used words):

attention, crossmae, rethinking, patch, dependence, for, masked, autoencoders, tl, dr, learning, visual, representation, doesn, require, the, model, to, generate, self, consistent, images, overview, cross, partial, masking, and, inter, block, visualizations, results, citation,

Text of the page (most frequently used words):
the (28), and (20), crossmae (12), for (10), attention (10), #masked (9), tokens (8), block (8), decoder (8), image (6), reconstruct (6), cross (6), mask (5), reconstruction (5), patch (4), autoencoders (4), wang (4), with (4), trained (4), all (4), patches (4), rethinking (3), dependence (3), use (3), images (3), from (3), mae (3), not (3), visible (3), that (3), queries (3), self (3), blocks (3), this (2), project (2), letian (2), long (2), lian (2), renhao (2), baifeng (2), shi (2), xudong (2), adam (2), yala (2), trevor (2), darrell (2), alexei (2), efros (2), ken (2), goldberg (2), arxiv (2), work (2), citation (2), each (2), are (2), original (2), does (2), them (2), model (2), partial (2), can (2), which (2), its (2), full (2), representation (2), learning (2), imagenet (2), uses (2), output (2), encoder (2), sum (2), features (2), inter (2), feature (2), maps (2), overview (2), only (2), equal (2), credit, design, page, references, pages, lerf, deepmotionediting, nerf, article, fu2024rethinking, title, author, journal, preprint, 2401, 14391, year, 2024, you, find, helpful, please, consider, citing, our, results, set, left, right, ratio, since, outputs, have, overlaid, intriguingly, when, decode, one, forward, pass, shown, above, deviates, training, methodology, comparable, quality, models, suggests, might, essential, effective, example, reconstructions, validation, visualizations, vanilla, version, last, keys, first, takes, their, corresponding, positional, embeddings, subsequent, layers, previous, unlike, transformers, contain, decoupling, generation, different, leverage, low, level, via, weighs, intermediate, weighted, used, key, value, masking, concatenates, vit, passes, correspond, then, dropped, loss, applied, rest, pretraining, objective, instead, subset, introduce, decoding, show, greatly, enhances, efficiency, performance, tasks, like, classification, coco, instance, segmentation, significantly, reduced, computational, demands, visual, doesn, require, generate, consistent, code, paper, advising, contribution, ucsf, berkeley,

Text of the page (random words):
crossmae rethinking patch dependence for masked autoencoders crossmae rethinking patch dependence for masked autoencoders letian fu 1 long lian 1 renhao wang 1 baifeng shi 1 xudong wang 1 adam yala 1 2 trevor darrell 1 alexei a efros 1 ken goldberg 1 1 uc berkeley 2 ucsf equal contribution equal advising paper code citation tl dr learning visual representation doesn t require the model to generate self consistent images overview we introduce cross attention masked autoencoders crossmae which use only cross attention for decoding in mae we show that crossmae greatly enhances efficiency and performance in tasks like imagenet classification and coco instance segmentation with significantly reduced computational demands mae concatenates all mask tokens with the visible patch features from a vit encoder and passes them to a decoder with self attention blocks to reconstruct the original image patches that correspond to visible tokens are then dropped and an l2 loss is applied to the rest of the reconstruction as the pretraining objective crossmae instead uses cross attention blocks in the decoder to reconstruct only a subset of the masked tokens cross attention partial masking and inter block attention overview of crossmae a the vanilla version of crossmae uses the output of the last encoder block as the keys and queries for cross attention the first decoder block takes the sum of mask tokens and their corresponding positional embeddings as queries and subsequent layers use the output of the previous decoder block as queries to reconstruct the masked patches b unlike the decoder block in transformers the cross attention decoder block does not contain self attention decoupling the generation of different masked patches c crossmae s decoder blocks can leverage low level features for reconstruction via inter block attention it weighs the intermediate feature maps and the weighted sum of feature maps is used as the key and value for each decoder block visualizations example reconstructions of imagenet validation images for each set of 5 images from left to right are the original image masked image with a mask ratio of 75 mae crossmae trained to reconstruct 25 of image tokens or 1 3 of the mask tokens and crossmae trained to reconstruct all masked tokens since crossmae does not reconstruct them all model outputs have the visible patches overlaid intriguingly crossmae when trained for partial reconstruction can decode all mask tokens in one forward pass shown above which deviates from its training methodology its comparable reconstruction quality to full image trained models suggests that full image reconstruction might not be essential for effective representation learning results citation if you use this work or find it helpful please consider citing our work article fu2024rethinking title rethinking patch dependence for masked autoencoders author letian fu and long lian and renhao wang and baifeng shi and xudong wang and adam yala and trevor darrell and alexei a efros and ken goldberg journal arxiv preprint arxiv 2401 14391 year 2024 credit the design of this project page references the project pages of nerf deepmotionediting and lerf