Meta tags:

Headings (most frequently used words):

visual, intelligence, and, learning, members, lab, nbsp, laboratory, vilab, group, current, teaching, highlighted, recent, projects, prospective, contacts,

Text of the page (most frequently used words):
the (213), and (170), for (67), that (67), tasks (55), are (48), models (48), learning (46), this (46), with (46), training (44), can (41), #visual (35), model (34), epfl (33), task (33), vision (32), abstract (30), zamir (29), summer (27), website (27), data (27), code (26), image (26), intern (26), from (25), using (25), network (25), these (20), datasets (20), they (20), based (19), generation (19), well (18), different (18), more (18), modalities (18), test (18), multimodal (17), project (16), trained (16), time (16), which (16), into (15), show (15), pdf (15), neural (15), than (15), method (15), distribution (15), images (15), tokens (15), framework (14), such (14), one (14), multiple (14), multi (14), kar (14), methods (14), text (14), 2024 (14), performance (13), via (13), while (13), fine (13), results (13), set (13), 2023 (13), 2025 (13), our (12), when (12), pre (12), bachmann (12), video (12), computer (11), out (11), cross (11), information (11), them (11), semantic (11), language (11), atanov (11), how (11), level (10), representations (10), simple (10), diverse (10), through (10), 2021 (10), 2022 (10), planning (10), search (10), loop (9), input (9), recent (9), study (9), better (9), robust (9), adaptation (9), tuning (9), paper (9), propose (9), demonstrate (9), single (9), networks (9), many (9), inference (9), large (9), predictions (9), consistency (9), yeo (9), demo (9), perform (9), their (9), token (9), features (8), shifts (8), high (8), where (8), new (8), other (8), existing (8), several (8), leads (8), generated (8), its (8), standard (8), introduce (8), evaluate (8), downstream (8), representation (8), capabilities (8), prompt (8), environment (8), you (7), first (7), used (7), generalization (7), approaches (7), side (7), common (7), not (7), solve (7), same (7), also (7), proposed (7), unsupervised (7), supervised (7), various (7), best (7), guided (7), use (7), corruptions (7), efficient (7), modal (7), any (7), between (7), space (7), feedback (7), design (7), without (7), sensors (7), generative (7), diversification (7), like (7), tokenization (7), grid (7), 2026 (7), dynamic (7), however (6), system (6), control (6), deep (6), make (6), end (6), but (6), under (6), compared (6), real (6), malik (6), adapt (6), performing (6), fixed (6), feature (6), process (6), baselines (6), taskonomy (6), question (6), solving (6), only (6), should (6), object (6), depth (6), called (6), human (6), present (6), domains (6), adversarial (6), mechanisms (6), masked (6), rgb (6), across (6), modeling (6), given (6), find (6), target (6), encoding (6), user (6), gpt (6), address (5), students (5), phd (5), allows (5), often (5), components (5), learn (5), complexity (5), mid (5), perceptual (5), state (5), scale (5), navigation (5), sax (5), available (5), promising (5), eccv (5), wide (5), estimation (5), inputs (5), evaluations (5), benchmark (5), specific (5), research (5), achieve (5), imagenet (5), range (5), abstraction (5), geometric (5), remarkable (5), number (5), structure (5), two (5), key (5), modality (5), flexible (5), experiments (5), mizrahi (5), classical (5), provide (5), examples (5), foundation (5), gao (5), dehghan (5), ood (5), approach (5), vlms (5), individual (5), singh (5), enables (5), prompts (5), length (5), ordered (5), videos (5), coarse (5), intelligence (4), lab (4), please (4), highly (4), members (4), perception (4), sample (4), work (4), domain (4), consequently (4), 2020 (4), due (4), most (4), less (4), including (4), transfer (4), icml (4), live (4), pipeline (4), world (4), sampling (4), art (4), non (4), prediction (4), robustness (4), clip (4), controlled (4), transformations (4), leading (4), multimae (4), all (4), significantly (4), classification (4), segmentation (4), long (4), algorithms (4), algorithm (4), optimal (4), neurips (4), what (4), generalize (4), generate (4), fail (4), pixels (4), crucial (4), tuned (4), broad (4), current (4), limited (4), exhibit (4), versatile (4), achieves (4), discrete (4), further (4), abilities (4), second (4), guide (4), iclr (4), encoders (4), preference (4), structured (4), effective (4), garjani (4), develop (4), sequences (4), performs (4), thread (4), setup (4), videoflextok (4), regions (4), brain (4), stimuli (4), school (3), communication (3), sciences (3), there (3), apply (3), motivated (3), need (3), competitive (3), directly (3), your (3), conventional (3), usually (3), manner (3), contrast (3), advances (3), final (3), result (3), objectives (3), alternative (3), incorporating (3), successfully (3), experimental (3), findings (3), lifelong (3), has (3), commonly (3), employed (3), among (3), issues (3), does (3), scenarios (3), consistently (3), guibas (3), spotlight (3), applications (3), learned (3), together (3), computed (3), accuracy (3), produce (3), detection (3), etc (3), made (3), therefore (3), computational (3), over (3), observe (3), thus (3), performed (3), replica (3), cvpr (3), parameters (3), steer (3), enabling (3), train (3), dataset (3), surface (3), normal (3), iccv (3), making (3), cues (3), shift (3), hence (3), each (3), oral (3), sketching (3), understanding (3), sketch (3), ability (3), loss (3), augmentation (3), unlike (3), effectively (3), direction (3), besides (3), predictive (3), makes (3), observations (3), horizon (3), create (3), states (3), discovery (3), example (3), setting (3), embodied (3), practical (3), closed (3), flow (3), decoder (3), output (3), maps (3), small (3), box (3), choice (3), finally (3), cannot (3), ones (3), benoit (3), jiang (3), biases (3), brave (3), compressed (3), users (3), involves (3), aligned (3), manual (3), engineering (3), salehi (3), role (3), effectiveness (3), pixel (3), comparable (3), tens (3), open (3), have (3), techniques (3), joint (3), mechanism (3), autoregressive (3), raw (3), flextok (3), projects (3), allardice (3), beyond (3), analysis (3), self (3), multimodality (3), enable (3), tokenizers (3), structures (3), selectivity (3), spring (3), now (3), vilab (3), map (2), stephanie (2), baillargues (2), amir (2), people (2), program (2), free (2), fill (2), form (2), student (2), fit (2), interns (2), send (2), email (2), identify (2), prospective (2), robotics (2), factors (2), hand (2), then (2), component (2), explicit (2), during (2), comes (2), cost (2), asynchronously (2), traditional (2), generic (2), invariances (2), improve (2), lead (2), randomization (2), problems (2), policies (2), scratch (2), manipulation (2), chen (2), savarese (2), may (2), rather (2), start (2), enough (2), previously (2), priors (2), extractor (2), others (2), straightforward (2), additive (2), works (2), particular (2), little (2), reinforcement (2), answering (2), require (2), simultaneously (2), needs (2), evaluated (2), poses (2), few (2), slides (2), entails (2), independent (2), broadly (2), applicable (2), fully (2), arbitrary (2), quantity (2), energy (2), metric (2), award (2), scans (2), changing (2), interesting (2), architectures (2), starter (2), benchmarks (2), despite (2), having (2), least (2), according (2), mostly (2), omnidata (2), eftekhar (2), scalable (2), able (2), ensembles (2), nature (2), essential (2), scene (2), requires (2), prior (2), concepts (2), even (2), machines (2), levels (2), rely (2), contrastive (2), pretraining (2), sketches (2), curves (2), varying (2), subject (2), vinker (2), semantically (2), geometry (2), fly (2), expose (2), suggest (2), benchmarking (2), strategy (2), autoencoders (2), optionally (2), accept (2), additional (2), objective (2), includes (2), outputs (2), coding (2), improved (2), pseudo (2), hypersim (2), agents (2), dimensional (2), handling (2), general (2), purpose (2), palmer (2), segments (2), memory (2), mohammadi (2), developing (2), generalizes (2), intriguing (2), would (2), instead (2), score (2), rise (2), failure (2), modes (2), labels (2), filatov (2), finding (2), agent (2), realistic (2), odometry (2), especially (2), sensor (2), power (2), expensive (2), access (2), memmel (2), adapting (2), attempt (2), signal (2), function (2), named (2), rapid (2), notably (2), flexibility (2), coco (2), sodagar (2), machine (2), specialized (2), possibility (2), similarly (2), step (2), unified (2), transformer (2), encoder (2), scalability (2), unseen (2), analyses (2), potential (2), exploration (2), spurious (2), recently (2), hypotheses (2), class (2), sensitive (2), unlabeled (2), practice (2), major (2), typically (2), progress (2), broadening (2), top (2), frozen (2), smaller (2), highlight (2), desirable (2), inefficient (2), capturing (2), preferences (2), attributes (2), toward (2), towards (2), generations (2), facto (2), resolution (2), camera (2), position (2), found (2), allow (2), complex (2), raise (2), low (2), just (2), photoreceptors (2), useful (2), optimization (2), showing (2), designs (2), computationally (2), expand (2), dinov2 (2), specialist (2), novel (2), interact (2), whether (2), specifically (2), grained (2), griffiths (2), enabled (2), zero (2), shot (2), context (2), icl (2), labeled (2), eliminating (2), amirloo (2), curated (2), alekseev (2), ray (2), akhoondi (2), quality (2), inherent (2), tokenizer (2), variable (2), hierarchically (2), reconstructions (2), sequence (2), understand (2), gemini (2), challenges (2), api (2), chaining (2), affect (2), reasoning (2), ramachandran (2), predict (2), supervision (2), leveraging (2), devices (2), rich (2), specialization (2), internet (2), khattak (2), tarhan (2), toskov (2), organized (2), local (2), details (2), count (2), predicts (2), verifier (2), verifiers (2), scaling (2), rezaei (2), jovanović (2), silico (2), synthesis (2), evolutionary (2), synthesized (2), lateral (2), probing (2), highlighted (2), fall (2), teaching (2), ali (2), former (2), exchange (2), group (2), template, credit, indy, inj, 230, 231, 232, 234, 211, administrative, contacts, outside, recommend, applying, internship, eligible, otherwise, feel, will, reach, here, postdocs, always, looking, talented, don, admissions, centralized, admission, professors, labs, interested, believe, case, unique, after, submit, application, science, edic, applicants, separate, extract, engineered, possible, brittle, fails, unexpectedly, minuscule, effects, easy, decode, provided, aid, higher, harder, larger, trains, failed, reported, robots, simulations, pinto, lewis, armeni, corl, desired, prefer, randomly, initialized, lacking, being, wishing, encode, preset, weights, adapts, lightweight, fused, unchanged, solutions, resolves, some, basic, prone, overfitting, yields, suffer, catastrophic, forgetting, icifar, imitation, habitat, nlp, squad, zhang, save, computation, unfortunately, inferior, overall, compete, employing, cooperation, competition, settings, assigning, cooperating, competing, offers, trade, off, standley, expected, consistent, augmenting, formulation, path, invariance, graph, accurate, informative, measuring, intrinsic, correlates, error, confidence, roc, auc, cocodoom, apolloscape, versus, cycle, analytical, cheerla, suri, cao, nominee, introduces, parametrically, render, comprehensive, emphasize, addition, lines, tooling, suffice, reached, seen, outperforms, midas, wild, oasis, dockerized, cli, python, pytorch, dataloaders, download, scripts, utilities, https, middle, ensembling, strong, premise, idea, respond, differently, merge, merging, principled, uncertainty, associated, cifar, considerably, counterpart, heart, minimal, line, drawings, identifying, properties, knowledge, depictions, challenging, artists, simplifications, utilize, distill, alike, define, bézier, differentiable, rasterizer, optimize, respect, degree, strokes, maintaining, recognizability, underlying, drawn, pajouheshgar, bermano, cohen, shamir, transactions, graphics, proceedings, siggraph, clipasso, aware, primary, distinction, incorporated, likely, occur, natural, occlusions, extendable, applied, vulnerability, opens, differs, autoencoding, aspects, accordingly, predicting, masking, patches, tractable, ensure, indeed, exact, flexibly, configurations, yielding, avoid, needing, entirely, labeling, widely, ade20k, nyuv2, intriguingly, impressive, capability, autonomy, priori, unknown, act, sensory, past, experience, iii, capable, prm, rrt, proficient, turn, necessary, statistical, contingencies, combines, combine, latent, distance, embeddings, captures, easily, policy, traverse, retrieve, observed, trajectory, replay, buffer, restitch, approximately, paths, connect, pair, goal, creates, tight, experiential, beker, action, reorganization, decide, want, fixing, searching, fix, look, indicate, anything, automatically, finds, optimizing, agreement, give, understandings, tool, shed, light, discovered, splits, selecting, datapoints, split, testing, sohmshetty, localizing, noisy, substitute, unreliable, gps, compass, indoor, environments, slam, solid, requirements, noise, changes, suite, limit, themselves, millions, samples, suites, change, intentionally, looped, resources, consumption, catastrophically, furthermore, simulator, suitable, get, ignored, simulation, hinder, reusability, invariant, anticipate, counter, implemented, realizes, amortized, optimizer, rna, orders, magnitude, faster, signals, efficiency, scannet, optical, discussion, formulations, comparing, similar, hinting, take, scheme, consists, unifying, mapping, randomized, subset, excel, conditioned, variety, expressive, editing, stage, massively, contain, explain, equally, correct, lose, developed, lee, pagliardini, problem, aims, contributing, underperform, away, sweet, spot, alone, insufficient, architecture, waterbirds, office, home, absolute, drop, depends, vice, versa, dependent, above, pitfalls, alleviated, increasing, clearer, critical, influencing, practitioners, researchers, rigotti, unraveling, composed, interprets, encoded, shortcomings, blindness, certain, hallucination, comprehensively, inductive, vlm, configuration, surprisingly, consolidates, fed, captioning, vqa, reduces, aforementioned, requiring, trainable, contextualized, tonioni, poklukar, kulshrestha, tombari, gives, personalized, creating, unpersonalized, appeal, audience, relies, iterative, undesirable, personalize, inviting, comment, selection, explaining, why, dislike, comments, infer, liked, disliked, producing, series, studies, shafiei, viper, personalization, choose, placement, orientation, intuition, extremely, designed, throughout, organisms, behaviors, following, questions, play, explore, resolutions, representing, photoreceptor, continuous, reasonably, plays, survey, intuitive, devised, manually, humans, cases, spielberg, cameras, multitask, unifiedio, upon, corpora, imagebind, sam, 4dhumans, ways, metadata, color, palettes, vectors, instance, represented, doing, controllable, distillation, three, billion, parameter, resulting, sourced, allowing, enhance, substantial, effort, construct, since, direct, approximation, gadetsky, mighan, brbic, learners, previous, employ, defined, either, expertise, automated, uses, maximize, vulnerabilities, improving, interest, call, combining, types, correlations, illustrate, advantages, tmlr, diffusion, providing, titok, shown, redundancies, 256x256, resampled, anywhere, 256, compressing, rectified, nested, dropout, produces, plausible, regardless, chosen, style, fid, fini, nouby, resampling, clear, detail, popular, mini, pro, flash, claude, sonnet, qwen2, llama, established, variants, main, natively, express, proprietary, accessible, weight, translating, equivalent, promptable, compatible, standardized, native, latest, shows, hallucinated, objects, misalignment, variations, improvements, getting, respectable, generalists, presumably, primarily, close, evaluating, another, fundamental, deploying, household, robot, involve, equipped, sensing, presents, opportunity, sensed, developmental, psychology, biological, leverage, build, surroundings, restrict, device, performant, tst, collection, insights, collecting, generalist, scenario, external, reduced, ablations, points, substituting, tradeoff, specialise, held, spaces, compression, dictate, preserved, represent, spatiotemporal, corresponding, original, consumes, irrespective, represents, emergently, capture, semantics, motion, later, add, longer, budget, achieving, gfvd, viclip, prohibitive, frame, 672, fewer, hjelm, converting, manageable, units, describe, word, pieces, order, worthwhile, candidate, explored, testbed, hypothesize, amenable, rooted, fact, intermediate, carry, meaning, reliably, steering, behavior, counterparts, moreover, thanks, pure, systematically, beam, lookahead, impact, guidance, processes, functionally, synthesize, probe, been, largely, static, leaving, processing, underexplored, generates, optimized, cortex, voxel, responses, maximizing, predicted, activity, roi, efficiently, discovers, hyper, activating, surpass, handcrafted, localizer, recover, known, selectivities, ventral, dorsal, pathways, reveal, systematic, differences, sensitivity, temporal, dynamics, searchlight, provides, insight, progression, increasingly, social, along, stream, supported, naturalistic, taken, vivo, tang, zhou, isik, schrimpf, arxiv, nevo, eng, 615, topics, autonomous, com, 304, intelligent, systems, 500, product, management, 503, minds, stefan, stepanovic, namrata, deka, abhijeet, jagdev, ainaz, yael, jessica, david, louis, amaudruz, shijian, qianyi, ozgur, kara, marius, andrey, zahra, paulina, tomaszewska, mohammad, postdoc, callan, birkmann, abdülkadir, gökce, artem, zholus, harold, liangze, pooya, esmaeil, hosna, oyarhoseini, ruchira, isabella, aleksandr, narek, alvandian, rahul, assistant, jiawei, nataša, parham, jinya, sakurai, kseniia, petrushina, joanna, cheng, michael, murphy, junru, lin, google, deepmind, teresa, advised, jitendra, anthropic, alexander, sasha, apple, oğuzhan, fatih, zaynab, awofeso, jess, sun, oriol, barbany, sogand, masa, duric, walid, ait, said, albias, havolli, nil, biescas, rue, efe, undergraduate, master, won, jun, kim, muhammad, uzair, mingqiao, zhitong, kunal, pratap, jason, rishubh, andrei, roman, manager, prof, github, interests, active, swiss, federal, institute, technology, laboratory, nbsp, contact,

Text of the page (random words):
ion by providing compressed discrete representations that are more efficient to process than raw pixels while traditional approaches use 2d grid tokenization recent methods like titok have shown that 1d tokenization can achieve high generation quality by eliminating grid redundancies however these methods typically use a fixed number of tokens and thus cannot adapt to an image s inherent complexity we introduce flextok a tokenizer that projects 2d images into variable length ordered 1d token sequences for example a 256x256 image can be resampled into anywhere from 1 to 256 discrete tokens hierarchically and semantically compressing its information by training a rectified flow model as the decoder and using nested dropout flextok produces plausible reconstructions regardless of the chosen token sequence length we evaluate our approach in an autoregressive generation setting using a simple gpt style transformer on imagenet this approach achieves an fid controlled training data generation with diffusion models t yeo a atanov h benoit a alekseev r ray p e akhoondi a zamir tmlr 2025 website code abstract we present a method to control a text to image generative model to produce training data useful for supervised learning unlike previous works that employ an open loop approach via pre defined prompts to generate new data using either a language model or human expertise we develop an automated closed loop system that involves two feedback mechanisms the first mechanism uses feedback from a given supervised model to find adversarial prompts that result in generated images that maximize the model s loss and consequently expose its vulnerabilities while these adversarial prompts generate training examples curated for improving the given model they are not curated for a specific target distribution of interest which can be inefficient therefore we introduce the second feedback mechanism that can optionally guide the generation process towards a desirable target distribution we call the method combining these two mechanisms guided adversarial prompts the proposed closed loop system allows us to control the training data generation for a given model and target image distribution we evaluate on different tasks datasets and architectures with different types of distribution shifts corruptions spurious correlations unseen domains and illustrate the advantages of the proposed feedback mechanisms compared to open loop approaches large vision language models are unsupervised in context learners a gadetsky a atanov y jiang z gao g h mighan e amirloo a zamir m brbic iclr 2025 website code abstract recent advances in large language and vision language models have enabled zero shot inference allowing models to solve new tasks without task specific training various adaptation techniques such as prompt engineering in context learning icl and supervised fine tuning can further enhance the model s performance on a downstream task but they require substantial manual effort to construct effective prompts or labeled examples in this work we introduce a joint inference framework for fully unsupervised adaptation eliminating the need for manual prompt engineering and labeled examples unlike zero shot inference which makes independent predictions the joint inference makes predictions simultaneously for all inputs in a given task since direct joint inference involves computationally expensive optimization we develop efficient approximation techniques leading to two unsupervised adaptation methods unsupervised fine tuning and unsupervised icl 4m 21 an any to any vision model for tens of tasks and modalities r bachmann o f kar d mizrahi a garjani m gao d griffiths j hu a dehghan a zamir neurips 2024 website code demo abstract current multimodal and multitask foundation models like 4m or unifiedio show promising results but in practice their out of the box abilities to accept diverse inputs and perform diverse tasks are limited by the usually rather small number of modalities and tasks they are trained on in this paper we expand upon the capabilities of them by training a single model on tens of highly diverse modalities and by performing co training on large scale multimodal datasets and text corpora this includes training on several semantic and geometric modalities feature maps from recent state of the art models like dinov2 and imagebind pseudo labels of specialist models like sam and 4dhumans and a range of new modalities that allow for novel ways to interact with the model and steer the generation for example image metadata or color palettes a crucial step in this process is performing discrete tokenization on various modalities whether they are image like neural network feature maps vectors structured data like instance segmentation or human poses or data that can be represented as text through this we expand on the out of the box capabilities of multimodal models and specifically show the possibility of training one model to solve at least 3x more tasks modalities than existing ones and doing so without a loss in performance this enables more fine grained and controllable multimodal generation capabilities and allows us to study the distillation of models trained on diverse data and objectives into a unified model we successfully scale the training to a three billion parameter model using tens of modalities and different datasets the resulting models and training code are open sourced at 4m epfl ch solving vision tasks using simple photoreceptors instead of cameras a atanov j fu r singh i yu a spielberg a zamir eccv 2024 website abstract a de facto standard in solving computer vision problems is to use a common high resolution camera and choose its placement on an agent i e position and orientation based on human intuition on the other hand extremely simple and well designed visual sensors found throughout nature allow many organisms to perform diverse complex behaviors in this work motivated by these examples we raise the following questions 1 how effective simple visual sensors are in solving vision tasks 2 what role does their design play in their effectiveness we explore simple sensors with resolutions as low as one by one pixel representing a single photoreceptor first we demonstrate that just a few photoreceptors can be enough to solve many tasks such as visual navigation and continuous control reasonably well with performance comparable to that of a high resolution camera second we show that the design of these simple visual sensors plays a crucial role in their ability to provide useful information and successfully solve these tasks to find a well performing design we present a computational design optimization algorithm and evaluate its effectiveness across different tasks and domains showing promising results finally we perform a human survey to evaluate the effectiveness of intuitive designs devised manually by humans showing that the computationally found design is among the best designs in most cases viper visual personalization of generative models via individual preference learning s salehi m shafiei t yeo r bachmann a zamir eccv 2024 website code demo abstract different users find different images generated for the same prompt desirable this gives rise to personalized image generation which involves creating images aligned with an individual s visual preference current generative models are however unpersonalized as they are tuned to produce outputs that appeal to a broad audience using them to generate images aligned with individual users relies on iterative manual prompt engineering by the user which is inefficient and undesirable we propose to personalize the image generation process by first capturing the generic preferences of the user in a one time process by inviting them to comment on a small selection of images explaining why they like or dislike each based on these comments we infer a user s structured liked and disliked visual attributes i e their visual preference using a large language model these attributes are used to guide a text to image model toward producing images that are tuned towards the individual user s visual preference through a series of user studies and large language model guided evaluations we demonstrate that the proposed method results in generations that are well aligned with individual users visual preferences brave broadening the visual encoding of vision language models o f kar a tonioni p poklukar a kulshrestha a zamir f tombari eccv 2024 oral website abstract vision language models vlms are typically composed of a vision encoder e g clip and a language model lm that interprets the encoded features to solve downstream tasks despite remarkable progress vlms are subject to several shortcomings due to the limited capabilities of vision encoders e g blindness to certain image features visual hallucination etc to address these issues we study broadening the visual encoding capabilities of vlms we first comprehensively benchmark several vision encoders with different inductive biases for solving vlm tasks we observe that there is no single encoding configuration that consistently achieves top performance across different tasks and encoders with different biases can perform surprisingly similarly motivated by this we introduce a method named brave that consolidates features from multiple frozen encoders into a more versatile representation that can be directly fed as the input to a frozen lm brave achieves state of the art performance on a broad range of captioning and vqa benchmarks and significantly reduces the aforementioned issues of vlms while requiring a smaller number of trainable parameters than existing methods and having a more compressed representation our results highlight the potential of incorporating different visual biases for a more broad and contextualized visual understanding of vlms unraveling the key components of ood generalization via diversification h benoit l jiang a atanov o f kar m rigotti a zamir iclr 2024 pdf abstract supervised learning datasets may contain multiple cues that explain the training set equally well i e learning any of them would lead to the correct predictions on the training data however many of them can be spurious i e lose their predictive power under a distribution shift and consequently fail to generalize to out of distribution ood data recently developed diversification methods lee et al 2023 pagliardini et al 2023 approach this problem by finding multiple diverse hypotheses that rely on different features this paper aims to study this class of methods and identify the key components contributing to their ood generalization abilities we show that 1 diversification methods are highly sensitive to the distribution of the unlabeled data used for diversification and can underperform significantly when away from a method specific sweet spot 2 diversification alone is insufficient for ood generalization the choice of the used learning algorithm e g the model s architecture and pretraining is crucial in standard experiments classification on waterbirds and office home datasets using the second best choice leads to an up to 20 absolute drop in accuracy 3 the optimal choice of learning algorithm depends on the unlabeled data and vice versa i e they are co dependent 4 finally we show that in practice the above pitfalls cannot be alleviated by increasing the number of diverse hypotheses the major feature of diversification methods these findings provide a clearer understanding of the critical design factors influencing the ood generalization abilities of diversification methods they can guide practitioners in how to use the existing methods best and guide researchers in developing new better ones 4m massively multimodal masked modeling d mizrahi r bachmann o f kar t yeo m gao a dehghan a zamir neurips 2023 spotlight project website code pdf abstract current machine learning models for vision are often highly specialized and limited to a single modality and task in contrast recent large language models exhibit a wide range of capabilities hinting at a possibility for similarly versatile models in computer vision in this paper we take a step in this direction and propose a multimodal training scheme called 4m it consists of training a single unified transformer encoder decoder using a masked modeling objective across a wide range of input output modalities including text images geometric and semantic modalities as well as neural network feature maps 4m achieves scalability by unifying the representation space of all modalities through mapping them into discrete tokens and performing multimodal masked modeling on a small randomized subset of tokens 4m leads to models that exhibit several key capabilities 1 they can perform a diverse set of vision tasks out of the box 2 they excel when fine tuned for unseen downstream tasks or new input modalities and 3 they can function as a generative model that can be conditioned on arbitrary modalities enabling a wide variety of expressive multimodal editing capabilities with remarkable flexibility through experimental analyses we demonstrate the potential of 4m for training versatile and scalable foundation models for vision tasks setting the stage for further exploration in multimodal learning for vision and other domains rapid network adaptation learning to adapt neural networks using test time feedback t yeo o f kar z sodagar a zamir iccv 2023 project website code pdf abstract we propose a method for adapting neural networks to distribution shifts at test time in contrast to training time robustness mechanisms that attempt to anticipate and counter the shift we create a closed loop system and make use of a test time feedback signal to adapt a network on the fly we show that this loop can be effectively implemented using a learning based function which realizes an amortized optimizer for the network this leads to an adaptation method named rapid network adaptation rna that is notably more flexible and orders of magnitude faster than the baselines through a broad set of experiments using various adaptation signals and target tasks we study the efficiency and flexibility of this method we perform the evaluations using various datasets taskonomy replica scannet hypersim coco imagenet tasks depth optical flow semantic segmentation classification and distribution shifts cross datasets 2d and 3d common corruptions with promising results we end with a discussion on general formulations for handling distribution shifts and our observations from comparing with similar approaches from other domains modality invariant visual odometry for embodied navigation m memmel r bachmann a zamir cvpr 2023 project website code pdf abstract effectively localizing an agent in a realistic noisy setting is crucial for many embodied vision tasks visual odometry vo is a practical substitute for unreliable gps and compass sensors especially in in...