Meta tags:
author=Stephanie Fu;
description=A simple, whitespace theme for academics. Based on [*folio](htt????/github.com/bogoli/-folio) design. ;
keywords=jekyll, jekyll-theme, academic-website, portfolio-website;

Headings (most frequently used words):

stephanie, fu, news, publications, selected, all, invited, talks,

Text of the page (most frequently used words):
and (159), the (105), that (67), visual (53), models (46), vision (44), model (34), with (32), image (32), human (31), paper (30), our (30), these (29), this (26), #stephanie (24), representations (23), images (23), data (22), for (22), resolution (22), abs (21), tasks (20), their (19), perceptual (19), features (19), learning (17), vlms (17), 2025 (17), over (17), can (17), introduce (17), website (17), similarity (15), are (15), across (15), code (15), using (14), find (14), performance (14), information (14), language (14), new (13), from (13), inferences (12), within (11), existing (11), metric (11), results (11), computer (10), shape (10), retrieval (10), dataset (10), judgments (10), while (10), 2024 (10), evaluate (10), humans (10), trevor (10), darrell (10), sequential (10), video (10), cvpr (9), which (9), but (9), metrics (9), semantic (9), large (9), like (9), task (9), benchmarks (9), have (9), synthetic (8), here (8), trained (8), not (8), show (8), neurips (8), object (8), diverse (8), objects (8), phillip (8), isola (8), deep (8), prediction (8), segmentation (8), depth (8), work (8), featup (8), high (8), multi (8), about (8), vtc (8), autogaze (8), perception (7), inference (7), digital (7), fibre (7), single (7), when (7), zhang (7), more (7), between (7), all (7), despite (7), spatial (7), generation (7), alignment (7), understanding (7), best (6), foveal (6), sampling (6), integration (6), pairs (6), different (6), one (6), use (6), approaches (6), level (6), patches (6), layout (6), content (6), first (6), better (6), attributes (6), outperforms (6), framework (6), into (6), then (6), scale (6), participants (6), well (6), behaviors (6), tyler (6), bonnen (6), including (6), series (6), mtc (6), mllms (6), long (6), videos (6), mllm (6), dreamsim (5), dimensions (5), hidden (5), plain (5), sight (5), overlook (5), workshop (5), devices (5), any (5), entire (5), time (5), ability (5), cir (5), search (5), datasets (5), methods (5), identify (5), they (5), freeman (5), other (5), how (5), furthermore (5), both (5), netanel (5), tamir (5), shobhita (5), sundaram (5), even (5), consistency (5), training (5), llms (5), generated (5), location (5), benchmark (5), encoders (5), hugging (5), face (5), range (5), common (5), such (5), wide (5), environment (5), centric (5), eval (4), fomo (4), incorporating (4), ccn (4), colm (4), system (4), approach (4), through (4), real (4), neural (4), also (4), user (4), evaluating (4), fail (4), mark (4), hamilton (4), william (4), current (4), similarities (4), differences (4), pose (4), develop (4), collect (4), observers (4), recent (4), text (4), being (4), lucy (4), chai (4), semantics (4), enabling (4), downstream (4), zero (4), shot (4), however (4), directly (4), perform (4), because (4), agnostic (4), reconstruct (4), original (4), end (4), does (4), natural (4), modeling (4), open (4), design (4), given (4), set (4), trials (4), gaze (4), evaluation (4), via (4), understand (4), general (4), purpose (4), investigate (4), aligning (4), estimation (4), stereoscopic (4), present (4), integrate (4), throughout (4), top (4), been (4), temporal (4), cortex (4), integrates (4), contemporary (4), depend (4), locations (4), attend (4), vits (4), redundancy (4), before (4), david (4), 2026 (3), 2023 (3), oct (3), essential (3), capabilities (3), applications (3), scalable (3), presents (3), create (3), provide (3), domain (3), transfer (3), demonstrate (3), where (3), distribution (3), provides (3), unique (3), generalizes (3), approximate (3), evaluations (3), measures (3), yield (3), iclr (3), pixels (3), terms (3), achieve (3), various (3), fall (3), learned (3), richard (3), tali (3), dekel (3), research (3), often (3), lack (3), view (3), preliminary (3), representation (3), llm (3), highlight (3), relevant (3), standard (3), has (3), utility (3), state (3), art (3), many (3), cognitive (3), abstract (3), foundation (3), future (3), lead (3), previously (3), out (3), knowledge (3), benefit (3), rapid (3), effectively (3), quality (3), experience (3), stereo (3), than (3), outstanding (3), oral (3), way (2), jun (2), aug (2), building (2), modern (2), fibres (2), containing (2), enable (2), fabrics (2), physiological (2), body (2), machine (2), used (2), temperature (2), memory (2), increasing (2), connections (2), networks (2), infer (2), blog (2), wang (2), johnny (2), ioannis (2), yoel (2), fink (2), based (2), mosaic (2), allows (2), users (2), span (2), conditional (2), structures (2), finally (2), lei (2), recommendation (2), architectures (2), complex (2), several (2), techniques (2), axiomatic (2), shapley (2), taylor (2), indices (2), explanations (2), operate (2), compare (2), low (2), colors (2), textures (2), capture (2), mid (2), assesses (2), holistically (2), step (2), alike (2), ways (2), critical (2), nearly (2), automatic (2), shared (2), perturbed (2), along (2), observe (2), popular (2), short (2), explaining (2), tuned (2), align (2), analyze (2), affected (2), focuses (2), heavily (2), foreground (2), sensitive (2), color (2), notably (2), giving (2), strong (2), reconstruction (2), prior (2), spotlight (2), cornerstone (2), capturing (2), community (2), solve (2), few (2), regime (2), dense (2), aggressively (2), pool (2), areas (2), restore (2), lost (2), two (2), variants (2), guides (2), signal (2), forward (2), pass (2), fits (2), implicit (2), loss (2), analogies (2), nerfs (2), retain (2), swapped (2), gains (2), without (2), significantly (2), feature (2), upsampling (2), super (2), class (2), activation (2), map (2), laura (2), brandt (2), axel (2), feldman (2), zhoutong (2), what (2), strings (2), teach (2), world (2), aspects (2), potential (2), train (2), just (2), check (2), makes (2), access (2), address (2), openstreetview (2), nguyen (2), leverage (2), experimental (2), sciences (2), requires (2), contain (2), same (2), considerable (2), viewpoint (2), variation (2), draw (2), include (2), chairs (2), shapes (2), procedurally (2), nonsense (2), after (2), constructing (2), 2000 (2), sets (2), administer (2), collecting (2), 35k (2), behavioral (2), 500 (2), includes (2), explicit (2), choice (2), intermediate (2), reaction (2), dinov2 (2), mae (2), clip (2), outperform (2), margin (2), underlying (2), correlated (2), allocate (2), processing (2), challenging (2), accessed (2), project (2), page (2), yutong (2), bai (2), thomas (2), connell (2), yoni (2), friedman (2), nancy (2), kanwisher (2), joshua (2), tenenbaum (2), alexei (2), efros (2), multiview (2), focused (2), frontier (2), governance (2), posed (2), broadly (2), limitations (2), judge (2), according (2), scene (2), subject (2), camera (2), abstractions (2), improperly (2), weigh (2), thus (2), make (2), misaligned (2), benefited (2), contexts (2), perceptually (2), aligned (2), settings (2), remains (2), unclear (2), impacts (2), usability (2), finetune (2), triplets (2), them (2), yields (2), improve (2), upon (2), backbones (2), counting (2), instance (2), augmented (2), addition (2), widely (2), preserved (2), specialized (2), domains (2), medical (2), imaging (2), frames (2), suggest (2), injecting (2), inductive (2), bias (2), contribute (2), lukas (2), muttenthaler (2), simon (2), kornblith (2), headsets (2), preference (2), preferences (2), isqoe (2), interface (2), specify (2), realize (2), possibility (2), must (2), successfully (2), linguistic (2), compares (2), direct (2), readout (2), modalities (2), correspondence (2), substantially (2), worse (2), dropping (2), near (2), chance (2), analyses (2), vlm (2), namely (2), degradation (2), brittleness (2), prompt (2), role (2), solving (2), bottleneck (2), performing (2), lies (2), third (2), category (2), easily (2), accessible (2), inherit (2), priors (2), helps (2), diagnose (2), failure (2), modes (2), source (2), useful (2), investigations (2), award (2), devin (2), guillory (2), inherently (2), largely (2), constraints (2), retina (2), demand (2), shift (2), substrates (2), support (2), characterized (2), ventral (2), rapidly (2), extracts (2), each (2), medial (2), outputs (2), neurocomputational (2), motif (2), absent (2), surprisingly (2), those (2), novel (2), embodies (2), strategy (2), emblematic (2), determines (2), sequentially (2), processes (2), foveated (2), inputs (2), report (2), choices (2), stable (2), optimization (2), subsequent (2), modal (2), advanced (2), struggle (2), process (2), every (2), pixel (2), equally (2), transformers (2), significant (2), spatiotemporal (2), lightweight (2), module (2), removes (2), redundant (2), processed (2), vit (2), next (2), token (2), reinforcement (2), autoregressively (2), selects (2), minimal (2), specified (2), error (2), threshold (2), eliminating (2), preserving (2), empirically (2), reduces (2), tokens (2), 100x (2), accelerates (2), 19x (2), scaling (2), frame (2), achieving (2), superior (2), videomme (2), hlvid (2), form (2), minute (2), scaled (2), improves (2), baseline (2), previous (2), baifeng (2), shi (2), lian (2), hanrong (2), eigen (2), aaron (2), reite (2), boyi (2), jan (2), kautz (2), song (2), han (2), chan (2), pavlo (2), molchanov (2), hongxu (2), yin (2), attention (2), efficient (2), autoregressive (2), gazing (2), equal (2), publications (2), selected (2), was (2), conference (2), mit (2), tedxmit (2), science (2), advised (2), phd (2), student (2), copyright, powered, theme, hosted, github, pages, folio, jekyll, mail, usually, reach, voxel51, meetup, jul, lessons, columbia, university, kriegeskorte, lab, invited, talks, blocks, electronic, could, monitoring, interfaces, preform, produce, tens, metres, flexible, hundreds, interspersed, sensors, density, thinspace, texttimes, thinspace105, bits, per, metre, ensemble, individually, addressable, independently, operated, connection, edge, overcoming, perennial, device, limitation, reliability, incorporated, shirt, collects, stores, multiple, days, enables, wearer, activity, accuracy, network, 1650, neuronal, stored, realise, strand, only, measure, store, parameters, harbour, required, sensory, intriguing, opportunities, worn, sense, memorise, learn, situational, context, nature, communications, gabriel, loke, tural, khudiyev, brian, syamantak, payra, yorai, shaoul, fung, chatziveroglou, pin, wen, chou, itamar, chinn, wei, yan, anna, gitelson, kahn, john, joannopoulos, electronics, fabric, interactive, web, app, semantically, related, artworks, cultures, media, millennia, application, combines, supplied, filters, conditions, technique, similar, distinct, subsets, corpus, generic, adapt, theoretical, bounds, efficiency, quantify, systems, performs, non, parametric, style, blind, spots, generative, adversarial, gan, properly, true, 2020, competition, demonstration, track, mindren, bui, darius, bopp, zhenbang, chen, felix, tran, margaret, marina, rogers, chris, hoder, finding, artistic, culture, contrastive, power, technologies, impact, billions, worldwide, difficult, interpret, there, competing, explain, engine, behavior, theory, fair, credit, assignment, solution, explainability, literature, formalism, violate, fairness, derive, sidestep, shortcomings, naturally, handle, counterfactual, specifically, implicitly, second, order, extend, cam, gradcam, lime, shap, sbsm, engines, extensions, extract, pairwise, correspondences, opaque, box, fast, kernel, method, estimating, require, orders, magnitude, fewer, function, converge, game, theoretic, consistent, 2022, scott, lundberg, relationships, systematically, abilities, generate, recognize, assortment, concepts, complexity, consume, output, represent, study, although, look, correct, indicate, precise, numerous, experiments, self, supervised, utilizing, capable, making, assessments, pratyusha, sharma, tamar, rott, shaham, manel, baradad, adrian, rodriguez, munoz, shivam, duggal, antonio, torralba, determining, anywhere, earth, particularly, algorithms, yet, absence, reliably, localizable, limited, its, issue, comprising, million, geo, referenced, street, covering, 225, countries, territories, contrast, enforce, strict, test, separation, allowing, relevance, geographical, beyond, mere, memorization, conduct, extensive, strategies, guillaume, astruc, nicolas, dufour, siglidis, constantin, aronssohn, nacim, bouia, romain, loiseau, van, charles, raude, elliot, vincent, lintao, hongyu, zhou, loic, landrieu, roads, global, geolocation, regulations, powerful, narrowly, vague, inconsistently, defined, leading, unstable, efforts, critically, policy, debates, consider, clear, link, relatively, small, outside, typical, definitions, equivalent, outcomes, exposed, sufficiently, specific, illustrate, importance, considering, size, factors, assessing, risks, today, emphasize, risk, regulating, reactively, path, towards, careful, quantitative, simplified, regulatory, ritwik, gupta, leah, walker, rodolfo, corona, suzanne, petryk, janet, napolitano, andrew, reddie, addressing, policies, advancements, virtual, reality, measuring, sqoe, become, delivering, immersive, comfortable, experiences, most, focus, isolated, viewing, discomfort, traditionally, faced, gaps, scope, comprised, featuring, distortions, artifacts, labeled, annotations, collected, headset, findings, indicating, notable, degree, additionally, assessment, aligns, comparing, mono, conversion, cv4metaverse, shir, amir, ranel, itzhaky, noam, atia, ron, sokolovsky, miriam, farber, good, contribution, advising, released, will, presenting, poster, won, extended, computational, neuroscience, news, 2019, founded, helped, launch, inaugural, csail, since, brought, communities, greater, boston, area, together, regular, events, speakers, past, upcoming, conferences, interested, recently, thinking, intelligence, build, graduated, meng, bachelors, degrees, music, during, undergrad, fortunate, exciting, under, bill, bair, supported, college, engineering, fellowship, currently, funded, nsf, grfp, eecs, berkeley, teaching, toggle, navigation,

Text of the page (random words):
new metric dreamsim tuned to better align with human perception we analyze how our metric is affected by different visual attributes and find that it focuses heavily on foreground objects and semantic content while also being sensitive to color and layout notably despite being trained on synthetic data our metric generalizes to real images giving strong results on retrieval and reconstruction tasks furthermore our metric outperforms both prior learned metrics and recent large vision models on these tasks attend before attention efficient and scalable video understanding via autoregressive gazing baifeng shi stephanie fu long lian hanrong ye david eigen aaron reite boyi li jan kautz song han david m chan pavlo molchanov trevor darrell and hongxu yin cvpr 2026 highlight abs paper code hugging face website multi modal large language models mllms have advanced general purpose video understanding but struggle with long high resolution videos they process every pixel equally in their vision transformers vits or llms despite significant spatiotemporal redundancy we introduce autogaze a lightweight module that removes redundant patches before processed by a vit or an mllm trained with next token prediction and reinforcement learning autogaze autoregressively selects a minimal set of multi scale patches that can reconstruct the video within a user specified error threshold eliminating redundancy while preserving information empirically autogaze reduces visual tokens by 4x 100x and accelerates vits and mllms by up to 19x enabling scaling mllms to 1k frame 4k resolution videos and achieving superior results on video benchmarks e g 67 0 on videomme furthermore we introduce hlvid the first high resolution long form video qa benchmark with 5 minute 4k resolution videos where an mllm scaled with autogaze improves over the baseline by 10 1 and outperforms the previous best mllm by 4 5 incorporating foveal sampling and integration to model 3d shape inferences stephanie fu trevor darrell and tyler bonnen ccn 2025 oral top 6 5 abs paper human vision is inherently sequential this is largely because of foveal constraints on the retina which demand that we shift our gaze to collect high resolution information from throughout the environment within the domain of visual object perception the neural substrates that support these sequential visual inferences have been well characterized ventral temporal cortex vtc rapidly extracts visual features at each spatial location while medial temporal cortex mtc integrates over the sequential outputs of vtc this neurocomputational motif is absent in contemporary deep learning models of human vision not surprisingly contemporary models approximate the rapid visual inferences that depend on vtc but not those behaviors that depend on mtc eg novel 3d shape inference here we develop a modeling framework that embodies the sequential sampling integration strategy emblematic of human vision given an image this model first determines relevant locations to attend to sequentially processes these locations as foveated inputs using a vtc like model then integrates over these sequential visual features within a mtc like model here we report preliminary results on the design choices that lead to stable model optimization and subsequent model behaviors hidden in plain sight vlms overlook their visual representations stephanie fu tyler bonnen devin guillory and trevor darrell colm 2025 outstanding paper top 0 9 cvpr eval fomo 2 workshop best paper award abs paper website language provides a natural interface to specify and evaluate performance on visual tasks to realize this possibility vision language models vlms must successfully integrate visual and linguistic information our work compares vlms to a direct readout of their visual encoders to understand their ability to integrate across these modalities across a series of vision centric benchmarks e g depth estimation correspondence we find that vlms perform substantially worse than their visual encoders dropping to near chance performance we investigate these results through a series of analyses across the entire vlm namely 1 the degradation of vision representations 2 brittleness to task prompt and 3 the language model s role in solving the task we find that the bottleneck in performing these vision centric tasks lies in this third category vlms are not effectively using visual information easily accessible throughout the entire model and they inherit the language priors present in the llm our work helps diagnose the failure modes of open source vlms and presents a series of evaluations useful for future investigations into visual understanding within vlms what makes for a good stereoscopic image netanel y tamir shir amir ranel itzhaky noam atia shobhita sundaram stephanie fu ron sokolovsky phillip isola tali dekel richard zhang and miriam farber cvpr 2025 cv4metaverse workshop abs paper website with rapid advancements in virtual reality vr headsets effectively measuring stereoscopic quality of experience sqoe has become essential for delivering immersive and comfortable 3d experiences however most existing stereo metrics focus on isolated aspects of the viewing experience such as visual discomfort or image quality and have traditionally faced data limitations to address these gaps we present scope stereoscopic content preference evaluation a new dataset comprised of real and synthetic stereoscopic images featuring a wide range of common perceptual distortions and artifacts the dataset is labeled with preference annotations collected on a vr headset with our findings indicating a notable degree of consistency in user preferences across different headsets additionally we present isqoe a new model for stereo quality of experience assessment trained on our dataset we show that isqoe aligns better with human preferences than existing methods when comparing mono to stereo conversion methods when does perceptual alignment benefit vision representations shobhita sundaram stephanie fu lukas muttenthaler netanel y tamir lucy chai simon kornblith trevor darrell and phillip isola neurips 2024 abs paper code website humans judge perceptual similarity according to diverse visual attributes including scene layout subject location and camera pose existing vision models understand a wide range of semantic abstractions but improperly weigh these attributes and thus make inferences misaligned with human perception while vision representations have previously benefited from alignment in contexts like image generation the utility of perceptually aligned representations in more general purpose settings remains unclear here we investigate how aligning vision model representations to human perceptual judgments impacts their usability across diverse computer vision tasks we finetune state of the art models on human similarity judgments for image triplets and evaluate them across standard vision benchmarks we find that aligning models to perceptual judgments yields representations that improve upon the original backbones across many downstream tasks including counting segmentation depth estimation instance retrieval and retrieval augmented generation in addition we find that performance is widely preserved on other tasks including specialized out of distribution domains such as in medical imaging and 3d environment frames our results suggest that injecting an inductive bias about human perceptual knowledge into vision models can contribute to better representations data centric ai governance addressing the limitations of model focused policies ritwik gupta leah walker rodolfo corona stephanie fu suzanne petryk janet napolitano trevor darrell and andrew w reddie abs paper current regulations on powerful ai capabilities are narrowly focused on foundation or frontier models however these terms are vague and inconsistently defined leading to an unstable foundation for governance efforts critically policy debates often fail to consider the data used with these models despite the clear link between data and model performance even relatively small models that fall outside the typical definitions of foundation and frontier models can achieve equivalent outcomes when exposed to sufficiently specific datasets in this work we illustrate the importance of considering dataset size and content as essential factors in assessing the risks posed by models both today and in the future more broadly we emphasize the risk posed by over regulating reactively and provide a path towards careful quantitative evaluation of capabilities that can lead to a simplified regulatory environment evaluating multiview object consistency in humans and image models tyler bonnen stephanie fu yutong bai thomas o connell yoni friedman nancy kanwisher joshua b tenenbaum and alexei a efros neurips datasets benchmarks 2024 abs paper code hugging face website we introduce a benchmark to directly evaluate the alignment between human observers and vision models on a 3d shape inference task we leverage an experimental design from the cognitive sciences which requires zero shot visual inferences about object shape given a set of images participants identify which contain the same different objects despite considerable viewpoint variation we draw from a diverse range of images that include common objects e g chairs as well as abstract shapes i e procedurally generated nonsense objects after constructing over 2000 unique image sets we administer these tasks to human participants collecting 35k trials of behavioral data from over 500 participants this includes explicit choice behaviors as well as intermediate measures such as reaction time and gaze data we then evaluate the performance of common vision models e g dinov2 mae clip we find that humans outperform all models by a wide margin using a multi scale evaluation approach we identify underlying similarities and differences between models and humans while human model performance is correlated humans allocate more time processing on challenging trials all images data and code can be accessed via our project page openstreetview 5m the many roads to global visual geolocation guillaume astruc nicolas dufour ioannis siglidis constantin aronssohn nacim bouia stephanie fu romain loiseau van nguyen nguyen charles raude elliot vincent lintao xu hongyu zhou and loic landrieu cvpr 2024 abs paper code hugging face website determining the location of an image anywhere on earth is a complex visual task which makes it particularly relevant for evaluating computer vision algorithms yet the absence of standard large scale open access datasets with reliably localizable images has limited its potential to address this issue we introduce openstreetview 5m a large scale open access dataset comprising over 5 1 million geo referenced street view images covering 225 countries and territories in contrast to existing benchmarks we enforce a strict train test separation allowing us to evaluate the relevance of learned geographical features beyond mere memorization to demonstrate the utility of our dataset we conduct an extensive benchmark of various state of the art image encoders spatial representations and training strategies a vision check up for language models pratyusha sharma tamar rott shaham manel baradad stephanie fu adrian rodriguez munoz shivam duggal phillip isola and antonio torralba cvpr 2024 abs paper website what does learning to model relationships between strings teach large language models llms about the visual world we systematically evaluate llms abilities to generate and recognize an assortment of visual concepts of increasing complexity and then demonstrate how a preliminary visual representation learning system can be trained using models of text as language models lack the ability to consume or output visual information as pixels we use code to represent images in our study although llm generated images do not look like natural images results on image generation and the ability of models to correct these generated images indicate that precise modeling of strings can teach language models about numerous aspects of the visual world furthermore experiments on self supervised visual representation learning utilizing images generated with text models highlight the potential to train vision models capable of making semantic assessments of natural images using just llms featup a model agnostic framework for features at any resolution stephanie fu mark hamilton laura brandt axel feldman zhoutong zhang and william t freeman iclr 2024 abs paper code website deep features are a cornerstone of computer vision research capturing image semantics and enabling the community to solve downstream tasks even in the zero or few shot regime however these features often lack the spatial resolution to directly perform dense prediction tasks like segmentation and depth prediction because models aggressively pool information over large areas in this work we introduce featup a task and model agnostic framework to restore lost spatial information in deep features we introduce two variants of featup one that guides features with high resolution signal in a single forward pass and one that fits an implicit model to a single image to reconstruct features at any resolution both approaches use a multi view consistency loss with deep analogies to nerfs our features retain their original semantics and can be swapped into existing applications to yield resolution and performance gains even without re training we show that featup significantly outperforms other feature upsampling and image super resolution approaches in class activation map generation transfer learning for segmentation and depth prediction and end to end training for semantic segmentation dreamsim learning new dimensions of human visual similarity using synthetic data stephanie fu netanel tamir shobhita sundaram lucy chai richard zhang tali dekel and phillip isola neurips 2023 spotlight abs paper code website current perceptual similarity metrics operate at the level of pixels and patches these metrics compare images in terms of their low level colors and textures but fail to capture mid level similarities and differences in image layout object pose and semantic content in this paper we develop a perceptual metric that assesses images holistically our first step is to collect a new dataset of human similarity judgments over image pairs that are alike in diverse ways critical to this dataset is that judgments are nearly automatic and shared by all observers to achieve this we use recent text to image models to create synthetic pairs that are perturbed along various dimensions we observe that popular perceptual metrics fall short of explaining our new data and we introduce a new metric dreamsim tuned to better align with human perception we analyze how our metric is affected by different visual attributes and find that it focuses heavily on foreground o...