Meta tags:
description=SplatTalk;
keywords=SplatTalk;

Headings (most frequently used words):

splattalk, 3d, vqa, with, gaussian, splatting, abstract, method, results, paper, poster, video, iccv, 2025,

Text of the page (most frequently used words):
the (15), language (12), gaussian (10), are (7), and (7), features (6), visual (6), #splatting (5), for (5), with (5), dimensional (5), llm (5), vqa (5), from (4), during (4), into (4), feature (4), that (4), models (4), splattalk (4), images (4), model (4), this (3), these (3), then (3), low (3), maps (3), high (3), direct (3), inputs (3), space (3), pretrained (3), self (2), supervised (2), multiple (2), rgb (2), input (2), first (2), encoded (2), latent (2), specifically (2), tokens (2), inference (2), directly (2), using (2), method (2), enabling (2), while (2), vision (2), tasks (2), question (2), answering (2), only (2), our (2), trained (2), website, template, borrowed, nerfies, video, paper, poster, results, training, phase, views, triplets, decoded, parameters, rendering, along, ensure, proper, supervision, train, autoencoder, unbounded, obtained, llava, serving, onto, hypersphere, extracted, gaussians, mapped, back, original, decoder, subsequently, used, token, lora, fine, tuning, optional, click, zoom, right, left, guided, scene, understanding, important, advancing, applications, robotics, human, computer, interaction, comprehend, interact, environments, through, natural, vlms, have, achieved, remarkable, success, progress, domain, has, been, significantly, slower, due, complexity, data, cost, manual, annotations, work, introduce, novel, uses, generalizable, 3dgs, framework, produce, suitable, effective, zero, shot, scenes, posed, experiments, benchmarks, approach, outperforms, both, task, previous, lmm, based, utilizing, setting, achieving, competitive, performance, state, art, lmms, additionally, utilize, abstract, propose, multi, view, vlm, projected, via, multimodal, projector, learned, within, feed, forward, producing, field, encodes, spatial, semantic, information, queried, large, perform, overview, code, arxiv, google, deepmind, georgia, institute, technology, thomas, funkhouser, leonidas, guibas, kyle, genova, songyou, peng, anh, thai, iccv, 2025,

Text of the page (random words):
splattalk splattalk 3d vqa with gaussian splatting iccv 2025 anh thai 1 2 songyou peng 2 kyle genova 2 leonidas guibas 2 thomas funkhouser 2 1 georgia institute of technology 2 google deepmind arxiv code overview of splattalk we propose a self supervised 3d language gaussian splatting model trained from multi view rgb images first images are encoded using a pretrained 2d vision language model vlm and projected into visual language feature maps via a multimodal projector these feature maps are then learned within a feed forward 3d language gaussian splatting model producing a 3d language gaussian field that encodes spatial and semantic information in 3d space during inference the 3d gaussian features are directly queried by a large language model llm to perform 3d question answering 3d vqa tasks abstract language guided 3d scene understanding is important for advancing applications in robotics ar vr and human computer interaction enabling models to comprehend and interact with 3d environments through natural language while 2d vision language models vlms have achieved remarkable success in 2d vqa tasks progress in the 3d domain has been significantly slower due to the complexity of 3d data and the high cost of manual annotations in this work we introduce splattalk a novel method that uses a generalizable 3d gaussian splatting 3dgs framework to produce 3d tokens suitable for direct input into a pretrained llm enabling effective zero shot 3d visual question answering 3d vqa for scenes with only posed images during experiments on multiple benchmarks our approach outperforms both 3d models trained specifically for the task and previous 2d lmm based models utilizing only images our setting while achieving competitive performance with state of the art 3d lmms that additionally utilize 3d inputs method left during the self supervised 3d language gaussian splatting training phase multiple rgb input views are first encoded into gaussian latent features gaussian triplets these latent features are then decoded into gaussian parameters for rendering along with a low dimensional visual language feature to ensure proper supervision of this low dimensional feature we train an autoencoder that maps the high dimensional unbounded features obtained from llava ov specifically the visual tokens serving as direct inputs to the llm onto a low dimensional hypersphere space right during 3d vqa inference visual language features are directly extracted from the 3d gaussians these features are then mapped back to the original high dimensional space using the pretrained decoder and subsequently used as direct visual token inputs to the llm lora fine tuning of the llm is optional click to zoom results paper poster video this website template is borrowed from nerfies