Meta tags:
description=3D-VisTA;
keywords=3D-VisTA;

Headings (most frequently used words):

3d, vista, pre, trained, transformer, for, vision, and, text, alignment, tasks, tl, dr, we, propose, that, can, be, easily, adapted, to, various, downstream, abstract, contribution, scanscribe, dataset, model, examples, on, 3dvl, bibtex,

Text of the page (most frequently used words):
and (39), the (20), for (17), vista (16), text (15), from (11), pre (11), scene (11), tasks (8), transformer (7), trained (6), vision (6), #scanscribe (6), #alignment (5), that (5), training (5), scenes (5), self (5), various (5), downstream (5), scans (5), dataset (5), with (5), results (4), object (4), fine (4), scannet (4), descriptions (4), language (4), task (4), visual (3), simple (3), unified (3), point (3), cloud (3), via (3), module (3), are (3), modal (3), can (3), easily (3), model (3), rgb (3), indoor (3), scan (3), question (3), pairs (3), annotations (3), 278k (3), grounding (3), performance (3), modeling (3), propose (3), ziyu (2), zhu (2), xiaojian (2), yixin (2), chen (2), zhidong (2), deng (2), siyuan (2), huang (2), qing (2), scanrefer (2), scanqa (2), improves (2), aligning (2), sentence (2), first (2), encoding (2), multi (2), fusion (2), supervised (2), existing (2), datasets (2), based (2), both (2), templates (2), gpt (2), generated (2), state (2), art (2), ranging (2), answering (2), situated (2), reasoning (2), demonstrates (2), superior (2), data (2), efficiency (2), obtaining (2), strong (2), even (2), limited (2), masked (2), matching (2), further (2), tuning (2), construct (2), large (2), scale (2), contains (2), 995 (2), 185 (2), unique (2), simply (2), utilizes (2), attention (2), without (2), any (2), specific (2), design (2), which (2), intelligence (2), sophisticated (2), adapted (2), website, template, borrowed, nerfies, inproceedings, 3dvista, title, author, booktitle, iccv, year, 2023, bibtex, qualitative, sqa3d, green, red, denote, ground, truth, predicted, boxes, respectively, show, understanding, spatial, relations, concepts, situations, examples, 3dvl, takes, pair, input, encodes, processes, then, tokens, fused, capture, correspondence, between, objects, using, learning, tuned, collect, transform, into, including, answer, referring, expressions, referit3d, adopt, generate, their, graph, ultimately, collected, tune, achieve, per, formances, dense, captioning, also, introduce, scheme, effectively, learns, simplifies, superised, proposed, mechanism, complex, our, main, contributions, contribution, enhance, its, originating, along, paired, achieves, moreover, during, emerging, field, aims, connect, physical, world, natural, crucial, achieving, embodied, current, models, rely, heavily, modules, auxiliary, losses, optimization, tricks, calls, this, paper, layers, single, abstract, code, arxiv, beijing, institute, general, artificial, bigai, tsinghua, university,

Text of the page (random words):
3d vista 3d vista pre trained transformer for 3d vision and text alignment ziyu zhu 1 xiaojian ma 2 yixin chen 2 zhidong deng 1 siyuan huang 2 qing li 2 1 tsinghua university 2 beijing institute for general artificial intelligence bigai arxiv code dataset tl dr we propose 3d vista a pre trained transformer for 3d vision and text alignment that can be easily adapted to various downstream tasks abstract 3d vision language grounding 3d vl is an emerging field that aims to connect the 3d physical world with natural language which is crucial for achieving embodied intelligence current 3d vl models rely heavily on sophisticated modules auxiliary losses and optimization tricks which calls for a simple and unified model in this paper we propose 3d vista a pre trained transformer for 3d vision and text alignment that can be easily adapted to various downstream tasks 3d vista simply utilizes self attention layers for both single modal modeling and multi modal fusion without any sophisticated task specific design to further enhance its performance on 3d vl tasks we construct scanscribe the first large scale 3d scene text pairs dataset for 3d vl pre training scanscribe contains 2 995 rgb d scans for 1 185 unique indoor scenes originating from scannet and 3r scan datasets along with paired 278k scene descriptions generated from existing 3d vl tasks templates and gpt 3 3d vista is pre trained on scanscribe via masked language object modeling and scene text matching it achieves state of the art results on various 3d vl tasks ranging from visual grounding and question answering to situated reasoning moreover 3d vista demonstrates superior data efficiency obtaining strong performance even with limited annotations during downstream task fine tuning contribution our main contributions are 3d vista model we propose 3d vista a simple and unified transformer for aligning 3d vision and text the proposed transformer simply utilizes the self attention mechanism without any complex task specific design scanscribe dataset we construct scanscribe a large scale 3d vl pre training dataset that contains 278k 3d scene text pairs for 2 995 rgb d scans of 1 185 unique indoor scenes 3d self superised pre training we introduce a self supervised pre training scheme for 3d vl with masked language object modeling and scene text matching it effectively learns the 3d point cloud and text alignment and further simplifies and improves downstream task fine tuning performance we fine tune 3d vista and achieve state of the art per formances on various 3d vl tasks ranging from visual grounding question answering and dense captioning to situated reasoning 3d vista also demonstrates superior data efficiency obtaining strong results even with limited annotations scanscribe dataset we collect rgb d scans of indoor scenes from scannet and 3r scan for the scans from scannet we transform the text from existing datasets based on scannet into scene descriptions including the question answer pairs from scanqa and the referring expressions from scanrefer and referit3d for the scans from 3r scan we adopt both templates and gpt 3 to generate scene descriptions based on their scene graph annotations ultimately 278k scene descriptions are generated for the collected 3d scenes 3d vista model 3d vista is a simple and unified transformer for aligning 3d scenes and text 3d vista takes a pair of scene point cloud and sentence as input it first encodes the sentence via a text encoding module and processes the point cloud via a scene encoding module then the text and 3d object tokens are fused by a multi modal fusion module to capture the correspondence between 3d objects and text 3d vista is pre trained using self supervised learning and can be easily fine tuned to various downstream tasks examples on 3dvl tasks qualitative results on scanrefer scanqa and sqa3d in a b c d e green and red denote the ground truth and predicted object boxes respectively the results show that pre training improves the understanding of spatial relations visual concepts and situations s bibtex inproceedings 3dvista title 3d vista pre trained transformer for 3d vision and text alignment author ziyu zhu and xiaojian ma and yixin chen and zhidong deng and siyuan huang and qing li booktitle iccv year 2023 website template borrowed from nerfies