Meta tags:
description=Real-3DQA;
keywords=3D Spatial Reasoning, 3D Question Answering, 3D Large Language Models, 3D Vision-Language, Benchmark;

Headings (most frequently used words):

3d, understanding, rotation, what, viewpoint, llms, do, language, understand, are, real, for, still, after, situated, large, models, really, spatial, relationships, current, llm, benchmarks, measuring, should, benchmark, genuine, look, like, consistency, test, visualizer, did, we, discover, next, bibtex, strong, debiasing, sqa3d, to, 3dqa, iclr, 2026, toward, finer, grained, invariant, model, design, beyond, qa, grounding, captioning, need, awareness, too, rethinking, the, role, of, in,

Text of the page (most frequently used words):
the (29), and (24), llms (11), understanding (11), viewpoint (10), #spatial (9), benchmarks (9), model (9), #rotation (8), genuine (7), for (7), not (7), real (7), language (6), models (6), should (6), questions (6), sqa3d (6), this (5), can (5), that (5), linguistic (5), reasoning (5), current (5), observer (5), scene (5), correctly (5), llm (5), answer (5), question (5), scores (5), 3dqa (5), with (5), blind (5), understand (4), relationships (4), iro (4), how (4), situated (4), must (4), our (4), what (4), part (4), shortcuts (4), existing (4), text (4), only (4), benchmark (4), university (4), from (3), under (3), findings (3), problem (3), performance (3), knowledge (3), open (3), shortcut (3), these (3), captioning (3), data (3), invariant (3), just (3), evaluate (3), viewpoints (3), truly (3), all (3), zero (3), answers (3), 180 (3), 270 (3), after (3), strong (3), leo (3), same (3), your (3), does (3), video (3), test (3), international (2), title (2), large (2), really (2), author (2), xianzheng (2), tao (2), sun (2), shuai (2), chen (2), yash (2), bhalgat (2), jindong (2), angel (2), chang (2), armeni (2), laina (2), songyou (2), peng (2), victor (2), adrian (2), prisacariu (2), learning (2), representations (2), 2026 (2), arxiv (2), future (2), demo (2), solution (2), role (2), but (2), common (2), sense (2), while (2), key (2), priors (2), egocentric (2), situation (2), aware (2), direction (2), grounding (2), architectures (2), important (2), toward (2), design (2), directions (2), proportion (2), answered (2), across (2), its (2), takeaway (2), different (2), rotate (2), correct (2), still (2), debiasing (2), revealing (2), dramatic (2), drop (2), comparison (2), reveals (2), are (2), rotating (2), browser (2), support (2), tag (2), regardless (2), because (2), geometry (2), change (2), pipeline (2), through (2), alone (2), like (2), fundamental (2), full (2), isn (2), one (2), achieve (2), various (2), input (2), equal (2), oxford (2), template, borrowed, nerfies, website, licensed, creative, commons, attribution, sharealike, license, inproceedings, ma2026real3dqa, booktitle, fourteenth, conference, year, url, https, org, abs, 2603, 23523, copy, bibtex, plays, dual, inflates, also, provides, valuable, properly, leverage, preventing, bypassing, remains, challenge, world, rethinking, similar, inconsistency, issues, exist, however, areas, lack, descriptions, building, evaluation, tasks, promising, necessary, beyond, need, awareness, too, training, pipelines, have, introducing, augmentation, equivariant, step, explicit, robustness, true, comprehension, grounded, perspective, level, overview, specific, positions, within, finer, grained, point, several, vision, community, next, every, meaning, maintain, simple, changes, becomes, clear, four, near, replay, numbers, rotated, overall, vrs, each, check, many, times, higher, proportions, indicate, suffer, decrease, their, high, were, largely, driven, inflated, em_r, did, discover, prediction, click, drag, view, scroll, zoom, out, existence, scene0025, distance, scene0389, counting, scene0690, scene0278, observe, receives, rotates, place, visualizer, generates, equivalent, them, understands, doesn, keeping, fixed, updating, accordingly, consistency, construction, comparing, counterparts, general, eliminate, solvable, based, principles, build, below, comprehends, produce, underlying, structure, consistently, genuinely, answerable, way, matching, accuracy, those, fail, perception, cheat, bypass, argue, reliable, satisfy, two, criteria, look, means, evaluations, significantly, overestimate, ability, masking, limitations, shown, limited, pattern, holds, competitive, multiple, matches, even, outperforms, between, widely, used, generalist, capable, dialogue, task, planning, navigation, state, art, impressive, suggesting, happens, when, train, measuring, minute, intro, help, you, quickly, grasp, core, insights, paper, sound, recommended, find, match, exposing, flaw, plain, careful, might, guessing, without, seeing, code, contribution, correspondence, supervision, google, deepmind, simon, fraser, stanford, avl, vgg, initial, iclr,

Text of the page (random words):
real 3dqa do 3d large language models really understand 3d spatial relationships iclr 2026 initial scores 8 8 6 6 xianzheng ma 1 2 tao sun 3 shuai chen 1 2 yash bhalgat 1 jindong gu 5 angel x chang 4 iro armeni 3 iro laina 1 songyou peng 5 victor adrian prisacariu 2 1 vgg university of oxford 2 avl university of oxford 3 stanford university 4 simon fraser university 5 google deepmind equal contribution correspondence author equal supervision arxiv demo code data be careful your 3d llm isn t understanding 3d it might be just guessing without seeing we find that plain llms with zero 3d input can match full 3d llms on existing benchmarks exposing a fundamental flaw in how we evaluate 3d spatial reasoning your browser does not support the video tag a 1 minute intro video to help you quickly grasp the core insights of our paper sound on recommended part 1 the problem are current 3d llm benchmarks measuring real 3d understanding state of the art 3d llms achieve impressive scores on benchmarks like sqa3d suggesting strong 3d spatial reasoning but what happens when we train a blind text only model with zero 3d input on the same data and is this just a problem with one benchmark a various models on sqa3d em b leo on various benchmarks em leo is a widely used generalist 3d llm capable of 3d captioning qa dialogue task planning and navigation comparison between blind text only models and 3d llms on existing 3d qa benchmarks revealing strong linguistic shortcuts as shown in a the blind model matches or even outperforms full 3d llms on sqa3d and as b reveals this isn t limited to one benchmark across multiple 3d qa benchmarks the same pattern holds linguistic shortcuts alone can achieve competitive scores this means existing evaluations significantly overestimate genuine 3d reasoning ability masking fundamental limitations part 2 our solution what should a benchmark for genuine 3d understanding look like we argue that a reliable 3d benchmark must satisfy two criteria it must bypass language shortcuts questions should not be answerable through linguistic priors or common sense knowledge alone if a blind text only model can cheat its way to matching a 3d llm s accuracy those questions fail to test real 3d perception it must genuinely evaluate 3d spatial understanding a model that truly comprehends 3d geometry should produce consistently correct answers regardless of the observer s viewpoint because the underlying scene structure does not change based on these principles we build real 3dqa through the pipeline below construction pipeline of real 3dqa comparing 3d llms with blind text only counterparts and general llms to eliminate shortcut solvable questions viewpoint rotation a consistency test for 3d understanding if a model truly understands 3d spatial relationships it should answer correctly regardless of the observer s viewpoint because the scene geometry doesn t change we test this by rotating the observer s viewpoint 0 90 180 270 while keeping the scene and question fixed and updating the answer accordingly your browser does not support the video tag rotating the observer s viewpoint generates equivalent questions models with genuine 3d understanding should answer all of them correctly viewpoint rotation visualizer observe how the same spatial question receives different answers as the viewpoint rotates in place scene0278 direction scene0690 counting scene0389 distance scene0025 existence ️ click drag to rotate view scroll to zoom in out 0 90 180 270 situation question gt answer leo prediction part 3 key findings what did we discover are 3d llms still strong after debiasing sqa3d to real 3dqa em em_r performance comparison on sqa3d vs real 3dqa the dramatic drop reveals inflated scores on existing benchmarks takeaway all models suffer a dramatic performance drop after debiasing up to 92 decrease revealing that their high scores on sqa3d were largely driven by linguistic shortcuts not genuine 3d understanding do 3d llms still understand after viewpoint rotation for each question we rotate the viewpoint by 0 90 180 270 and check how many times the model answers correctly higher 4 4 correct proportions indicate genuine rotation invariant understanding proportion of questions answered correctly under different numbers of rotated viewpoints and overall vrs replay takeaway the proportion of questions answered correctly across all four viewpoints is near zero for every model meaning no current 3d llm can maintain its understanding under simple viewpoint changes the answer to our title question becomes clear current 3d llms do not truly understand 3d spatial relationships part 4 open directions what s next our findings point to several important open directions for the 3d vision language community toward finer grained situated 3d understanding true 3d comprehension must be situated grounded in the observer s egocentric perspective not just a scene level overview future benchmarks should evaluate understanding from specific viewpoints and positions within the scene rotation invariant model design current model architectures and training pipelines have no explicit design for rotation robustness introducing rotation aware representations data augmentation or equivariant architectures is an important step toward genuine rotation invariant 3d reasoning beyond qa grounding captioning need situated awareness too similar shortcut learning and spatial inconsistency issues exist in 3d grounding and captioning however current benchmarks in these areas lack situated egocentric descriptions building situation aware evaluation for these tasks is a promising and necessary direction ️ rethinking the role of language in 3d llms language plays a dual role it can be a shortcut that inflates performance but it also provides valuable world priors and common sense knowledge how to properly leverage linguistic knowledge while preventing it from bypassing genuine 3d reasoning remains a key open challenge problem solution demo findings future bibtex copy inproceedings ma2026real3dqa title do 3d large language models really understand 3d spatial relationships author xianzheng ma and tao sun and shuai chen and yash bhalgat and jindong gu and angel x chang and iro armeni and iro laina and songyou peng and victor adrian prisacariu booktitle the fourteenth international conference on learning representations year 2026 url https arxiv org abs 2603 23523 this website is licensed under a creative commons attribution sharealike 4 0 international license template borrowed from nerfies