Meta tags:
description=Beacon3D: A Benchmark for 3D Vision-Language Understanding;
keywords=3D Vision-Language, 3D Scene Understanding, Benchmark, Multi-modal Model;

Headings (most frequently used words):

3d, of, the, existing, vl, benchmarks, beacon3d, unveiling, mist, over, vision, language, understanding, object, centric, evaluation, with, chain, analysis, overview, summary, limitations, highlights, benchmark, from, to, data, visualizer, results, and, findings, bibtex, cvpr, 2025,

Text of the page (most frequently used words):
and (40), the (33), grounding (32), data (21), object (19), metrics (17), chain (15), #centric (13), evaluation (13), beacon3d (12), language (11), model (11), analysis (9), chains (9), test (9), coherence (8), that (7), models (7), for (7), types (7), shows (6), performance (6), tasks (6), llms (5), capability (5), fine (5), gqa (5), two (5), case (5), per (5), quality (5), understanding (4), huang (4), zhu (4), than (4), more (4), scene (4), coarse (4), texts (4), left (4), right (4), question (4), task (4), existing (4), robustness (4), visual (4), high (4), benchmarks (4), flawed (4), from (3), this (3), over (3), vision (3), with (3), our (3), distribution (3), four (3), across (3), grained (3), fails (3), highlights (3), broken (3), denotes (3), are (3), scannet (3), indicating (3), three (3), ignorance (3), such (3), unveiling (2), mist (2), jiangyong (2), jia (2), baoxiong (2), wang (2), yan (2), ziyu (2), linghu (2), xiongkun (2), qing (2), song (2), chun (2), siyuan (2), cvpr (2), 2025 (2), find (2), indicates (2), into (2), evaluating (2), visualize (2), observe (2), skills (2), shortcut (2), behavior (2), significant (2), current (2), lack (2), objects (2), susceptible (2), variations (2), results (2), answer (2), multiscan (2), 3rscan (2), referential (2), target (2), each (2), appearance (2), identify (2), type (2), answers (2), but (2), which (2), different (2), design (2), diverse (2), cases (2), rephrasing (2), blind (2), sqa3d (2), studies (2), pitfalls (2), yields (2), accuracy (2), pitfall (2), weak (2), insufficient (2), human (2), study (2), flaws (2), row (2), ambiguous (2), benchmark (2), knowledge (2), university (2), template, borrowed, nerfies, website, licensed, under, creative, commons, attribution, sharealike, international, license, inproceedings, huang2025unveiling, title, author, booktitle, proceedings, ieee, cvf, conference, computer, pattern, recognition, year, you, work, helpful, please, consider, citing, bibtex, incorporating, weakens, does, not, fundamentally, enhance, suggests, main, bottleneck, lies, perception, alignment, rather, modeling, reasoning, strength, therefore, advancing, may, rely, stronger, foundation, leveraging, insights, challenging, difficulty, pronounced, when, crucial, solid, need, improve, better, limited, proportion, good, measure, both, hovering, around, reveals, substantial, gap, between, frequent, obj, elicit, drop, compared, suggesting, comprehensive, findings, text, scene_00106_05, 634b2183, scene0616_00, scene0050_00, select, then, click, image, visualizer, links, via, shared, queries, specific, aspect, forming, enables, correctly, ground, queried, content, can, recognize, showing, organize, following, scheme, helps, assess, granularities, boundary, example, contrast, previous, average, adopt, require, make, correct, predictions, all, effect, present, pilot, show, vulnerability, moderate, shifts, respectively, finetuning, unexpectedly, deficiency, simple, averaging, individual, pairs, vulnerable, like, falling, short, capturing, true, establish, detailed, annotation, guidelines, ensuring, precise, natural, address, prior, top, bottom, notable, including, questions, incomplete, could, undermine, reliability, isolation, limitations, summary, built, real, scenes, meticulously, selected, includes, 800, illustration, novel, answering, features, framework, adopts, ensure, utilizes, also, various, class, app, geometry, geo, spatial, spa, existence, exi, overview, code, demo, video, arxiv, equal, contribution, tsinghua, peking, beijing, institute, general, artificial, intelligence, bigai, arnold, pq3d, sceneverse, msr3d, leo, research,

Text of the page (random words):
beacon3d more research leo msr3d sceneverse pq3d arnold unveiling the mist over 3d vision language understanding object centric evaluation with chain of analysis cvpr 2025 jiangyong huang 1 2 baoxiong jia 1 yan wang 1 ziyu zhu 1 3 xiongkun linghu 1 qing li 1 song chun zhu 1 2 3 siyuan huang 1 1 beijing institute for general artificial intelligence bigai 2 peking university 3 tsinghua university indicates equal contribution arxiv video demo code data overview an illustration of beacon3d a novel benchmark for 3d grounding and question answering qa tasks beacon3d features an object centric evaluation framework with grounding chains g chains and grounding qa chains gqa chains for each object the evaluation adopts object centric metrics to ensure robustness and utilizes chain of analysis for studies in task coherence we also design various knowledge types such as class appearance app geometry geo spatial spa and existence exi beacon3d is built on 30 high quality real 3d scenes meticulously selected from scannet 3rscan and multiscan the object centric evaluation includes more than 800 objects and shows a diverse distribution of knowledge types in grounding and qa tasks summary limitations of existing 3d vl benchmarks ️ flawed test data insufficient evaluation metrics isolation of grounding and qa tasks highlights of the beacon3d benchmark ️ high quality test data object centric evaluation metrics grounding chain and grounding qa chain from existing 3d vl benchmarks to beacon3d ️ flawed test data we observe notable data flaws including ambiguous referential texts in the grounding task ambiguous questions and incomplete answers in the qa task such flawed test data could undermine the reliability of evaluation results flawed test data in existing 3d vl benchmarks the top row shows grounding data and the bottom row shows qa data ️ beacon3d high quality test data we establish detailed annotation guidelines ensuring precise and natural language to address prior data flaws the human study across different 3d vl benchmarks highlights the quality of beacon3d test data human study on the quality of test data the left shows grounding data and the right shows qa data insufficient evaluation metrics we find that simple metrics such as averaging accuracy over individual qa pairs are vulnerable to pitfalls like visual ignorance and weak language robustness falling short in capturing true model capability model pitfall visual ignorance model pitfall weak language robustness we present two pilot studies to show the vulnerability of existing evaluation metrics to two model pitfalls visual ignorance finetuning blind llms on sqa3d yields unexpectedly high accuracy indicating the deficiency in evaluating the visual capability of 3d vl models language robustness rephrasing language yields moderate and significant performance shifts in grounding and qa tasks respectively indicating that current 3d vl models are susceptible to language variations performance of blind llms on sqa3d denotes 3d vl model effect of rephrasing language on grounding left and qa right tasks beacon3d object centric evaluation metrics in contrast to previous per case average metrics we design three diverse test cases per object and adopt object centric metrics which require the model to make correct predictions in all three cases data example three grounding texts per object case centric metrics vs object centric metrics beacon3d grounding chain we organize the grounding data into grounding chains following a coarse to fine scheme which helps assess performance coherence across different granularities and identify the boundary of grounding capability beacon3d grounding qa chain beacon3d links qa data to grounding data via shared referential texts of the target object each question queries a specific aspect e g appearance of the object forming a grounding qa chain that enables analysis of grounding qa coherence we identify two types of broken coherence type 1 model fails to answer the queried content but can recognize that in grounding task showing a lack of qa skills type 2 model correctly answers the question but fails to ground the target object indicating shortcut behavior in qa data visualizer select a scene and then click an image to visualize scene and object centric data scannet scene0050_00 scannet scene0616_00 3rscan 634b2183 multiscan scene_00106_05 scene object grounding text question answer results and findings metrics object centric metrics elicit a significant performance drop compared to per case metrics suggesting that current 3d vl models lack a comprehensive understanding of objects and are susceptible to language variations model performance in grounding left and qa right tasks case denotes per case metrics and obj denotes object centric metrics chain analysis gqa chain we visualize four types of gqa chains and observe a limited proportion of good grounding qa coherence r 1 and r 2 measure the two types of broken coherence both hovering around 50 this reveals a substantial gap between the skills of grounding and qa and frequent shortcut behavior in qa gqa chain analysis distribution of four types of gqa chains left and two metrics for evaluating broken grounding qa coherence right chain analysis g chain evaluation across coarse to fine g chains shows that fine grained grounding is more challenging than coarse the difficulty is pronounced when the model fails on coarse texts as fine grained grounding is crucial to solid qa performance our chain analysis highlights the need to improve 3d vl models in fine grained grounding for better grounding qa coherence g chain analysis distribution of four types of g chains model insights our evaluation indicates that incorporating llms into 3d vl models weakens grounding capability and does not fundamentally enhance qa capability this suggests the main bottleneck lies in 3d perception and vl alignment rather than language modeling or reasoning llms strength therefore advancing 3d vl models may rely more on stronger foundation models for 3d scene understanding than on leveraging llms bibtex if you find our work helpful please consider citing us inproceedings huang2025unveiling title unveiling the mist over 3d vision language understanding object centric evaluation with chain of analysis author huang jiangyong and jia baoxiong and wang yan and zhu ziyu and linghu xiongkun and li qing and zhu song chun and huang siyuan booktitle proceedings of the ieee cvf conference on computer vision and pattern recognition cvpr year 2025 this website is licensed under a creative commons attribution sharealike 4 0 international license template borrowed from nerfies