Meta tags:
description=Reinforcement Fine-Tuning for Video-based 3D scene understanding.;
keywords=3D Scene Understanding;

Headings (most frequently used words):

3d, rft, training, on, evaluation, results, reinforcement, fine, tuning, for, video, based, scene, understanding, abstract, conceptual, comparison, framework, perception, spatial, reasoning, dynamics, analysis, objectives, of, sft, and, bibtex,

Text of the page (most frequently used words):
the (59), rft (38), and (32), sft (18), for (14), video (14), reward (14), #training (14), reasoning (13), this (11), #results (11), spatial (10), based (8), with (8), detection (8), task (8), evaluation (8), performance (8), that (7), fine (7), tuning (7), scene (7), understanding (7), model (7), iou (7), models (7), reinforcement (6), baseline (6), perception (6), policy (6), between (6), token (5), tasks (5), metrics (5), you (4), website (4), huang (4), optimization (4), from (4), verifiable (4), score (4), paradigm (4), main (4), significantly (4), outperforms (4), larger (4), grounding (4), llm (4), directly (4), code (3), which (3), prediction (3), distribution (3), landscape (3), predictions (3), output (3), cross (3), entropy (3), indirect (3), proxy (3), geometric (3), where (3), objectives (3), dynamics (3), notably (3), after (3), improvement (3), optimizes (3), while (3), rlvr (3), data (3), present (3), quantitative (3), visual (3), comparison (3), using (3), free (2), international (2), linghu (2), xiongkun (2), jiangyong (2), jia (2), baoxiong (2), siyuan (2), learning (2), 2026 (2), objective (2), shifts (2), compared (2), shift (2), discrete (2), signal (2), box (2), level (2), following (2), figure (2), illustrates (2), both (2), accuracy (2), rewards (2), efficacy (2), saturation (2), continuous (2), over (2), consistent (2), global (2), indicates (2), initial (2), recall (2), moreover (2), effective (2), refinement (2), two (2), performances (2), demonstrate (2), yields (2), improvements (2), vsi (2), bench (2), test (2), consistently (2), improves (2), optimizing (2), capabilities (2), conclusions (2), enhances (2), aware (2), like (2), via (2), gradient (2), relative (2), framework (2), supervised (2), loss (2), acts (2), leading (2), gap (2), final (2), robust (2), first (2), large (2), language (2), functions (2), more (2), means, are, borrow, just, ask, link, back, page, footer, please, remember, remove, analytics, included, header, not, want, your, source, licensed, under, creative, commons, attribution, sharealike, license, inproceedings, linghu20263d, title, author, booktitle, proceedings, conference, machine, year, find, our, helpful, feel, cite, bibtex, density, shows, toward, higher, overlap, regions, supports, core, motivation, direct, encourages, geometrically, better, beyond, imitation, supervises, tokens, producing, sharp, target, around, exact, ground, truth, sequence, only, quality, instead, uses, decoded, yielding, smoother, metric, aligned, nearby, receive, meaningful, partial, credit, landscapes, analysis, despite, minor, fluctuations, exhibit, upward, trends, confirming, curves, show, sign, 4000, steps, attribute, nature, unlike, continuously, refines, coordinates, features, text, space, coarser, feedback, leads, earlier, than, potentially, limits, granularity, rise, confirms, effectively, analyzing, components, reveals, strategic, steadily, increases, peaks, early, then, slightly, declines, transitions, tightening, boxes, reducing, false, negatives, latter, phase, dense, serves, critical, regulator, sparser, balance, precision, end, implies, stable, maximization, ablation, study, cot, shown, fig, regarding, checkpoints, compare, their, before, especially, elicits, addition, observe, continually, moderateperformance, drop, suggests, advantage, include, zero, shot, base, some, demonstrating, effectiveness, content, proposal, scanrefer, table, report, ours, scannetdetection, process, consists, stages, vlm, establish, generates, completions, problems, calculated, format, adherence, structured, optimized, maintaining, divergence, constraint, frozen, reference, warm, left, standard, relies, per, middle, identical, formats, right, utilizes, scalar, derived, through, decoding, parsing, module, ensuring, paradigms, conceptual, extensive, experiments, achieves, state, art, various, benchmarks, further, reveal, good, properties, such, valuable, insights, into, strategies, impact, hope, can, serve, promising, future, development, bridge, extend, towards, activates, multimodal, mlllms, followed, group, grpo, strictly, design, specific, provide, signals, guide, has, emerged, transformative, enhancing, llms, yet, its, potential, remains, underexplored, existing, approaches, largely, rely, misalignment, abstract, checkpoint, arxiv, equal, contribution, corresponding, authors, peking, university, beijing, institute, general, artificial, intelligence, icml, leo, beacon3d, msr3d, scenecot, research,

Text of the page (random words):
3d rft reinforcement fine tuning for video based 3d scene understanding more research scenecot msr3d beacon3d leo 3d rft reinforcement fine tuning for video based 3d scene understanding icml 2026 xiongkun linghu 1 jiangyong huang 1 2 baoxiong jia 1 siyuan huang 1 1 beijing institute for general artificial intelligence 2 peking university equal contribution corresponding authors arxiv code checkpoint abstract reinforcement learning with verifiable rewards rlvr has emerged as a transformative paradigm for enhancing the reasoning capabilities of large language models llms yet its potential in 3d scene understanding remains underexplored existing approaches largely rely on supervised fine tuning sft where the token level cross entropy loss acts as an indirect proxy for optimization leading to a misalignment between training objectives and task performances to bridge this gap we present reinforcement fine tuning for video based 3d scene understanding 3d rft the first framework to extend rlvr to video based 3d perception and reasoning 3d rft shifts the paradigm by directly optimizing the model towards evaluation metrics 3d rft first activates 3d aware multimodal large language models mlllms via sft followed by reinforcement fine tuning using group relative policy optimization grpo with strictly verifiable reward functions we design task specific reward functions directly from metrics like 3d iou and f1 score to provide more effective signals to guide model training extensive experiments demonstrate that 3d rft 4b achieves state of the art performance on various video based 3d scene understanding tasks notably 3d rft 4b significantly outperforms larger models e g vg llm 8b on 3d video detection 3d visual grounding and spatial reasoning benchmarks we further reveal good properties of 3d rft such as robust efficacy and valuable insights into training strategies and data impact we hope 3d rft can serve as a robust and promising paradigm for future development of 3d scene understanding conceptual comparison comparison between sft and 3d rft training paradigms left standard supervised fine tuning sft relies on a per token cross entropy loss which acts as an indirect proxy leading to a gap between training objectives and final evaluation metrics middle identical output formats for 3d video detection grounding and spatial reasoning tasks right 3d rft utilizes a scalar reward policy gradient derived directly from evaluation metrics e g f1 score 3d iou and accuracy through a decoding and parsing module ensuring the model directly optimizes for the final task performance 3d rft training framework the process consists of two main stages 1 sft warm up initial training of the 3d aware vlm using sft data to establish a baseline policy 2 rl training the policy model generates completions for video based problems a verifiable reward is calculated based on format reward adherence to structured output and task reward performance in 3d video detection 3d visual grounding and spatial reasoning using metrics like 3d iou and f1 score the model is optimized via policy gradient while maintaining a kl divergence constraint relative to the frozen reference model evaluation results on 3d perception quantitative results on scannetdetection in this table we present the comparison with baseline models and report the performance improvement between the sft baseline vg llm 4b and 3d rft 4b ours the main conclusions 1 3d rft 4b significantly enhances detection performance over the sft baseline 2 3d rft 4b outperforms the larger vg llm 8b quantitative results on scanrefer the content in indicates results with proposal refinement the main conclusions 1 3d rft significantly enhances performance compared to the sft baseline 2 3d rft 4b outperforms the larger vg llm 8b 3d video detection 3d visual grounding evaluation results on 3d spatial reasoning quantitative results on vsi bench we include zero shot results of base models and test results of models after sft and rft 3d rft consistently improves spatial reasoning performance notably 3d rft 4b significantly outperforms some larger models demonstrating the effectiveness of rft in optimizing spatial reasoning capabilities 3d rft consistently improves spatial reasoning performance as shown in fig 3 we present evaluation results regarding two sft checkpoints and compare their performances between before and after rft the results demonstrate that rft yields consistent improvements on vsi bench especially on ta test moreover rft on ta task elicits improvements on da task in addition we observe that continually training with sft yields a moderateperformance drop which suggests the advantage of rft main results ablation study on cot data training dynamics 3d video detection the following figure illustrates the training dynamics of 3d rft on 3d video detection the consistent rise in both evaluation f1 score a and f1 reward b confirms that rft effectively optimizes perception analyzing the reward components reveals a strategic shift while the global f1 reward b steadily increases the iou reward c peaks early and then slightly declines this indicates the policy transitions from initial geometric refinement tightening boxes to recall maximization reducing false negatives in the latter phase the dense iou reward serves as a critical regulator while the sparser f1 reward optimizes the global balance between precision and recall moreover the continuous improvement at the end implies that rlvr is a stable and effective paradigm for 3d perception tasks 3d spatial reasoning the following figure illustrates the training dynamics of 3d rft on the spatial reasoning task despite minor fluctuations both evaluation accuracy and training rewards exhibit upward trends confirming rft s efficacy notably the curves show a sign of saturation after 4000 steps we attribute this to the nature of the optimization landscape unlike perception tasks where rft continuously refines continuous geometric coordinates the reasoning task features a discrete text space with coarser feedback this leads to the earlier saturation than 3d perception tasks and potentially limits the granularity of improvement over sft 3d video detection 3d spatial reasoning analysis on training objectives of sft and rft objective landscapes of sft and rft sft supervises discrete output tokens with cross entropy producing a sharp target signal around the exact ground truth token sequence for 3d box prediction this token level signal is only an indirect proxy for geometric quality rft instead uses the decoded box iou as a verifiable reward yielding a smoother metric aligned landscape where nearby 3d predictions receive meaningful partial credit prediction distribution shift from sft to rft the density distribution of 3d iou shows that rft shifts predictions toward higher overlap regions compared with the sft baseline this supports the core motivation of 3d rft direct reward optimization encourages geometrically better predictions beyond token imitation objective landscape prediction distribution bibtex if you find our model helpful feel free to cite it inproceedings linghu20263d title 3d rft reinforcement fine tuning for video based 3d scene understanding author linghu xiongkun and huang jiangyong and jia baoxiong and huang siyuan booktitle proceedings of the international conference on machine learning year 2026 this website is licensed under a creative commons attribution sharealike 4 0 international license this means you are free to borrow the source code of this website we just ask that you link back to this page in the footer please remember to remove the analytics code included in the header of the website which you do not want on your website