Meta tags:
description=Pre-training Auto-regressive Robotic Models with 4D Representations;
keywords=Robotics, 4D Representations, Machine Learning;

Headings (most frequently used words):

4d, representations, pre, training, auto, regressive, robotic, models, with, abstract, methodology, results, bibtex, average, success, rate, by, task, category,

Text of the page (most frequently used words):
the (21), and (17), #representations (14), robotic (11), pre (9), #training (7), video (7), human (6), from (6), points (5), stage (5), data (5), auto (4), regressive (4), models (4), with (4), arm4r (4), model (4), robot (4), control (4), across (4), our (4), arxiv (3), fine (3), stages (3), time (3), this (3), between (3), these (3), that (3), niu (2), dantong (2), sharma (2), yuvan (2), xue (2), haoru (2), biamby (2), giscard (2), zhang (2), junyi (2), ziteng (2), darrell (2), trevor (2), herzig (2), roei (2), 2025 (2), tasks (2), into (2), openvla (2), results (2), are (2), pick (2), success (2), task (2), grey (2), box (2), focus (2), learning (2), egocentric (2), dataset (2), next (2), trained (2), videos (2), point (2), tracking (2), monocular (2), experiments (2), shared (2), geometric (2), structure (2), linear (2), transformation (2), state (2), have (2), robotics (2), for (2), low (2), level (2), transfer (2), your (2), browser (2), does (2), not (2), support (2), tag (2), berkeley (2), article, niu2025pre, title, author, journal, preprint, 2502, 13142, year, bibtex, evaluate, total, grouped, five, categories, against, several, recent, related, works, outperformed, all, tested, baselines, including, sota, vla, methods, presented, table, below, ours, fast, llarva, atm, overall, avg, push, buttons, place, toys, stack, destack, cube, method, average, rate, category, finally, tuned, predict, proprioceptive, states, rather, than, enable, bottom, first, two, scene, wide, representation, predicting, where, trains, large, epickitchens100, tunes, smaller, demonstrations, scenes, step, helps, transition, camera, dynamics, embodiment, gaps, tuning, top, three, methodology, also, generalizable, enough, usable, non, described, section, main, consists, object, interaction, result, solving, problem, which, involves, finding, coordinates, discrete, given, shown, benefits, downstream, hypothesize, because, foundation, massive, unlabeled, datasets, revolutionized, natural, language, computer, vision, exhibiting, remarkable, generalization, capabilities, thus, highlighting, importance, yet, efforts, struggled, achieve, similar, limited, either, need, costly, annotations, lack, effectively, physical, world, paper, introduce, leverages, learned, yield, better, pretrained, specifically, utilizing, derived, lifting, space, via, depth, estimation, maintain, enabling, efficient, show, can, efficiently, consistently, improves, performance, various, environments, configurations, abstract, scroll, more, code, icml, research, mathjax, example,

Text of the page (random words):
pre training auto regressive robotic models with 4d representations mathjax example pre training auto regressive robotic models with 4d representations dantong niu yuvan sharma haoru xue giscard biamby junyi zhang ziteng ji trevor darrell roei herzig berkeley ai research uc berkeley icml 2025 arxiv code data scroll for more your browser does not support the video tag your browser does not support the video tag abstract foundation models pre trained on massive unlabeled datasets have revolutionized natural language and computer vision exhibiting remarkable generalization capabilities thus highlighting the importance of pre training yet efforts in robotics have struggled to achieve similar success limited by either the need for costly robotic annotations or the lack of representations that effectively model the physical world in this paper we introduce arm4r an auto regressive robotic model that leverages low level 4d representations learned from human video data to yield a better pretrained robotic model specifically we focus on utilizing 3d point tracking representations from videos derived by lifting 2d representations into 3d space via monocular depth estimation across time these 4d representations maintain a shared geometric structure between the points and robot state representations up to a linear transformation enabling efficient transfer learning from human video data to low level robotic control our experiments show that arm4r can transfer efficiently from human video data to robotics and consistently improves performance on tasks across various robot environments and configurations 4d representations our 4d representations result from solving the 3d point tracking problem which involves finding the 3d coordinates of discrete points across time given a monocular video as shown by our experiments pre training on these 4d representations benefits the downstream task of robotic control we hypothesize this is because of the shared geometric structure up to a linear transformation between the 3d points and robot state representations these 4d representations are also generalizable enough to be usable with non robotic data as described in the next section our main pre training stage consists of egocentric human object interaction videos methodology arm4r is trained in three stages top grey box stages 1 and 2 the first two stages focus on learning a scene wide 4d representation by predicting 3d points across time where stage 1 pre trains on a large egocentric human dataset epickitchens100 and stage 2 fine tunes on a smaller dataset 1 2k demonstrations of robotic scenes this step helps transition from the camera dynamics and embodiment gaps between the human video pre training and the control fine tuning in the next stage bottom grey box stage 3 finally the model is fine tuned to predict robot proprioceptive states rather than 3d points to enable robotic control results we evaluate arm4r on 13 total tasks grouped into five categories against several recent related works it outperformed all tested baselines including sota vla methods π 0 and openvla the results are presented in the table below average success rate by task category method pick cube destack stack pick place toys push buttons overall avg atm 7 1 6 7 2 0 10 3 2 0 6 4 llarva 46 7 9 3 9 2 9 3 11 7 18 3 π 0 fast 40 4 15 6 23 9 14 0 10 3 21 2 openvla 71 8 53 5 33 2 18 0 11 6 37 2 ours 96 0 94 7 61 6 92 7 54 8 83 1 bibtex article niu2025pre title pre training auto regressive robotic models with 4d representations author niu dantong and sharma yuvan and xue haoru and biamby giscard and zhang junyi and ji ziteng and darrell trevor and herzig roei journal arxiv preprint arxiv 2502 13142 year 2025