Meta tags:
description=Robot Utility Models are trained on a diverse set of environments and objects, and then can be deployed in novel environments with novel objects without any further data or training.;
keywords=Home robotics, Home robots, Robots in home, Robot Utility Models, RUMs, Open source robot learning;

Headings (most frequently used words):

robot, in, utility, models, code, general, policies, for, zero, shot, deployment, new, environments, rums, the, videos, hardware, dataset, paper, action, automatically, retrying, upon, failure, stick, v2, gripper, iphone, mount, get, on, github,

Text of the page (most frequently used words):
robot (23), and (22), the (22), for (14), models (12), environments (11), #utility (9), new (9), data (9), rums (9), #policies (8), training (8), with (8), zero (7), shot (7), paper (6), dataset (6), code (5), our (5), tasks (5), opening (5), hello (5), that (5), mahi (4), general (4), deployment (4), gripper (4), stick (4), stretch (4), can (4), shafiullah (3), haritheja (3), etukuru (3), github (3), arxiv (3), read (3), videos (3), action (3), xarm (3), retrying (3), environment (3), licensed (2), contact (2), repo (2), author (2), norihito (2), naka (2), zijin (2), seungjae (2), lee (2), julian (2), mehu (2), aaron (2), edsinger (2), chris (2), paxton (2), soumith (2), chintala (2), lerrel (2), pinto (2), nur (2), muhammad (2), rgb (2), each (2), average (2), demonstrations (2), download (2), files (2), mount (2), set (2), improve (2), more (2), hardware (2), into (2), determines (2), task (2), has (2), mllm (2), automatically (2), pick (2), lab (2), attempted (2), also (2), have (2), shown (2), world (2), manipulation (2), generalize (2), such (2), policy (2), picking (2), objects (2), success (2), rate (2), unseen (2), novel (2), dot (2), source, website, content, ans, mit, questions, documentation, get, article, etukuru2024robot, title, journal, preprint, 2409, 05865, year, 2024, citation, bibtex, pdf, actions, 473, diversity, visualizer, release, containing, 1000, across, contains, fps, well, full, annotations, pose, angle, normalized, between, 5509, trajectories, 180, made, possible, add, onto, your, own, arm, printed, dynamixel, allowing, identical, pov, thus, facilitating, seemless, transfer, robots, iphone, creation, bill, materials, redesigned, addressing, some, previous, limitations, designed, user, experience, becoming, ergonomic, capable, stronger, than, before, feed, summary, observations, multimodal, llm, which, whether, not, hand, succeeded, failed, resets, initial, state, retries, upon, failure, autoshuffle, shuffle, text5, text4, text3, text2, text1, tissue, reorientation, bag, door, drawer, home, they, few, see, sample, rollouts, below, nutshell, particularly, those, trained, large, amounts, recently, plethora, real, navigation, capabilities, several, independent, efforts, given, sufficient, demonstrated, variations, however, needing, finetune, every, stands, stark, contrast, language, vision, deployed, open, problems, this, work, present, framework, deploying, directly, without, any, finetuning, create, efficiently, develop, tools, quickly, collect, mobile, integrate, multi, modal, imitation, learning, deploy, device, cheap, commodity, external, verifier, train, five, cabinet, doors, drawers, napkins, bags, reorienting, fallen, system, achieves, interacting, moreover, succeed, different, camera, ups, further, fine, tuning, primary, among, lessons, are, importance, over, algorithm, class, guidance, about, scaling, necessity, diverse, yet, high, quality, recipe, introspection, performance, individual, expand, abstract, corresponding, nyu, edu, denotes, equal, contribution, additional, start, using,

Text of the page (random words):
robot utility models general policies for zero shot deployment in new environments rums videos hardware dataset paper code contact us start using our rums robot utility models general policies for zero shot deployment in new environments 90 success rate in novel environments with 0 additional data or training haritheja etukuru norihito naka zijin hu seungjae lee julian mehu aaron edsinger chris paxton soumith chintala lerrel pinto nur muhammad mahi shafiullah corresponding author mahi at cs dot nyu dot edu denotes equal contribution rums in action read the paper github repo expand abstract robot models particularly those trained with large amounts of data have recently shown a plethora of real world manipulation and navigation capabilities several independent efforts have shown that given sufficient training data in an environment robot policies can generalize to demonstrated variations in that environment however needing to finetune robot models to every new environment stands in stark contrast to models in language or vision that can be deployed zero shot for open world problems in this work we present robot utility models rums a framework for training and deploying zero shot robot policies that can directly generalize to new environments without any finetuning to create rums efficiently we develop new tools to quickly collect data for mobile manipulation tasks integrate such data into a policy with multi modal imitation learning and deploy policies on device on hello robot stretch a cheap commodity robot with an external mllm verifier for retrying we train five such utility models for opening cabinet doors opening drawers picking up napkins picking up paper bags and reorienting fallen objects our system on average achieves 90 success rate in unseen novel environments interacting with unseen objects moreover the utility models can also succeed in different robot and camera set ups with no further data training or fine tuning primary among our lessons are the importance of training data over training algorithm and policy class guidance about data scaling necessity for diverse yet high quality demonstrations and a recipe for robot introspection and retrying to improve performance on individual environments rums in a nutshell videos rums in action our rums attempted 5 tasks each in 5 environments on a hello robot stretch they also attempted a few tasks on an xarm see sample rollouts below hello robot stretch home hello robot stretch lab xarm 7 lab drawer opening door opening bag pick up reorientation tissue pick up text1 text2 text3 text4 text5 shuffle autoshuffle rums automatically retrying upon failure we feed in a summary of robot observations into a multimodal llm which determines whether or not the task at hand has succeeded if the mllm determines that the task has failed the robot automatically resets to a new initial state and retries hardware the stick v2 we ve redesigned the stick addressing some of the previous limitations stick v2 is designed to improve on user experience becoming more ergonomic more capable and stronger than before download the 3d files bill of materials dataset creation code robot gripper iphone mount hello robot xarm 7 we ve made it possible to add the stick gripper onto your own robot arm with a 3d printed mount and dynamixel set allowing for an identical pov thus facilitating seemless zero shot transfer of policies to new robots download the 3d files dataset 5 tasks 180 environments 5509 trajectories we release the training dataset for our robot utility models containing 5 tasks each with on average 1000 training demonstrations across 36 environments the dataset contains rgb videos at 30 fps as well as full action annotations for 6d pose of the gripper and the gripper s opening angle normalized between 0 1 data diversity visualizer rgb actions dataset 473 mb paper robot utility models general policies for zero shot deployment in new environments read the paper arxiv read the paper pdf citation bibtex article etukuru2024robot title robot utility models general policies for zero shot deployment in new environments author haritheja etukuru and norihito naka and zijin hu and seungjae lee and julian mehu and aaron edsinger and chris paxton and soumith chintala and lerrel pinto and nur muhammad mahi shafiullah journal arxiv preprint arxiv 2409 05865 year 2024 code get the code on github github repo documentation questions contact us robot utility models by haritheja etukuru and mahi shafiullah the source code is licensed mit the website content is licensed cc ans 4 0