Meta tags:
description=OK-Robot is an open framework for a modular system for zero-shot, language conditioned pick-and-drop tasks in arbitrary homes. OK-Robot accomplishes this with a modular design, using 3D VoxelMap for open-vocabulary navigation, AnyGrasp and LangSam for open-vocabulary grasping, and a dropping primitive for placing.;
keywords=Home robotics, Home robots, Robots in home, OK-Robot, Open-knowledge robots, Open source robot learning;

Headings (most frequently used words):

ok, robot, in, code, open, for, the, videos, analysis, paper, an, modular, framework, zero, shot, language, conditioned, pick, and, drop, tasks, arbitrary, homes, action, understanding, performance, of, what, really, matters, integrating, knowledge, models, robotics, get, on, github,

Text of the page (most frequently used words):
the (26), robot (18), and (16), open (8), for (8), paper (7), home (7), #models (6), robotics (6), from (5), knowledge (5), #performance (5), code (4), new (4), environments (4), homes (4), that (4), pick (4), drop (4), tasks (4), language (4), systems (4), mahi (3), shafiullah (3), discord (3), github (3), what (3), really (3), matters (3), integrating (3), arxiv (3), read (3), analysis (3), can (3), object (3), manipulation (3), vision (3), navigation (3), grasping (3), licensed (2), website (2), questions (2), ask (2), repo (2), liu (2), peiqi (2), orru (2), yaswanth (2), paxton (2), chris (2), nur (2), muhammad (2), pinto (2), lerrel (2), understanding (2), failure (2), presented (2), figure (2), above (2), most (2), while (2), zero (2), shot (2), completely (2), when (2), success (2), rate (2), are (2), bag (2), red (2), videos (2), objects (2), based (2), mobile (2), these (2), framework (2), combining (2), vlms (2), primitives (2), with (2), equal (2), source, content, template, taken, dobb, ans, mit, documentation, get, article, liu2024okrobot, title, author, journal, preprint, 2401, 12202, year, 2024, citation, bibtex, pdf, section, over, modes, discuss, frequent, cases, our, method, show, generalization, probe, better, understand, how, succeeds, fails, find, novel, closer, look, also, notice, long, tail, causes, which, see, leading, three, cause, failures, failing, retrieve, right, navigate, semantic, memory, getting, difficult, pose, module, hardware, difficulties, shuffle, orange, package, dust, bin, purple, shampoo, white, rack, blue, moisturizer, target, navaratna, oil, table, detergent, pack, shopping, all, york, city, attempted, 171, here, sample, trials, each, showing, action, remarkable, progress, has, been, made, recent, years, fields, now, have, capable, recognizing, queries, effectively, control, handle, wide, range, despite, advancements, general, purpose, applications, still, lag, behind, even, though, they, rely, fundamental, capabilities, recognition, this, adopt, first, approach, develop, called, detection, movement, offers, integrated, solution, operations, without, requiring, any, training, evaluate, its, run, real, world, results, demonstrate, achieves, ended, representing, state, art, vocabulary, ovmm, nearly, prior, work, cleaner, uncluttered, increases, however, important, insight, gained, critical, role, nuanced, details, like, robotic, modules, expand, abstract, server, contributions, advising, jay, vakil, modular, conditioned, arbitrary, contact,

Text of the page (random words):
ok robot what really matters in integrating open knowledge models for robotics ok robot videos analysis paper code contact us questions ask on discord ok robot an open modular framework for zero shot language conditioned pick and drop tasks in arbitrary homes peiqi liu yaswanth orru jay vakil chris paxton nur muhammad mahi shafiullah lerrel pinto equal contributions equal advising ok robot in homes read the paper github repo discord server expand abstract remarkable progress has been made in recent years in the fields of vision language and robotics we now have vision models capable of recognizing objects based on language queries navigation systems that can effectively control mobile systems and grasping models that can handle a wide range of objects despite these advancements general purpose applications of robotics still lag behind even though they rely on these fundamental capabilities of recognition navigation and grasping in this paper we adopt a systems first approach to develop a new open knowledge based robotics framework called ok robot by combining vision language models vlms for object detection navigation primitives for movement and grasping primitives for object manipulation ok robot offers a integrated solution for pick and drop operations without requiring any training to evaluate its performance we run ok robot in 10 real world home environments the results demonstrate that ok robot achieves a 58 5 success rate in open ended pick and drop tasks representing a new state of the art in open vocabulary mobile manipulation ovmm with nearly 1 8x the performance of prior work on cleaner uncluttered environments ok robot s performance increases to 82 however the most important insight gained from ok robot is the critical role of nuanced details when combining open knowledge systems like vlms with robotic modules videos ok robot in action in 10 home environments of new york city ok robot attempted 171 pick and drop tasks here are sample trials from 5 homes each showing 5 tasks home 2 home 6 home 7 home 8 home 10 all detergent pack to red shopping bag red navaratna oil to table blue moisturizer to target bag purple shampoo to white rack orange package to dust bin shuffle analysis understanding the performance of ok robot while our method can show zero shot generalization in completely new environments we probe ok robot to better understand when and how it succeeds and fails while we find a 58 5 success rate at completely novel homes at a closer look we also notice a long tail of failure causes which is presented in the figure above we see that the leading three cause of failures are failing to retrieve the right object to navigate to from the semantic memory 9 3 getting a difficult pose from the manipulation module 8 0 and hardware difficulties 7 5 in the understanding the performance of ok robot section of the paper we go over the analysis of the failure modes presented in the figure above and discuss the most frequent cases paper ok robot what really matters in integrating open knowledge models for robotics read the paper arxiv read the paper pdf citation bibtex article liu2024okrobot title ok robot what really matters in integrating open knowledge models for robotics author liu peiqi and orru yaswanth and paxton chris and shafiullah nur muhammad mahi and pinto lerrel journal arxiv preprint arxiv 2401 12202 year 2024 code get the code on github github repo documentation questions ask on discord ok robot by mahi shafiullah the source code is licensed mit the website content is licensed cc ans 4 0 the website template is taken from dobb e