Meta tags:
description=Language to Rewards for Robotic Skill Synthesis;

Headings (most frequently used words):

video, language, to, rewards, for, robotic, skill, synthesis, abstract, approach, overview, reward, translator, motion, controller, results, validation, on, real, hardware, prompts, citation, acknowledgements, paper, blogpost, code,

Text of the page (most frequently used words):
the (74), and (46), robot (27), #reward (26), #motion (15), llms (15), for (13), that (12), language (11), our (11), code (10), tasks (9), instruction (8), robotic (7), using (7), two (7), system (7), level (7), this (7), from (6), method (6), while (6), are (6), llm (6), proposed (6), low (6), control (6), real (5), manipulator (5), open (5), mpc (5), where (5), skills (5), can (5), actions (5), synthesis (4), coder (4), descriptor (4), mujoco (4), hardware (4), good (4), now (4), controller (4), through (4), baseline (4), user (4), policies (4), based (4), translator (4), step (4), interactive (4), description (4), new (4), with (4), rewards (3), skill (3), dexterous (3), quadruped (3), demonstrate (3), pushing (3), drawer (3), hand (3), further (3), directly (3), map (3), instead (3), primitive (3), functions (3), design (3), shown (3), approach (3), prompt (3), diverse (3), generated (3), achieve (3), learning (3), work (3), mjpc (3), optimized (3), planning (3), into (3), large (3), models (3), have (3), interface (3), like (2), their (2), feedback (2), template (2), wenhao (2), gileadi (2), nimrod (2), chuyuan (2), kirmani (2), sean (2), lee (2), kuang (2), huei (2), gonzalez (2), arenas (2), montse (2), lewis (2), chiang (2), hao (2), tien (2), erez (2), tom (2), hasenclever (2), leonard (2), humplik (2), jan (2), ichter (2), brian (2), xiao (2), ted (2), peng (2), zeng (2), andy (2), zhang (2), tingnan (2), heess (2), nicolas (2), sadigh (2), dorsa (2), tan (2), jie (2), tassa (2), yuval (2), xia (2), fei (2), 2023 (2), arxiv (2), points (2), perform (2), object (2), grasping (2), able (2), generate (2), apple (2), keep (2), make (2), feet (2), human (2), walk (2), they (2), examples (2), complex (2), interactions (2), instructions (2), set (2), pre (2), defined (2), below (2), achieves (2), performance (2), evaluate (2), simulated (2), each (2), results (2), specified (2), trajectory (2), optimization (2), use (2), sequence (2), sends (2), action (2), corresponding (2), its (2), next (2), empowers (2), desired (2), specific (2), trained (2), context (2), goal (2), problem (2), coding (2), during (2), stage (2), instruct (2), progress (2), leveraging (2), behaviors (2), writing (2), existing (2), primitives (2), engineered (2), other (2), behavior (2), dependent (2), underrepresented (2), training (2), corpora (2), capabilities (2), robotics (2), semantic (2), designed (2), video (2), authors, would, acknowledge, ken, caluwaerts, kristian, hartikainen, steven, bohez, carolina, parada, marc, toussaint, greater, teams, google, deepmind, contributions, website, was, borrowed, jon, barron, acknowledgements, article, yu2023language, title, author, year, journal, preprint, 2306, 08647, citation, prompts, implement, version, onto, mobile, detect, objects, image, space, vocabulary, detector, vlm, extract, associated, point, cloud, behind, mask, outlier, rejection, might, belong, background, birds, eye, view, fit, minimum, volume, rectangle, take, extremes, determine, extent, axis, sim, transfer, relevant, synthesize, validation, put, inside, release, move, away, close, stand, upright, back, you, actually, don, need, front, paws, certain, height, just, leave, moonwalk, moon, means, should, backward, swings, moving, forward, correct, your, answer, showcase, teach, multiple, rounds, compare, against, baselines, only, going, generates, plan, cap, common, commands, available, notably, higher, success, rate, task, categories, comparable, rest, showing, effectiveness, reliability, l2r, response, within, block, text, gray, systems, capability, some, resulting, motions, found, needs, function, translatorto, maximize, accumulated, there, few, possible, ways, including, reinforcement, offline, receding, horizon, model, predictive, specifically, source, implementation, simulator, plans, applies, current, timestamp, advances, updated, states, planner, initiate, cycle, frequent, robustness, uncertainties, importantly, enables, correction, build, tuning, highly, domain, requires, expert, knowledge, unsurprising, generic, datasets, cannot, explore, ability, more, concretely, decompose, stages, interpret, expand, input, natural, following, another, translate, specifying, recent, rapid, has, inspired, notable, developments, drive, oriented, dialogue, agents, these, methods, impart, modes, compositional, generalization, focus, concatenate, together, library, either, manually, learned, priori, modulate, still, remains, due, aim, develop, leverages, power, acquire, consists, key, components, built, uponpre, interacts, understands, intents, andmodulates, all, parametersψand, weightsw, takes, interactively, optimize, optimal, overview, demonstrated, exciting, acquiring, ranging, logical, reasoning, researchers, also, explored, advance, however, since, efforts, applying, largely, treated, planners, relied, flexible, representations, richness, makes, them, suitable, introduce, paradigm, harnesses, realization, utilizing, define, parameters, accomplish, variety, intermediate, effectively, bridge, gap, between, high, corrections, meanwhile, combining, time, optimizer, creation, experience, users, immediately, observe, provide, systematically, total, reliably, tackles, validated, arm, manipulation, such, non, prehensile, emerge, abstract, corl, oral, blogpost, paper,

Text of the page (random words):
language to rewards for robotic skill synthesis language to rewards for robotic skill synthesis wenhao yu nimrod gileadi chuyuan fu sean kirmani kuang huei lee montse gonzalez arenas hao tien lewis chiang tom erez leonard hasenclever jan humplik brian ichter ted xiao peng xu andy zeng tingnan zhang nicolas heess dorsa sadigh jie tan yuval tassa fei xia paper video blogpost code corl 2023 oral video abstract large language models llms have demonstrated exciting progress in acquiring diverse new capabilities through in context learning ranging from logical reasoning to code writing robotics researchers have also explored using llms to advance the capabilities of robotic control however since low level robot actions are hardware dependent and underrepresented in llm training corpora existing efforts in applying llms to robotics have largely treated llms as semantic planners or relied on human engineered control primitives to interface with the robot on the other hand reward functions are shown to be flexible representations that can be optimized for control policies to achieve diverse tasks while their semantic richness makes them suitable to be specified by llms in this work we introduce a new paradigm that harnesses this realization by utilizing llms to define reward parameters that can be optimized and accomplish variety of robotic tasks using reward as the intermediate interface generated by llms we can effectively bridge the gap between high level language instructions or corrections to low level robot actions meanwhile combining this with a real time optimizer mujoco mpc empowers an interactive behavior creation experience where users can immediately observe the results and provide feedback to the system to systematically evaluate the performance of our proposed method we designed a total of 17 tasks for a simulated quadruped robot and a dexterous manipulator robot we demonstrate that our proposed method reliably tackles 90 of the designed tasks while a baseline using primitive skills as the interface with code as policies achieves 50 of the tasks we further validated our method on a real robot arm where complex manipulation skills such as non prehensile pushing emerge through our interactive system approach overview the recent rapid progress in large language models llms has inspired notable developments in leveraging llms to drive robot behaviors from step by step planning goal oriented dialogue to robot code writing agents while these methods impart new modes of compositional generalization they focus on using language to concatenate together new behaviors from an existing library of control primitives that are either manually engineered or learned a priori on the other hand leveraging llms to directly modulate low level robot behavior still remains an open problem due to that low level robot actions are hardware dependent and underrepresented in llm training corpora in this work we aim to develop an interactive system that leverages the power of llms to acquire low level robotic skills our proposed system consists of two key components i a reward translator built uponpre trained large language models llms 10 that interacts with and understands user intents andmodulates all reward parametersψand weightsw and ii a motion controller based on mujoco mpc that takes the generated reward and interactively optimize the optimal action sequence reward translator we build the reward translator based on llms to map user interactions to reward functions corresponding to the desired robot motion as reward tuning is highly domain specific and requires expert knowledge it is unsurprising that llms trained on generic language datasets cannot directly generate a reward for a specific hardware instead we explore the in context learning ability of llms to achieve this goal more concretely we decompose the problem of language to reward into two stages motion description and reward coding during motion description stage we design a prompt to instruct an llm to interpret and expand the user input into a natural language description of the desired robot motion following a pre defined template while during the reward coding stage we use another prompt to instruct the llm to translate the motion description into reward specifying code motion controller the motion controller needs to map the reward function generated by the reward translatorto low level robot actions that maximize the accumulated reward specified by the reward translator there are a few possible ways to achieve this including using reinforcement learning rl offline trajectory optimization or as in this work receding horizon trajectory optimization i e model predictive control mpc specifically we use an open source implementation based on the mujoco simulator mjpc at each control step mjpc plans a sequence of optimized actions and sends to the robot the robot applies the action corresponding to its current timestamp advances to the next step and sends the updated robot states to the mjpc planner to initiate the next planning cycle the frequent re planning in mpc empowers its robustness to uncertainties in the system and importantly enables interactive motion synthesis and correction results we evaluate our approach on two simulated robotic systems a quadruped robot and a dexterous robot manipulator we design a diverse set of tasks for each robot to demonstrate the capability of our proposed system some examples of the resulting robot motions can be found below prompt text in gray l2r response shown within code block we further compare our method against two baselines 1 a reward coder only baseline where an llm directly map user instructions to reward code instead of going through the motion descriptor and 2 a code as policies baseline where the llm generates a plan for the robot motion using a set of pre defined robot primitive skills instead of reward functions for the code as policies cap baseline we design the primitive skills based on common commands available to the robot as shown below our proposed approach achieves notably higher success rate for 11 17 task categories and comparable performance for the rest tasks showing the effectiveness and reliability of the proposed method we further showcase two examples where we teach the robot to perform complex tasks through multiple rounds of interactions instruction 1 make the robot stand upright on two back feet like a human instruction 2 good you actually don t need to keep the front paws at certain height just leave it to the controller instruction 3 good now make the robot do a moonwalk instruction 4 moon walk means the robot should walk backward while the feet swings as if they are moving forward correct your answer instruction 1 open the drawer instruction 2 good now put the apple inside the drawer while keep it open instruction 3 good now release the apple and move hand away instruction 4 now close the drawer validation on real hardware we implement a version of our method onto a mobile manipulator we detect objects in image space using an open vocabulary detector f vlm we extract the associated points from point cloud behind the mask and perform outlier rejection for points that might belong to the background from a birds eye view we fit a minimum volume rectangle and take the extremes to determine the extent in the z axis we demonstrate sim to real transfer on two tasks object pushing and object grasping our system is able to generate relevant reward code and the mujoco mpc is able to synthesize the pushing and grasping motion prompts quadruped motion descriptor reward coder dexterous manipulator motion descriptor reward coder real manipulator motion descriptor reward coder citation article yu2023language title language to rewards for robotic skill synthesis author yu wenhao and gileadi nimrod and fu chuyuan and kirmani sean and lee kuang huei and gonzalez arenas montse and lewis chiang hao tien and erez tom and hasenclever leonard and humplik jan and ichter brian and xiao ted and xu peng and zeng andy and zhang tingnan and heess nicolas and sadigh dorsa and tan jie and tassa yuval and xia fei year 2023 journal arxiv preprint arxiv 2306 08647 acknowledgements the authors would like to acknowledge ken caluwaerts kristian hartikainen steven bohez carolina parada marc toussaint and the greater teams at google deepmind for their feedback and contributions the website template was borrowed from jon barron