Meta tags:

Headings (most frequently used words):

eureka, human, via, pen, spinning, level, reward, design, coding, large, language, models, abstract, rewards, and, policies, gallery, components, experiments, bibtex, evaluation, results, dexterous, curriculum, learning, from, feedback,

Text of the page (most frequently used words):
#eureka (49), the (48), reward (42), and (28), human (28), rewards (16), from (14), pen (14), tasks (12), #spinning (10), level (9), for (9), that (8), this (8), learning (8), functions (8), code (7), context (7), with (7), via (6), large (6), can (6), feedback (6), reflection (6), task (6), isaac (6), design (5), language (5), shown (5), generate (5), using (5), dexterity (5), environments (5), evaluation (5), our (5), first (5), environment (5), coding (4), models (4), arxiv (4), all (4), shadow (4), hand (4), pre (4), then (4), search (4), improvement (4), distinct (4), robot (4), manipulation (4), complex (4), components (4), its (3), progressively (3), more (3), enables (3), reinforcement (3), function (3), quality (3), many (3), solve (3), trained (3), policy (3), dexterous (3), curriculum (3), generates (3), correlated (3), outperform (3), addition (3), them (3), over (3), time (3), diverse (3), form (3), suite (3), new (3), improve (3), such (3), zero (3), shot (3), without (3), open (3), llms (3), jason (2), william (2), liang (2), guanzhi (2), wang (2), huang (2), osbert (2), bastani (2), dinesh (2), jayaraman (2), yuke (2), zhu (2), linxi (2), fan (2), anima (2), anandkumar (2), response (2), within (2), block (2), select (2), image (2), above (2), incorporate (2), behavior (2), how (2), humanoid (2), learned (2), wide (2), than (2), original (2), rlhf (2), initialization (2), improves (2), benefits (2), better (2), defined (2), patterns (2), cycles (2), configurations (2), fine (2), tuning (2), quickly (2), spin (2), policies (2), even (2), between (2), ones (2), combining (2), across (2), expert (2), average (2), normalized (2), high (2), robots (2), isaacgym (2), include (2), morphologies (2), hands (2), bidexterous (2), skills (2), evaluate (2), various (2), llm (2), gpt (2), free (2), changing (2), existing (2), state (2), art (2), gpu (2), accelerated (2), nvidia (2), gym (2), rapid (2), parallel (2), any (2), specific (2), readily (2), unmodified (2), source (2), evolutionary (2), demonstrate (2), performing (2), tricks (2), palm (2), also (2), each (2), resulting (2), upenn (2), website, template, borrowed, l2r, vima, nerfies, article, ma2023eureka, title, author, yecheng, year, 2023, journal, preprint, 2310, 12931, bibtex, modify, they, induce, safer, aligned, agent, example, show, teach, run, upright, handful, which, replaces, previous, automated, final, iteration, preferred, users, margin, running, gait, study, whether, starting, common, scenario, real, world, applications, advantageous, regardless, init, uniformly, both, effectively, requires, continuously, rotate, achieve, some, possible, instructing, orienting, random, target, reach, desired, sequence, adapts, successfully, row, contrast, neither, scratch, complete, single, cycle, finetuned, pretrained, reorientation, assess, novelty, computing, correlations, mostly, weakly, observe, few, cases, are, negatively, while, significantly, outperforming, harder, less, novel, produces, eventually, exceed, scale, detailed, evolutionay, consistent, written, particular, realizes, much, greater, gains, dimensional, super, results, consist, implemented, simulator, covering, set, quadruped, bipedal, quadrotor, cobot, arm, coverage, factors, ensure, depth, including, benchmark, contains, manual, require, pair, range, ranging, object, handover, rotating, cup, 180, degrees, thoroughly, embodiments, testing, ability, forms, input, experiments, after, constructs, summarizes, key, statistics, training, uses, enable, backbone, flexibly, types, targeted, modification, hyperparameter, functional, introducing, leveraging, simulation, able, batch, candidates, enabling, scalable, space, massively, raw, plausible, programs, prompt, engineering, allows, designer, producing, try, generalist, achieves, specifically, takes, description, executable, iterates, outputs, evolving, overview, main, axis, center, video, grid, perpendicular, thus, defining, similar, finger, pass, trick, train, several, other, variations, different, axes, where, xyz, component, chosen, numerous, unique, gallery, demo, visualize, best, spans, two, sourced, benchmarks, have, excelled, semantic, planners, sequential, decision, making, however, harnessing, learn, low, remains, problem, bridge, fundamental, gap, present, algorithm, powered, exploits, remarkable, generation, writing, capabilities, perform, optimization, used, acquire, prompting, templates, engineered, outperforms, experts, leading, generality, gradient, approach, incorporating, inputs, safety, generated, model, updating, finally, setting, simulated, capable, adeptly, manipulating, circles, speed, abstract, pdf, corresponding, authors, jasonyma, seas, edu, jimfan, gmail, com, equal, advising, austin, caltech, jim,

Text of the page (random words):
eureka human level reward design via coding large language models eureka human level reward design via coding large language models jason ma 1 2 william liang 2 guanzhi wang 1 3 de an huang 1 osbert bastani 2 dinesh jayaraman 2 yuke zhu 1 4 linxi jim fan 1 anima anandkumar 1 3 1 nvidia 2 upenn 3 caltech 4 ut austin equal advising corresponding authors jasonyma seas upenn edu dr jimfan ai gmail com arxiv pdf code abstract large language models llms have excelled as high level semantic planners for sequential decision making tasks however harnessing them to learn complex low level manipulation tasks such as dexterous pen spinning remains an open problem we bridge this fundamental gap and present eureka a human level reward design algorithm powered by llms eureka exploits the remarkable zero shot generation code writing and in context improvement capabilities of state of the art llms such as gpt 4 to perform evolutionary optimization over reward code the resulting rewards can then be used to acquire complex skills via reinforcement learning without any task specific prompting or pre defined reward templates eureka generates reward functions that outperform expert human engineered rewards in a diverse suite of 29 open source rl environments that include 10 distinct robot morphologies eureka outperforms human experts on 83 of the tasks leading to an average normalized improvement of 52 the generality of eureka also enables a new gradient free in context learning approach to reinforcement learning from human feedback rlhf readily incorporating human inputs to improve the quality and the safety of the generated rewards without model updating finally using eureka rewards in a curriculum learning setting we demonstrate for the first time a simulated shadow hand capable of performing pen spinning tricks adeptly manipulating a pen in circles at rapid speed eureka rewards and policies in this demo we visualize the unmodified best eureka reward for each environment and the policy trained using this reward our environment suite spans 10 robots and 29 distinct tasks across two open sourced benchmarks isaac gym isaac and bidexterous manipulation dexterity isaac dexterity select an image above eureka response shown within code block eureka pen spinning gallery combining eureka with curriculum learning we demonstrate for the first time a shadow hand performing various pen spinning tricks our main pen spinning axis center video in the grid is perpendicular to the palm of the hand thus defining the spin as parallel to the palm similar to the finger pass trick in addition we also train several other variations with different axes where each xyz component is chosen from 1 0 1 resulting in numerous unique patterns eureka components overview eureka achieves human level reward design by in context evolving reward functions more specifically eureka first takes unmodified environment source code and language task description as context to zero shot generate executable reward functions from a coding llm then it iterates between evolutionary reward search gpu accelerated reward evaluation and reward reflection to progressively improve its reward outputs environment as context by using the raw environment code as context eureka can zero shot generate plausible reward programs without any task specific prompt engineering this allows eureka to be a generalist reward designer readily producing reward functions on first try for all our environments rapid reward evaluation via massively parallel rl leveraging state of the art gpu accelerated simulation in nvidia isaac gym eureka is able to quickly evaluate the quality of a large batch of reward candidates enabling scalable search in the reward function space eureka reward reflection after reward evaluation eureka constructs reward reflection that summarizes the key statistics of the rl training then eureka uses this reward reflection to enable the backbone llm gpt 4 to flexibly improve the reward functions with many distinct types of free form targeted modification such as 1 changing the hyperparameter of existing reward components 2 changing the functional form of existing reward components and 3 introducing new reward components experiments we thoroughly evaluate eureka on a diverse suite of robot embodiments and tasks testing its ability to generate reward functions solve new tasks and incorporate various forms of human input our environments consist of 10 distinct robots and 29 tasks implemented using the isaacgym simulator first we include 9 original environments from isaacgym isaac covering a diverse set of robot morphologies from quadruped bipedal quadrotor cobot arm to dexterous hands in addition to coverage over robot form factors we ensure depth in our evaluation by including all 20 tasks from the bidexterous manipulation dexterity benchmark dexterity contains 20 complex bi manual tasks that require a pair of shadow hands to solve a wide range of complex manipulation skills ranging from object handover to rotating a cup by 180 degrees evaluation results eureka can generate super human level reward functions across 29 tasks eureka rewards outperform expert human written ones on 83 of them with an average normalized improvement of 52 in particular eureka realizes much greater gains on high dimensional dexterity environments eureka evolutionay reward search enables consistent reward improvement over time eureka progressively produces better rewards that eventually exceed human level by combining large scale reward search with detailed reward reflection feedback eureka generates novel rewards we assess the novelty of eureka rewards by computing the correlations between eureka and human rewards on all isaac tasks as shown eureka mostly generates weakly correlated reward functions that outperform the human ones in addition we observe that the harder the task is the less correlated the eureka rewards in a few cases eureka rewards are even negatively correlated with human rewards while significantly outperforming them dexterous pen spinning via curriculum learning pretrained pen reorientation finetuned pen spinning the pen spinning task requires a shadow hand to continuously rotate a pen to achieve some pre defined spinning patterns for as many cycles as possible we solve this task by 1 instructing eureka to generate a reward function for re orienting the pen to random target configurations and then 2 fine tuning this pre trained policy using the eureka reward to reach the desired sequence of pen spinning configurations as shown eureka fine tuning quickly adapts the policy successfully spin the pen for many cycles in a row in contrast neither pre trained or learning from scratch policies can complete even a single cycle eureka from human feedback eureka effectively improves and benefits from human reward initialization we study whether starting with a human reward function initialization a common scenario in real world rl applications is advantageous for eureka as shown regardless of the quality of the human rewards eureka improves and benefits from human rewards as eureka human init is uniformly better than both eureka and human on all tasks eureka enables in context reinforcement learning from human feedback rlhf eureka can incorporate human feedback to modify its rewards so that they progressively induce safer and more human aligned agent behavior in this example we show how eureka can teach a humanoid how to run upright from a handful of human feedback which replaces the previous automated reward reflection the final learned behavior iteration 5 is more preferred by human users by a wide margin than the original eureka learned humanoid running gait select an image above eureka response shown within code block bibtex article ma2023eureka title eureka human level reward design via coding large language models author yecheng jason ma and william liang and guanzhi wang and de an huang and osbert bastani and dinesh jayaraman and yuke zhu and linxi fan and anima anandkumar year 2023 journal arxiv preprint arxiv arxiv 2310 12931 website template borrowed from nerfies vima and l2r