Meta tags:
description=Promptable Behaviors: Personalizing Multi-Objective Rewards from Human Preferences;

Headings (most frequently used words):

from, human, reward, preferences, scenario, weight, prediction, promptable, behaviors, we, framework, diverse, the, agent, energy, conservation, preference, feedback, trajectory, comparison, personalizing, multi, objective, rewards, how, can, effectively, customize, robot, for, users, propose, novel, personalization, that, deals, with, without, re, training, abstract, in, realistic, scenarios, results, full, demonstrations, bibtex, our, method, achieves, high, success, rates, while, efficiently, optimizing, behavior, conflicting, objectives, affect, each, other, predicting, weights, new, house, demonstration, pairwise, group, quiet, operation, language, instruction,

Text of the page (most frequently used words):
the (76), and (27), reward (26), human (21), #preferences (18), weight (16), with (14), for (13), house (12), #behaviors (12), trajectory (11), agent (11), that (10), objective (10), prediction (9), user (9), object (9), weights (9), methods (9), each (9), method (9), from (8), while (8), scenarios (8), table (8), promptable (8), scenario (7), preference (7), find (7), robot (7), three (7), objectives (7), diverse (7), this (6), objectnav (6), across (6), language (5), feedback (5), without (5), other (5), policy (5), safety (5), conflicting (5), vector (5), comparison (4), wants (4), framework (4), demonstrations (4), evaluate (4), trajectories (4), predicted (4), different (4), achieves (4), highest (4), rate (4), comparisons (4), instructions (4), procthor (4), distinct (4), performance (4), exploration (4), using (4), all (4), prioritized (4), proposed (4), our (4), navigation (4), which (3), quiet (3), operation (3), energy (3), conservation (3), group (3), new (3), any (3), also (3), generated (3), results (3), show (3), users (3), their (3), shows (3), figure (3), rewards (3), when (3), success (3), row (3), embclip (3), effectively (3), various (3), customizing (3), training (3), novel (3), personalization (3), multi (3), institute (3), website (2), midnight (2), disturbing (2), child (2), noise (2), not (2), objects (2), specific (2), inspecting (2), just (2), but (2), has (2), quickly (2), urgent (2), five (2), full (2), participants (2), trained (2), compare (2), two (2), win (2), significantly (2), outperforming (2), high (2), between (2), have (2), options (2), demonstrating (2), its (2), average (2), increases (2), decreases (2), mean (2), variance (2), calculated (2), times (2), much (2), configurations (2), single (2), sub (2), during (2), than (2), path (2), efficiency (2), personalized (2), goal (2), flee (2), environments (2), aligns (2), tasks (2), robothor (2), morl (2), inferring (2), provide (2), introduce (2), leveraging (2), types (2), propose (2), deals (2), use (2), personalizing (2), reinforcement (2), learning (2), target (2), robotic (2), technology (2), based, licensed, under, creative, commons, attribution, sharealike, international, license, nerfies, template, bibtex, instruction, pairwise, demonstration, sleeping, loud, after, rearranging, does, remember, where, certain, were, placed, areas, confirm, arrangement, post, rearrangement, moved, furniture, located, layout, video, home, check, appliance, away, limited, battery, life, getting, late, important, meeting, needs, conduct, real, experiments, given, follows, evaluation, matching, perform, evaluations, asking, especially, per, indicates, closely, align, intended, predict, optimal, measure, cosine, similarity, sim, designed, experts, calculate, generalized, gini, index, ggi, measures, peakedness, now, pipeline, mentioned, above, describe, labeling, providing, quantitative, own, advantage, predicting, prioritize, more, normalize, observe, trade, offs, increase, affect, visualized, final, location, illustrated, star, visualizations, validation, set, six, prioritization, uniform, prioritizing, are, accumulated, episode, averaged, episodes, then, normalized, colored, cells, indicate, values, column, instance, higher, lowest, among, additionally, spl, respectively, implies, maintains, general, satisfying, underlying, prioritizations, fleenav, simulator, evaluated, ensure, satisfactory, both, prompts, adjusting, infers, ai2, thor, rates, efficiently, optimizing, behavior, become, through, adjustments, inference, fine, tuning, simplifies, low, dimensional, variety, specifically, interaction, conditioned, multiple, key, idea, backbone, model, take, modular, approach, imagine, navigating, asked, who, fell, asleep, required, explore, thoroughly, order, collide, avoid, making, unnecessary, contrast, hurry, expects, rather, avoiding, collisions, these, contrasting, highlight, need, policies, adapt, however, conventional, approaches, shortcomings, dealing, since, unique, realistic, aligned, underexplored, challenge, field, embodied, paper, present, facilitates, efficient, agents, complex, train, adaptable, broad, spectrum, infer, interactions, ability, prompt, satisfy, abstract, how, can, customize, code, arxiv, korea, advanced, science, massachusetts, allen, artifical, intelligence, kiana, ehsani, ani, kembhavi, kimin, lee, chanwoo, park, luca, weihs, minyoung, hwang,

Text of the page (random words):
promptable behaviors promptable behaviors personalizing multi objective rewards from human preferences minyoung hwang 1 luca weihs 1 chanwoo park 2 kimin lee 3 ani kembhavi 1 kiana ehsani 1 1 allen institute for artifical intelligence 2 massachusetts institute of technology 3 korea advanced institute of science and technology arxiv code how can we effectively customize a robot for human users we propose promptable behaviors a novel personalization framework that deals with diverse preferences without re training the agent abstract customizing robotic behaviors to be aligned with diverse human preferences is an underexplored challenge in the field of embodied ai in this paper we present promptable behaviors a novel framework that facilitates efficient personalization of robotic agents to diverse human preferences in complex environments we use multi objective reinforcement learning to train a single policy adaptable to a broad spectrum of preferences we introduce three distinct methods to infer human preferences by leveraging different types of interactions 1 human demonstrations 2 preference feedback on trajectory comparisons and 3 language instructions we evaluate the proposed method in personalized object goal navigation and flee navigation tasks in procthor and robothor demonstrating the ability to prompt agent behaviors to satisfy human preferences in various scenarios diverse human preferences in realistic scenarios imagine a robot navigating in a house at midnight asked to find an object without disturbing a child who just fell asleep the robot is required to explore the house thoroughly in order to find the target object but not collide with any objects to avoid making unnecessary noise in contrast to this quiet operation scenario in the urgent scenario a user is in a hurry and expects a robot to find the target object quickly rather than avoiding collisions these contrasting scenarios highlight the need for customizing robot policies to adapt to diverse and specific human preferences however conventional approaches have shortcomings in dealing with diverse preferences since the agent has to be re trained for each unique human preference promptable behaviors we propose promptable behaviors a novel personalization framework that deals with diverse human preferences without re training the agent the key idea of our method is to use multi objective reinforcement learning morl as the backbone of personalizing a reward model we take a modular approach training a policy conditioned on a reward weight vector across multiple objectives and inferring the reward weight vector that aligns with the user s preference using morl agent behaviors become promptable through adjustments in the reward weight vector during inference without any policy fine tuning this significantly simplifies customizing robot behaviors to inferring a low dimensional reward weight vector we provide a variety of options for users to provide their preferences to the agent specifically we introduce three distinct methods of reward weight prediction leveraging different types of interaction 1 human demonstrations 2 preference feedback on trajectory comparisons and 3 language instructions results our method achieves high success rates while efficiently optimizing the agent behavior we evaluate our method on personalized object goal navigation objectnav and flee navigation fleenav in procthor and robothor environments in the ai2 thor simulator the policy is evaluated across various scenarios to ensure that it aligns with human preferences and achieves satisfactory performance in both tasks we show that the proposed method effectively prompts agent behaviors by adjusting the reward weight vector and infers reward weights from human preferences using three distinct reward weight prediction methods for instance in objectnav when house exploration is prioritized the proposed method shows the highest success rate row j in table 1 11 3 higher than embclip while prioritized embclip shows the lowest success rate row d in table 1 among all methods and reward configurations additionally our method achieves the highest spl and the path efficiency reward when path efficiency is prioritized row i in table 1 outperforming embclip by 19 3 and 56 1 respectively this implies that the proposed method effectively maintains general performance while satisfying the underlying preferences in various prioritizations table 1 performance in procthor objectnav we evaluate each method in the validation set with six different configurations of objective prioritization uniform reward weight across all objectives and prioritizing a single objective 4 times as much as other objectives sub rewards for each objective are accumulated during each episode averaged across episodes and then normalized using the mean and variance calculated across all methods colored cells indicate the highest values in each sub reward column figure 1 trajectory visualizations in each figure agent trajectory is visualized when an objective is prioritized 10 times as much as other objectives the agent s final location is illustrated as a star conflicting objectives affect each other we observe trade offs between two conflicting objectives in objectnav safety and house exploration as we increase the weight for safety the safety reward increases while the reward for its conflicting objective house exploration decreases figure 2 conflicting objectives as we prioritize safety more the average safety reward increases while the average reward of a conflicting objective house exploration decreases we normalize the rewards for each objective using the mean and variance calculated across all weights predicting reward weights from human preferences now we compare three reward weight prediction methods and show the results of promptable behaviors for the full pipeline as mentioned above the users have three distinct options to describe their preferences to the agent 1 demonstrating a trajectory 2 labeling their preferences on trajectory comparisons and 3 providing language instructions table 2 shows the quantitative performance of the three weight prediction methods each with its own advantage table 2 comparison of three weight prediction methods in procthor objectnav we predict the optimal reward weights from human demonstrations preference feedback on trajectory comparisons and language instructions we measure the cosine similarity sim between the predicted weights and the weights designed by human experts we also calculate generalized gini index ggi which measures the peakedness of the predicted weights we also perform human evaluations by asking participants to compare trajectories generated with the predicted reward weights for different scenarios results in table 3 show that group trajectory comparison especially with two trajectories per group achieves the highest win rate significantly outperforming other methods by up to 17 8 this high win rate indicates that the generated trajectories closely align with the intended scenarios table 3 human evaluation on scenario trajectory matching participants evaluate trajectories generated with the trained policy and the reward weights predicted for five scenarios in objectnav full framework demonstrations we conduct real human experiments given the five scenarios as follows urgent the user is getting late to an important meeting and needs to quickly find an object in the house energy conservation the user wants to check an appliance in the house while the user is away but the robot that has a limited battery life new home the user just moved in and wants to find which furniture or object is located while inspecting the layout of the house as a video post rearrangement after rearranging the house the user does not remember where certain objects were placed the user wants to find a specific object while also inspecting other areas to confirm the new arrangement quiet operation at midnight the user wants to find an object in the house without disturbing a sleeping child with any loud noise new house scenario reward weight prediction from human demonstration energy conservation scenario reward weight prediction from preference feedback pairwise trajectory comparison energy conservation scenario reward weight prediction from preference feedback group trajectory comparison quiet operation scenario reward weight prediction from language instruction bibtex this website is based on the nerfies website template which is licensed under a creative commons attribution sharealike 4 0 international license