Meta tags:
description=HumanEgo learns deployable robot manipulation policies from only minutes of human egocentric videos, without any robot data or internet-scale pretraining.;

Headings (most frequently used words):

humanego, results, of, test, human, video, data, the, embodiment, water, flowers, downstack, cups, serve, bread, open, generalization, zero, shot, robot, learning, from, minutes, egocentric, videos, real, world, clips, full, insights, overview, abstract, architecture, collection, and, preprocessing, dataset, bridges, gap, efficiently, efficiency, demonstrations, one, policy, many, conditions, what, drives, performance, faq, our, team, acknowledge, bibtex, adjust, table, charge, devices, unscrew, cap, door, cabinet, grab, tissue, cross,

Text of the page (most frequently used words):
the (230), and (127), data (64), human (50), that (47), for (43), #humanego (42), hand (36), robot (31), from (28), object (25), not (25), with (25), policy (25), manipulation (21), more (20), egocentric (19), what (19), are (18), same (18), task (18), video (18), one (17), this (16), you (16), can (16), visual (16), aria (16), minutes (15), icts (15), out (15), representation (15), but (15), spatial (14), world (14), tasks (13), all (13), pipeline (13), camera (13), why (13), learning (12), each (12), than (12), just (12), every (12), action (12), any (12), work (11), how (11), pose (11), ict (11), vision (11), embodiment (11), our (10), scene (10), open (10), motion (10), auxiliary (10), gap (10), interaction (10), many (9), well (9), objectives (9), only (9), real (9), free (9), hardware (9), tamp (9), success (9), videos (8), expect (8), tracking (8), less (8), dataset (8), does (8), use (8), signal (8), slam (8), quality (8), new (8), efficient (8), most (8), zero (7), shot (7), other (7), single (7), finger (7), where (7), much (7), about (7), dexterous (7), people (7), image (7), demonstrations (7), teleop (7), collection (7), project (6), source (6), precision (6), exactly (6), these (6), directly (6), built (6), off (6), has (6), paper (6), results (6), training (6), demonstrator (6), actually (6), head (6), three (6), gen (6), demonstration (6), without (6), across (6), teleoperation (6), encoder (6), rate (6), close (5), your (5), water (5), shelf (5), high (5), entity (5), dof (5), simply (5), which (5), per (5), architecture (5), genuinely (5), today (5), accuracy (5), very (5), objects (5), them (5), matter (5), first (5), also (5), performance (5), raw (5), tokens (5), better (5), fidelity (5), far (5), signals (5), provide (5), inside (5), way (5), person (5), glasses (5), gripper (5), full (5), learns (5), designed (5), model (5), preprocessing (5), whether (5), table (5), environment (5), labels (5), min (5), cross (5), generalization (5), test (5), furong (4), their (4), lab (4), glance (4), once (4), eyes (4), still (4), alone (4), specific (4), could (4), keep (4), estimation (4), coarse (4), rather (4), fine (4), right (4), everything (4), needs (4), control (4), will (4), problem (4), level (4), next (4), scale (4), nothing (4), they (4), think (4), reach (4), two (4), fixed (4), doesn (4), top (4), fundamentally (4), hands (4), often (4), look (4), arm (4), then (4), calibrated (4), devices (4), method (4), rgb (4), anything (4), third (4), even (4), result (4), poses (4), engineering (4), already (4), rich (4), yet (4), environments (4), planning (4), robust (4), skill (4), isn (4), serve (4), body (4), carefully (4), own (4), into (4), forecast (4), flow (4), matching (4), trajectory (4), time (4), supervision (4), dense (4), flowers (4), agnostic (4), conditions (4), wang (3), zhi (3), botao (3), kelin (3), lee (3), seungjae (3), gao (3), ruohan (3), huang (3), aloimonos (3), yiannis (3), bibtex (3), feedback (3), leo (3), like (3), memory (3), place (3), moving (3), principle (3), state (3), deformable (3), set (3), follow (3), models (3), error (3), encode (3), construction (3), tight (3), demand (3), fast (3), dynamic (3), regime (3), particular (3), carries (3), solve (3), precise (3), remains (3), natural (3), fully (3), should (3), yes (3), turns (3), research (3), when (3), best (3), input (3), study (3), things (3), pixels (3), geometry (3), step (3), tried (3), tracked (3), keypoints (3), frame (3), was (3), adding (3), centric (3), keypoint (3), upstream (3), downstream (3), box (3), device (3), those (3), else (3), realsense (3), meta (3), exocentric (3), before (3), move (3), policies (3), good (3), transfers (3), through (3), learned (3), find (3), comes (3), both (3), retargeting (3), captures (3), learn (3), configuration (3), higher (3), have (3), run (3), general (3), predicates (3), explicit (3), goal (3), structure (3), novel (3), generalizes (3), cost (3), relative (3), transferable (3), naturally (3), workspace (3), youtube (3), complementary (3), physical (3), lightweight (3), over (3), amount (3), matters (3), its (3), calibration (3), mps (3), grade (3), highest (3), robotics (3), clips (3), robots (3), efficiency (3), achieves (3), gallery (3), overview (3), bread (3), downstack (3), cups (3), page (2), template (2), arxiv (2), support (2), acknowledge (2), university (2), maryland (2), team (2), boundary (2), behind (2), pick (2), needle (2), precisely (2), either (2), makes (2), encouragingly (2), tied (2), trackers (2), ceiling (2), richer (2), touch (2), almost (2), system (2), because (2), continuous (2), sit (2), outside (2), reorientation (2), tool (2), hard (2), view (2), hardest (2), actively (2), thinking (2), question (2), themselves (2), need (2), evolve (2), runs (2), push (2), past (2), small (2), processing (2), harder (2), opposite (2), clearly (2), current (2), setting (2), loop (2), sees (2), here (2), requires (2), squeeze (2), appearance (2), back (2), onto (2), among (2), none (2), geometric (2), assume (2), inpainting (2), might (2), caveat (2), clean (2), drift (2), may (2), try (2), kinds (2), bounds (2), worn (2), give (2), output (2), actions (2), recover (2), quest (2), pro (2), iphone (2), biggest (2), sources (2), short (2), exact (2), zed (2), access (2), platform (2), near (2), term (2), capable (2), mostly (2), would (2), targets (2), skills (2), coordination (2), multi (2), mapping (2), instead (2), controller (2), obtain (2), efficiently (2), generally (2), make (2), required (2), symbolic (2), operator (2), inputs (2), structured (2), tuned (2), extracts (2), days (2), automatically (2), height (2), mount (2), embodiments (2), collecting (2), everyday (2), trajectories (2), rare (2), humans (2), mind (2), comparing (2), random (2), against (2), perception (2), extra (2), asked (2), asking (2), latent (2), sitting (2), steps (2), internal (2), information (2), simple (2), beyond (2), token (2), enough (2), invariant (2), central (2), between (2), everywhere (2), entities (2), noisy (2), delivers (2), synchronized (2), collect (2), accurate (2), wears (2), performs (2), demonstrating (2), bottom (2), pyramid (2), effectively (2), merely (2), cheap (2), substitute (2), superior (2), anyone (2), anywhere (2), anytime (2), specialized (2), faq (2), gains (2), key (2), bridging (2), methods (2), drives (2), arbitrary (2), condition (2), setup (2), evaluation (2), amplify (2), outperforms (2), four (2), bridges (2), recordings (2), bridge (2), abstract (2), insights (2), grab (2), tissue (2), cabinet (2), door (2), unscrew (2), cap (2), charge (2), adjust (2), semg (2), force (2), talk (2), ego (2), adapted, github, academic, nerfies, misc, humanego2026, title, author, year, 2026, eprint, 2605, 24934, archiveprefix, primaryclass, thank, members, helpful, throughout, prg, zikui, cai, yoonkyo, jung, eadom, dessalene, there, intuition, draws, cleanly, recall, idea, complete, pouring, blind, inserting, usb, threading, reacting, ones, coincidence, seen, side, improvable, component, sharper, raise, sensing, frontier, couldn, closed, after, limits, leans, foundation, sets, floor, rules, insertion, describes, cloth, rope, soft, bag, defined, fails, acts, snapshot, frequency, reaction, contact, currently, rigid, accumulates, shine, struggles, sub, centimeter, reactive, reliably, roughly, highly, motions, large, broad, strokes, limitations, personal, honest, sure, crack, cases, suspect, budget, researching, welcome, conversation, scaling, axis, report, pieces, automatic, driven, caps, paying, grows, love, longer, horizon, gets, exciting, don, last, word, quite, rely, expected, glad, finding, provokes, drive, within, deserves, serious, firmly, held, contribute, modest, second, initial, localizing, itself, doing, genuine, dominant, assumed, building, beats, claim, useless, worked, masking, drawing, moved, finally, made, outcome, load, finish, largely, validated, thesis, rendering, role, play, shines, delivering, resulting, crucially, improving, keeps, shrinking, worth, compatibility, lower, depend, specifically, depends, unchanged, builds, cleans, trains, some, natively, headset, onboard, others, feed, absolutely, apple, etc, mounted, moves, whole, build, stays, dropping, removes, plausibly, least, identical, disappears, simpler, capture, around, recorded, ran, came, strong, distributed, program, apply, kit, eligibility, application, details, projectaria, com, getting, started, available, ships, line, generation, consumes, generations, effort, straightforward, upgrade, trying, list, substantially, november, 2025, timing, get, transfer, grasping, transporting, regrasping, individual, cautious, tactile, forgiving, truly, delicate, stuff, squarely, roadmap, global, maps, standard, supporting, fingered, tracks, collapses, down, degree, freedom, main, change, now, outputs, scalar, dimensional, feasible, extension, redesign, parallel, jaw, while, valid, option, care, based, route, solves, ever, committing, haven, smuggled, modeling, burden, falls, priors, definitions, states, collision, buy, essentially, pays, specified, prior, knowledge, constructing, adds, swap, rearrange, clutter, constraints, accomplish, script, instance, handled, course, ticket, modeled, engineered, perfectly, legitimate, approaches, nowhere, equal, handful, perform, network, recovers, domain, operators, stack, assembled, expert, certainly, given, ours, uses, bound, kinematic, base, target, born, typically, means, entire, inherently, generates, cover, larger, switch, grip, strategies, fly, adapt, milliseconds, active, contrast, smoothness, speed, dexterity, bottleneck, piloting, tier, exists, produces, choppy, slow, unrepresentative, expensive, lossy, remote, smooth, physically, plausible, intrinsically, easier, produce, important, say, mean, captured, intent, stable, footage, apples, oranges, losses, share, forced, respond, critically, computed, gave, annotation, questions, causal, zoom, spaces, together, evolves, few, thing, add, sharing, manipulated, future, plane, projection, path, watching, ahead, consistency, trace, secretly, end, aren, predict, label, receives, timestep, regress, chunk, narrow, lot, bits, flowing, gradient, mix, choices, widely, empirically, jumps, believe, purpose, encoding, embodied, agent, reason, interactions, critical, inspired, observation, elegantly, compact, lives, format, estimators, works, variable, length, easy, implement, unified, class, citizens, obviously, strictly, imagine, glancing, closing, putting, flower, vase, brief, gives, map, rest, continued, confirmation, several, surfaces, proprioception, experience, viewpoint, describe, tall, powerful, really, experiments, show, propagate, loss, enable, converge, consumer, apart, category, wear, extended, sessions, stream, turnkey, session, production, mature, becomes, making, hours, bit, interactive, interacts, reproduce, want, smallest, someone, deliberately, cleaner, deployable, middle, passive, unlabeled, layer, largest, lowest, useful, analogous, known, internet, hosts, enormous, train, viable, shortcut, datasets, issues, suffer, shakes, super, quantity, poor, activities, uncompensated, see, corpus, long, operate, helping, homes, kitchens, labs, workshops, direct, minute, encodes, brain, jointly, lens, interacting, richest, under, explored, existence, rig, objective, individually, contributes, combine, five, configurations, plateau, strategy, yields, placements, maintains, different, retraining, scarce, minimal, trained, surpasses, act, exhibits, snr, smoother, idle, greater, diversity, excels, reasoning, half, consistent, improvements, baselines, overall, publicly, release, annotations, browse, recording, visualization, below, screen, water_flowers, serve_bread, 122, takes, particularly, suited, streams, wearable, machine, services, seconds, deployed, convenient, regardless, lighting, background, relationships, bimanual, transferring, challenging, due, kinematics, present, framework, lifting, average, matched, robustly, cameras, introduce, powered, contain, provides, neither, nor, defines, browser, tag, rodney, brooks, simulator, let, interact, physics, views, low, wrist, band, muscle, activity, aware, forceband, forceful, workshop, 3d视觉工坊, 源策未来, archon, robolens, 上海人工智能实验室, shanghai, 将门技术社区, techbeat, saturdayrobotic, saturday, umd, eai, times, 具身纪元, synced, 机器之心, featured, slides, wechat, group, linkedin, code, demos,

Text of the page (random words):
ts bridge the embodiment gap efficiency of human demos one policy many conditions what drives performance faq our team acknowledge bibtex human ego zero shot robot learning from minutes of human egocentric videos robot data free hardware agnostic data efficient zero shot transferable zhi leo wang botao he kelin yu seungjae lee ruohan gao furong huang yiannis aloimonos paper arxiv code 0 dataset data gallery video x 0 linkedin humanego wechat group 5 min talk en 1h talk cn slides bibtex as featured in 机器之心 synced 具身纪元 eai times umd today university of maryland saturday robotics saturdayrobotic techbeat 将门技术社区 shanghai ai lab 上海人工智能实验室 robolens 源策未来 archon robotics 3d视觉工坊 3d vision workshop follow up work forceband forceful manipulation with semg a low cost wrist worn semg band that turns muscle activity into per finger force labels for force aware robot manipulation views the world is its own best model it is better to use the world as its own simulator and let the body directly interact with the physical physics rodney brooks real world video clips water flowers downstack cups adjust table serve bread charge devices unscrew cap open door open cabinet grab tissue downstack cups cross embodiment test water flowers generalization test serve bread generalization test humanego full video your browser does not support the video tag insights human video is not merely a cheap substitute but a superior and more efficient data source for policy learning explicit spatial representation not visual fidelity is the key to bridging the embodiment gap neither hand nor object alone defines a skill what matters is their interaction beyond action labels human videos contain way more information richer supervision signal provides complementary gains overview we introduce humanego a robot data free hardware agnostic and data efficient pipeline that learns robot manipulation policies from minutes of raw human egocentric videos powered by a flow matching policy with dense auxiliary objectives abstract human egocentric video captures rich manipulation demonstrations without any robot hardware yet transferring these skills to robots remains challenging due to the embodiment gap between human and robot in both visual appearance and kinematics we present humanego a framework that bridges the embodiment gap by lifting each human demonstration to an entity level representation of hand object interaction and training a flow matching policy with dense auxiliary objectives that amplify supervision from every trajectory humanego is robot data free hardware agnostic data efficient and zero shot human to robot transferable with only 30 minutes of human videos per task humanego achieves 92 5 average success across four real world tasks 75 with just 15 minutes outperforms matched time robot teleoperation by 41 and robustly transfers zero shot across novel robots cameras and environments architecture system overview arm inpainting and visual keypoints bridge the visual gap interaction centric tokens ict encode spatial relationships among all entities a flow matching policy with dense auxiliary objectives learns bimanual robot actions from minutes scale human data data collection and data preprocessing data collection data collection by anyone anytime anywhere policies deployed to any lab any camera any robot all from only 30 minutes of data a human demonstrator wears aria glasses and performs each task in any convenient environment regardless of table height lighting or background and without specialized workspace or calibration data collection setup data preprocessing each demonstration takes only seconds aria glasses are particularly well suited for learning from human video their machine perception services mps provide high quality 6 dof slam tracking calibrated 3d hand pose estimation and synchronized egocentric rgb streams all from a single lightweight wearable device hand to gripper mapping dataset we publicly release the humanego dataset on leo tx humanego 122 recordings across two tasks serve_bread water_flowers with the raw aria recordings mps annotations and the full preprocessing output browse every recording s visualization below or open the gallery full screen dataset open gallery results humanego bridges the embodiment gap efficiently overall real world evaluation real world success rate for each method across all four tasks humanego with 30 min of data achieves the highest success rate on every task demonstrating consistent improvements over both human video baselines and robot teleoperation methods humanego achieves the highest success rate on every single task even with half the data humanego outperforms robot teleoperation humanego excels on tasks that demand precise coordination and spatial reasoning results the efficiency of human demonstrations human vs robot data human egocentric data exhibits higher snr smoother motion less idle time top and greater spatial and trajectory diversity bottom data efficiency success rate vs data collection time humanego trained on 8 min of human data surpasses act s 30 min robot data humanego learns effectively from minimal human data auxiliary objectives amplify learning when demonstrations are scarce human video is a more efficient data source than robot teleoperation results one policy many conditions cross condition real world evaluation cross embodiment environment setup zero shot cross condition generalization humanego maintains robust success across different conditions without retraining humanego is robust to arbitrary visual conditions humanego is robust to arbitrary object placements humanego is hardware agnostic results what drives performance of humanego representation study success rate for five input configurations visual only methods plateau at 32 5 with any strategy adding spatial tokens yields 52 5 pp auxiliary training study success rate at 15 min of data for each auxiliary objective individually object motion contributes the most 17 5 pp all three combine for 25 pp explicit spatial representation not visual fidelity is the key to bridging the embodiment gap auxiliary objectives provide complementary gains faq why use human data human egocentric video captures rich manipulation demonstrations without any robot hardware anyone can collect it anywhere anytime no specialized workspace no teleoperation rig no calibration more fundamentally we see the corpus of human world interaction as one of the richest yet most under explored data sources in existence if the long term goal is robots that operate effectively in the human world helping people in homes kitchens labs and workshops then the most direct and natural source of supervision is people themselves interacting with that exact world every minute of egocentric video encodes how a body a brain and the physical environment jointly solve a manipulation task from this lens human video isn t merely a cheap substitute for robot data it is a superior and more efficient data source for policy learning why efficient robot learning from human egocentric videos the internet hosts an enormous amount of human video and people often assume just train on youtube is a viable shortcut but if you actually look at what s inside these datasets you find all kinds of issues most clips have no accurate action labels many suffer from uncompensated head motion so the camera shakes everywhere and most are random everyday activities with no specific task in mind human data is super rich in quantity but actually very poor in quality we find it useful to think about an egocentric data pyramid analogous to the well known robotics data pyramid bottom largest lowest quality passive videos like youtube pixels only unlabeled noisy the human in frame isn t collecting data for us the biggest layer but also the hardest to use directly middle egocentric demonstrations someone deliberately wears a camera and performs a task with hand poses tracked cleaner has action labels the human is actively demonstrating but still not good enough for training a deployable policy top smallest highest quality teleop grade human data fully structured interactive with accurate action labels where the human interacts with the scene in a way a robot could reproduce this is what we want but it is very rare so the central question becomes how do we squeeze every bit of learning signal out of the small amount of teleop grade human data we can actually collect that is exactly what humanego is built for making the most of minutes not hours or days of high quality human egocentric data why use aria glasses aria glasses are today the most mature and capable platform for egocentric data collection two things set them apart from anything else in the category genuinely lightweight production grade hardware that a demonstrator can wear naturally for extended sessions in any environment and meta s mps pipeline which delivers calibrated slam hand pose estimation and synchronized multi stream egocentric rgb out of the box turnkey no per session calibration the precision of these signals is what really matters our experiments show that the accuracy of the upstream slam and hand tracking signal directly bounds downstream policy performance noisy poses propagate into the action loss and the learned representation the clean drift free trajectories from aria are exactly what enable humanego to converge on minutes of data no other consumer egocentric device today delivers this level of out of the box accuracy why are interaction centric tokens icts so powerful and why doesn t visual fidelity matter as much icts encode each entity hand and object by its 6 dof pose relative to other task entities not relative to the camera or a fixed world frame this makes the representation invariant to embodiment viewpoint and environment the same ict tokens describe the same skill whether the demonstrator is a human or a robot whether the camera is a realsense or a zed whether the table is tall or short the policy learns once and transfers everywhere as for why visual fidelity matters less than people expect vision is obviously central to how humans experience the world but for most manipulation tasks it isn t strictly required imagine glancing at a table closing your eyes and then putting a flower into a vase you can still do it once a brief look gives you a coarse spatial map the rest of the task runs on spatial memory proprioception touch and the geometric interaction between hand and object not on continued visual confirmation manipulation is fundamentally a spatial and interaction problem vision is just one of several inputs that surfaces that structure inspired by this we designed ict to make spatial observation and hand object interaction the first class citizens of the policy the result is an elegantly compact representation that is unified every entity hand or object lives in the same token format easy to implement off the shelf pose estimators are enough variable length works whether the scene has one object or many and embodiment invariant the same token whether the body is a human or a robot that mix of simple but right choices is precisely why humanego generalizes so widely empirically adding icts to raw human rgb jumps water flowers success from 7 5 to 85 a 77 5 pp gap that no amount of visual preprocessing can close we believe ict is a simple yet critical representation that should reach well beyond this paper a general purpose spatial encoding for any embodied agent that needs to reason about object interactions why do dense auxiliary objectives work think about what supervision the policy actually receives from a single demonstration at each timestep the model sees one image and one set of icts and is asked to regress one action chunk that s a single narrow learning signal action in action out with only minutes of data that s just not a lot of bits flowing through the gradient but each demonstration secretly carries far more information than the action label alone where does the object end up where do the hand and object project on the image over time how does the scene s internal state evolve all of that signal is sitting for free inside every trajectory we just aren t asking the model to predict it so we add three auxiliary objectives all sharing the same encoder as the flow matching head object motion forecast each manipulated object s future 6 dof trajectory 2d trace forecast the image plane projection of hand and object the path your eyes would follow watching the video and latent consistency forecast the encoder s own internal representation k steps ahead zoom out and all three are asking the model to do the same thing forecast how the scene evolves over the next few steps in three complementary spaces 3d physical 2d visual and the encoder s own latent state together this turns the encoder into a lightweight world model of hand object interaction sitting right inside the policy for free because all three losses share the same encoder the encoder is forced to learn the causal structure of manipulation not just what action comes next but why how the scene will respond and critically none of these targets cost us anything to obtain they are computed automatically from the same perception pipeline that gave us the icts no extra annotation no extra data just more questions asked of the same demonstration why is human data better and more efficient than robot data first an important caveat when we say human data we mean carefully designed human egocentric data captured with intent the manipulation task clearly in mind the camera stable on the head the hand pose tracked comparing random youtube footage against carefully designed teleop data is apples to oranges what we are comparing here is carefully designed human data against carefully designed teleop data higher quality data is intrinsically easier for humans to produce a demonstrator naturally generates motion that is smooth dexterous fast and physically plausible they cover a much larger workspace than any single robot switch grip strategies on the fly and adapt in milliseconds without any active control loop or training teleoperation by contrast is fundamentally a lossy remote control problem smoothness speed and dexterity all bottleneck on the operator s skill at piloting the robot top tier teleop exists but it is rare and expensive everyday teleop produces choppy slow or unrepresentative trajectories that the policy then has to learn from human data generalizes more naturally across embodiments and environments given the right processing pipeline ours uses an entity relative ict representation human data is inherently more transferable it isn t bound to any particular robot s kinematic configuration gripper base height or camera mount so the same dataset can serve many target embodiments teleop data is the opposite it is born inside one specific robot s body so a new arm a new gripper or even a new camera mount typically means re collecting the en...