Meta tags:

Headings (most frequently used words):

grasping, on, real, world, toys, results, randomized, zero, shot, demos, trained, tested, objects, maniskill, franka, droid, dexterous, hands, lego, simulation, h1, scaling, ablations, humanoid, ycb, benchmark, background, detection, pooling, creating, robot, citation, learning, to, grasp, anything, by, playing, with, random,

Text of the page (most frequently used words):
the (63), and (35), toys (31), grasping (30), object (23), zero (19), shot (19), #pooling (17), objects (17), real (16), world (16), #randomized (15), results (14), demonstrations (13), set (13), with (10), from (9), detection (9), for (9), tuned (8), success (8), rate (8), number (8), training (8), that (8), lego (7), ycb (7), model (7), large (7), scale (7), demos (6), droid (6), learning (6), trained (6), tested (6), fast (6), using (6), ours (6), attention (6), dexterous (5), models (5), 250 (5), evaluation (5), each (5), final (5), openvla (5), uses (5), simulation (5), our (5), pretraining (5), toy (5), tokens (5), maniskill (4), benchmark (4), franka (4), hands (4), random (4), are (4), across (4), overall (4), 500 (4), unique (4), per (4), transformer (4), policy (4), vit (4), generalizable (4), generalization (4), mean (4), primitives (4), features (4), visual (4), mask (4), patch (4), centric (4), robot (3), humanoid (3), grasp (3), arxiv (3), get (3), which (3), achieving (3), study (3), performance (3), shown (3), figure (3), find (3), outperforms (3), data (3), robotic (3), skills (3), can (3), simple (3), four (3), then (3), vision (3), feature (3), encoder (3), this (3), manipulation (3), robots (3), citation (2), creating (2), background (2), anything (2), playing (2), dantong (2), niu (2), yuvan (2), sharma (2), baifeng (2), shi (2), rachel (2), ding (2), matteo (2), gioia (2), haoru (2), xue (2), henry (2), tsai (2), konstantinos (2), kallidromitis (2), anirudh (2), pai (2), shankar (2), shastry (2), trevor (2), darrell (2), jitendra (2), malik (2), roei (2), herzig (2), 2510 (2), 12866 (2), setting (2), total (2), 1500 (2), during (2), times (2), predefined (2), grid (2), averaged (2), shapegrasp (2), llm (2), were (2), ablation (2), how (2), both (2), specifically (2), 1000 (2), left (2), panel (2), below (2), right (2), size (2), function (2), scaling (2), ablations (2), key (2), cézanne (2), into (2), shape (2), use (2), cuboids (2), spheres (2), cylinders (2), rings (2), while (2), target (2), output (2), methods (2), only (2), ensures (2), learn (2), novel (2), mechanism (2), dataset (2), generalize (2), equal (2), university (2), website, adapted, leverb, misc, niu2025learninggraspplayingrandom, title, author, year, 2025, eprint, archiveprefix, primaryclass, url, https, org, abs, 4x4, such, pre, choosing, points, was, perform, examine, influence, construct, six, sets, containing, 125, respectively, collect, train, varying, numbers, indicate, contrast, illustrates, where, 86m, parameters, achieve, best, has, stronger, impact, increasing, improves, but, diminishing, returns, vla, environment, all, same, finetuned, baselines, proving, det, cls, attn, oft, 2500, 2000, method, created, primitive, shapes, create, draw, inspiration, classic, idea, everyday, decomposed, combinations, basic, generate, selecting, five, assigning, them, positions, orientations, enforcing, intersections, ensure, forms, solid, structure, assigned, one, colors, red, green, blue, yellow, experiments, print, these, examples, extract, predict, future, actions, based, proprioception, extracts, focus, via, restrains, patches, performs, empirical, show, crucial, strong, compared, other, not, restrict, within, pool, see, section, obtained, applying, fed, standard, architecture, positional, embeddings, still, allow, understand, location, scene, contain, itself, preventing, between, non, first, obtain, segmentation, frame, ground, truth, masks, sam, focuses, grasped, generalizes, design, called, approach, provides, promising, path, toward, scalable, also, scales, diversity, evaluated, physical, achieves, outperforming, state, art, more, domain, induced, representation, composed, just, inspired, ability, children, investigate, whether, develop, small, policies, often, struggle, limiting, utility, paul, treat, nature, means, cylinder, sphere, cone, everything, brought, proper, perspective, github, enables, tldr, contribution, advising, work, done, interning, italai, california, berkeley, sapienza, rome, panasonic,

Text of the page (random words):
lego learning to grasp anything by playing with random toys dantong niu 1 yuvan sharma 1 baifeng shi 1 rachel ding 1 matteo gioia 2 haoru xue 1 henry tsai 1 konstantinos kallidromitis 3 anirudh pai 1 shankar shastry 1 trevor darrell 1 jitendra malik 1 roei herzig 1 equal contribution equal advising work done while interning at italai 1 university of california berkeley 2 sapienza university of rome 3 panasonic tldr training robots on random toys enables zero shot grasping of real world objects arxiv lego dataset github zero shot grasping demos dexterous hands humanoid trained on randomized toys tested on real world objects grasping demonstrations for randomized toys training set zero shot grasping on real world objects evaluation set zero shot grasping demos franka droid trained on randomized toys tested on real world objects grasping demonstrations for randomized toys training set zero shot grasping on ycb real world objects evaluation set zero shot grasping demos maniskill ycb benchmark trained on randomized toys tested on real world objects grasping demonstrations for randomized toys training set zero shot grasping on real world objects evaluation set background treat nature by means of the cylinder the sphere the cone everything brought into proper perspective paul cézanne robotic manipulation policies often struggle to generalize to novel objects limiting real world utility inspired by the ability of children to learn manipulation skills from simple toys we investigate whether robots can develop generalizable dexterous grasping skills by learning from a small set of simple randomized toy objects training uses objects composed of just four shape primitives spheres cuboids cylinders and rings and we find our model can generalize zero shot to 64 real world objects we find that the key to this generalization is an object centric visual representation induced by our detection pooling mechanism evaluated in simulation and on physical robots our model achieves a 67 real world grasping success rate on the ycb dataset outperforming state of the art methods using more in domain data we also study how zero shot generalization scales with the number diversity of training toys and demonstrations per toy this approach provides a promising path toward scalable and generalizable robotic manipulation learning detection pooling to learn a policy that generalizes to novel objects from randomized toys we design the vision encoder to be object centric using a mechanism called detection pooling detection pooling ensures that the visual feature focuses on the object to be grasped first we obtain the object segmentation mask for each frame using sam 2 or using ground truth masks in simulation we then use the object mask to set the attention mask in the vision encoder preventing attention between object and non object patch tokens this ensures object patch tokens only contain features from the object itself positional embeddings still allow the encoder to understand the object s location in the scene the final object centric visual feature is obtained by applying mean pooling on the object patch tokens which is then fed to the policy model a standard transformer architecture empirical results show that detection pooling is crucial for strong zero shot generalization compared to other pooling methods mean or attention pooling that do not restrict attention within the vit and only pool the final output tokens see results section a lego uses a vit with detection pooling to extract features of the target object and uses a transformer to predict future actions based on the visual features and the proprioception b the vit extracts features that focus on the target object via detection pooling which restrains the attention to the object patches using an attention mask and performs mean pooling on the output object patch tokens to get the final object centric vision feature creating randomized robot toys to create randomized toys for learning generalizable grasping skills we draw inspiration from cézanne s classic idea that everyday objects can be decomposed into combinations of simple shape primitives specifically we use four basic primitives cuboids spheres cylinders and rings and generate randomized toys by selecting up to five primitives per toy and assigning them random positions and orientations while enforcing intersections to ensure the overall object forms solid structure each toy is then assigned one of four colors red green blue or yellow for real world experiments we 3d print 250 of these toys examples of the randomized toys are shown in the figure below randomized robot toys created from primitive shapes results maniskill simulation grasping results results of zero shot grasping in the maniskill simulation environment on 65 objects from the ycb object benchmark all models shown are trained on the same number of randomized toy grasping demonstrations our model outperforms finetuned baselines achieving a 80 success rate with detection pooling proving key to generalization method demonstrations 250 500 1000 1500 2000 2500 openvla oft 30 10 36 35 22 31 15 38 14 71 12 79 π fast 8 85 7 60 7 69 8 56 4 23 4 13 ours attn pooling 34 71 40 10 44 23 48 27 49 81 51 63 ours cls pooling 24 71 20 29 36 92 41 44 42 40 49 81 ours mean pooling 32 98 30 38 36 15 39 90 40 29 40 58 ours det pooling 56 63 68 17 71 15 74 62 76 83 80 00 real world franka droid grasping results results of zero shot grasping in the real world franka droid setting on 64 objects from the ycb object benchmark models tuned on toys are trained with a total of 1500 grasping demonstrations across 250 toys during evaluation each ycb object is tested 16 times across a predefined 4x4 grid and results are averaged to get the final success rate lego outperforms large scale robotic models such as zero shot π fast and openvla and shapegrasp which uses a pre trained llm for choosing grasp points achieving an overall success rate of 67 openvla tuned on toys large scale pretraining 50 shapegrasp zero shot uses llm 50 π fast zero shot large scale pretraining with droid data 50 ours tuned on toys 50 π fast tuned on toys large scale pretraining with droid data 50 real world h1 2 dexterous hands grasping results results of zero shot grasping in the real world h1 2 humanoid with dexterous hands setting on 13 real world objects models were tuned on a total of 500 grasping demonstrations across 250 toys during evaluation each object was tested 5 times across a predefined grid and results were averaged to get the final success rate lego outperforms both large scale vla models tested π fast and openvla achieving an overall success rate of 51 openvla tuned on toys large scale pretraining 50 π fast tuned on toys large scale pretraining 50 ours tuned on toys 50 scaling ablations we perform an ablation study in simulation to examine how both the number of unique toys in the training set and the number of grasping demonstrations influence performance specifically we construct six object sets containing 1 25 125 250 500 and 1000 unique toys respectively for each set we collect 2 500 grasping demonstrations and train our model using varying numbers of demonstrations per set the results shown in the left panel of the figure below indicate that increasing the number of unique objects improves performance but with diminishing returns in contrast the number of demonstrations has a stronger impact on learning generalizable grasping the right panel of the figure illustrates the results of an ablation study on the size of the model s transformer policy where we find vit b 86m parameters to achieve the best overall performance scaling ablations left grasping success rate as a function of the number of unique toys in the training set and the number of demonstrations per set right grasping success rate as a function of the size of the transformer policy model citation misc niu2025learninggraspplayingrandom title learning to grasp anything by playing with random toys author dantong niu and yuvan sharma and baifeng shi and rachel ding and matteo gioia and haoru xue and henry tsai and konstantinos kallidromitis and anirudh pai and shankar shastry and trevor darrell and jitendra malik and roei herzig year 2025 eprint 2510 12866 archiveprefix arxiv primaryclass cs ro url https arxiv org abs 2510 12866 lego zero shot grasping demos dexterous hands humanoid zero shot grasping demos franka droid zero shot grasping demos maniskill ycb benchmark background detection pooling creating randomized robot toys results citation lego website adapted from leverb