Meta tags:
author=Krishna Murthy Jatavallabhula;
description=Krishna Murthy - Research scientist, Meta;

Headings (most frequently used words):

open, 3d, for, and, krishna, murthy, jatavallabhula, news, featured, publications, conceptgraphs, vocabulary, scene, graphs, perception, planning, conceptfusion, set, multimodal, mapping, gradsim, differentiable, simulation, system, identification, visuomotor, control, gradslam, dense, slam, meets, automatic, differentiation,

Text of the page (most frequently used words):
and (99), the (56), for (24), with (20), this (15), #differentiable (15), krishna (14), slam (14), 2020 (14), #murthy (12), jatavallabhula (12), open (12), work (11), that (11), 2024 (11), florian (10), 2021 (10), set (9), paull (8), liam (8), learning (8), from (8), conceptfusion (8), dense (7), video (7), shkurti (7), paper (7), models (7), sep (7), gradslam (6), icra (6), featured (6), systems (6), representation (6), approaches (6), maps (6), code (6), webpage (6), planning (6), concepts (6), conceptgraphs (6), author (5), iyer (5), ganesh (5), year (5), based (5), enables (5), system (5), experiments (5), scene (5), tenenbaum (5), torralba (5), antonio (5), world (5), phd (5), 2026 (4), inproceedings (4), title (4), booktitle (4), true (4), end (4), new (4), mapping (4), are (4), graphs (4), all (4), contributions (4), bib (4), abs (4), pdf (4), gradsim (4), visuomotor (4), control (4), martin (4), derek (4), which (4), image (4), such (4), require (4), them (4), our (4), kuwajerwala (4), alihusein (4), qiao (4), joshua (4), melo (4), celso (4), miguel (4), 2023 (4), rss (4), robot (4), semantic (4), they (4), language (4), perception (4), will (4), feb (4), organizing (4), university (4), 2022 (4), automatic (3), differentiation (3), idea (3), led (3), fusion (3), present (3), have (3), task (3), performance (3), representations (3), not (3), computational (3), graph (3), across (3), demonstrate (3), how (3), without (3), simulation (3), identification (3), miles (3), golemo (3), vikram (3), breandan (3), sanja (3), iclr (3), engine (3), physics (3), rendering (3), datasets (3), helped (3), time (3), dynamics (3), formation (3), physical (3), problem (3), best (3), labels (3), through (3), tasks (3), than (3), multimodal (3), tabletop (3), can (3), reason (3), these (3), reasoning (3), multi (3), foundation (3), spatial (3), vocabulary (3), robots (3), nvidia (3), nov (3), fellowship (3), google (3), jan (3), fall (3), serving (3), mit (3), 2025 (3), johns (3), hopkins (3), meets (2), was (2), point (2), pipelines (2), deep (2), question (2), central (2), leverage (2), data (2), directly (2), however (2), has (2), been (2), traditional (2), leveraging (2), capabilities (2), design (2), gradient (2), each (2), their (2), well (2), over (2), pixels (2), macklin (2), voleti (2), petrini (2), linda (2), weiss (2), considine (2), parent (2), levesque (2), jerome (2), xie (2), kevin (2), erleben (2), kenny (2), nowrouzezahrai (2), fidler (2), jointly (2), conceived (2), joined (2), focusing (2), combination (2), also (2), manuscript (2), fundamentally (2), precise (2), framework (2), supervision (2), better (2), omama (2), mohd (2), chen (2), tao (2), shuang (2), saryazdi (2), soroush (2), keetha (2), nikhil (2), tewari (2), ayush (2), madhava (2), dataset (2), building (2), environment (2), closed (2), only (2), pre (2), training (2), using (2), class (2), recent (2), prompts (2), both (2), enabling (2), images (2), audio (2), scale (2), about (2), features (2), view (2), long (2), number (2), real (2), environments (2), morin (2), sacha (2), sen (2), bipasha (2), agarwal (2), aditya (2), rivera (2), corban (2), paul (2), william (2), ellis (2), kirsty (2), chellappa (2), rama (2), gan (2), chuang (2), prior (2), large (2), vision (2), downstream (2), publications (2), 2019 (2), ieee (2), ral (2), jun (2), pioneers (2), cohort (2), workshop (2), neurips (2), seminar (2), series (2), papers (2), associate (2), editor (2), iros (2), mar (2), update (2), including (2), robotics (2), chair (2), postdoc (2), csail (2), meta (2), research (2), scientist (2), department (2), assistant (2), professor (2), sciphy (2), lab (2), action (2), copyright, hand, rolled, last, updated, july, astro, came, instrumental, implementing, surfel, possibilites, integrating, context, simultaneous, localization, newer, potential, inform, choice, because, take, allow, components, whole, achieved, creating, alternatives, non, component, typical, specifically, trust, region, optimizers, surface, measurement, schemes, differentiate, rays, sacrificing, amalgamation, backprop, way, opening, possibilities, meeting, included, bhairav, mehta, maxime, chevalier, boisvert, prototyped, initial, billiards, implemented, first, rigid, body, integrated, renderer, setup, sys, then, forces, him, overall, integration, ran, created, designed, authors, participated, writing, response, phase, nearly, equally, advised, effort, varying, result, highly, accurate, parameter, estimation, tackle, estimating, object, properties, mass, friction, elasticity, sequences, ill, posed, due, loss, information, during, current, solutions, labor, intensive, gather, infeasible, create, many, deformable, solids, cloth, overcomes, dependence, combining, multiphysics, model, evolution, unique, backpropagation, sequence, underlying, attributes, generated, furthermore, unified, computation, engines, challenging, relying, state, while, obtaining, competitive, techniques, project, wrote, much, curated, annotated, uncoco, navigation, interaction, objects, most, existing, integrate, largely, remain, confined, setting, bout, finite, defined, further, queried, text, address, issues, beyond, inherently, modal, diverse, range, possible, queries, map, geometry, working, concert, leverages, today, trained, internet, modalities, natural, pixel, aligned, fused, into, via, effective, zero, shot, needing, any, additional, finetuning, retains, tailed, supervised, outperforming, more, percent, margin, iou, extensively, evaluate, simulated, home, manipulation, autonomous, driving, platform, showcase, avenues, blending, multiple, aspects, off, perform, wide, variety, semantically, rich, yet, compact, efficient, driven, attempted, encode, semantics, tend, produce, per, feature, vectors, larger, nor, contain, relationships, between, entities, useful, propose, structured, scenes, built, fusing, output, association, resulting, generalize, novel, classes, need, collect, finetune, utility, specified, abstract, complex, released, library, kaolin, fully, virtually, presented, named, maplite, selected, jul, graphics, aug, mila, available, source, pytorch, here, honored, received, graduate, rethinking, dec, awarded, declined, teaching, mcgill, realistic, advanced, synthesis, workshops, iccv, pribr, diff3d, got, grade, exceptional, moved, start, potsdoc, josh, overdue, another, featuring, accepted, speaking, umd, microsoft, future, leaders, openreview, corl, may, completed, eventful, stint, serve, area, cvpr, fair, join, announcing, starting, oct, news, spent, team, candidate, recognized, awards, award, induction, universite, montreal, strive, build, full, stack, robotic, perceive, act, human, level, efficiency, ultimately, surpassing, lies, tackling, should, represent, around, use, interface, incoming, where, direct, linkedin, github, thesis, scholar, email, blog,

Text of the page (random words):
krishna murthy jatavallabhula krishna murthy jatavallabhula about blog publications krishna murthy jatavallabhula world models for robots email google scholar cv phd thesis github linkedin i am an incoming assistant professor with the cs department at johns hopkins university where i will direct the sciphy lab i strive to build full stack robotic systems that perceive reason and act with human level efficiency ultimately surpassing them my work lies at the perception action interface tackling both how robots should represent the world around them and how they use it for action prior to this i spent time as a research scientist at meta s robotics team as a postdoc at mit csail and as a phd candidate at universite de montreal my work has been recognized with phd fellowship awards from nvidia and google a best paper award from ieee ral and an induction to the rss pioneers 2020 cohort news oct 19 2025 announcing the sciphy lab at johns hopkins university starting fall 2026 jun 12 2025 by fall 2026 i will join the cs department at johns hopkins university as an assistant professor sep 15 2024 i joined fair meta as an ai research scientist sep 13 2024 i will serve as an area chair for cvpr 2025 sep 12 2024 i completed an eventful 2 5 year postdoc stint at mit csail may 31 2024 serving as openreview chair for corl 2024 feb 29 2024 speaking at the umd microsoft future leaders in robotics and ai seminar series jan 28 2024 6 papers accepted to icra 2024 jan 22 2024 serving as associate editor for iros and ra l sep 27 2023 another webpage update featuring new work including conceptgraphs feb 11 2023 long overdue webpage update including the featured conceptfusion work mar 14 2022 i moved to mit to start my potsdoc with josh tenenbaum and antonio torralba mar 10 2022 got my phd with grade exceptional feb 1 2022 serving as associate editor for iros 2022 sep 14 2021 organizing workshops diff3d iccv 2021 and the pribr at neurips 2021 sep 4 2021 teaching the realistic advanced image synthesis class at mcgill university fall 2021 jan 21 2021 awarded a google phd fellowship declined dec 20 2020 organizing the rethinking ml papers workshop at iclr 2021 nov 30 2020 honored to have received an nvidia graduate fellowship for 2021 22 nov 9 2020 gradslam is available as an open source pytorch framework here sep 2 2020 organizing the robot learning seminar series at mila aug 31 2020 organizing the differentiable vision graphics physics workshop at neurips 2020 jul 5 2020 selected to the rss pioneers cohort for 2020 jun 4 2020 our paper maplite named best paper ieee ral 2019 feb 11 2020 our paper on fully differentiable dense slam will be virtually presented at icra 2020 nov 13 2019 released nvidia kaolin a 3d deep learning library featured publications conceptgraphs open vocabulary 3d scene graphs for perception and planning qiao gu alihusein kuwajerwala sacha morin krishna murthy jatavallabhula bipasha sen aditya agarwal corban rivera william paul kirsty ellis rama chellappa chuang gan celso miguel de melo joshua b tenenbaum antonio torralba florian shkurti and liam paull icra 2024 pdf webpage code abs bib video contributions for robots to perform a wide variety of tasks they require a 3d representation of the world that is semantically rich yet compact and efficient for task driven perception and planning recent approaches have attempted to leverage features from large vision language models to encode semantics in 3d representations however these approaches tend to produce maps with per point feature vectors which do not scale well in larger environments nor do they contain semantic spatial relationships between entities in the environment which are useful for downstream planning in this work we propose conceptgraphs an open vocabulary graph structured representation for 3d scenes conceptgraphs is built by leveraging 2d foundation models and fusing their output to 3d by multi view association the resulting representations generalize to novel semantic classes without the need to collect large 3d datasets or finetune models we demonstrate the utility of this representation through a number of downstream planning tasks that are specified through abstract language prompts and require complex reasoning over spatial and semantic concepts i co led multiple aspects of this work building off of our prior work with conceptfusion inproceedings conceptgraphs title conceptgraphs open vocabulary 3d scene graphs for perception and planning author gu qiao and kuwajerwala alihusein and morin sacha and jatavallabhula krishna murthy and sen bipasha and agarwal aditya and rivera corban and paul william and ellis kirsty and chellappa rama and gan chuang and de melo celso miguel and tenenbaum joshua b and torralba antonio and shkurti florian and paull liam year 2024 booktitle icra featured true conceptfusion open set multimodal 3d mapping krishna murthy jatavallabhula alihusein kuwajerwala qiao gu mohd omama tao chen shuang li ganesh iyer soroush saryazdi nikhil keetha ayush tewari joshua b tenenbaum celso miguel de melo madhava krishna liam paull florian shkurti and antonio torralba rss 2023 pdf webpage code abs bib video contributions building 3d maps of the environment is central to robot navigation planning and interaction with objects in a scene most existing approaches that integrate semantic concepts with 3d maps largely remain confined to the closed set setting they can only reason bout a finite set of concepts pre defined at training time further these maps can only be queried using class labels or in recent work using text prompts we address both these issues with conceptfusion a scene representation that is i fundamentally open set enabling reasoning beyond a closed set of concepts ii inherently multi modal enabling a diverse range of possible queries to the 3d map from language to images to audio to 3d geometry all working in concert conceptfusion leverages the open set capabilities of today s foundation models pre trained on internet scale data to reason about concepts across modalities such as natural language images and audio we demonstrate that pixel aligned open set features can be fused into 3d maps via traditional slam and multi view fusion approaches this enables effective zero shot spatial reasoning not needing any additional training or finetuning and retains long tailed concepts better than supervised approaches outperforming them by more than 40 percent margin on 3d iou we extensively evaluate conceptfusion on a number of real world datasets simulated home environments a real world tabletop manipulation task and an autonomous driving platform we showcase new avenues for blending foundation models with 3d open set multimodal mapping i conceived the idea and led the project i also wrote much of the code and the paper i curated and annotated the uncoco dataset and helped with the tabletop robot experiments inproceedings conceptfusion title conceptfusion open set multimodal 3d mapping author jatavallabhula krishna murthy and kuwajerwala alihusein and gu qiao and omama mohd and chen tao and li shuang and iyer ganesh and saryazdi soroush and keetha nikhil and tewari ayush and tenenbaum joshua b and de melo celso miguel and krishna madhava and paull liam and shkurti florian and torralba antonio year 2023 booktitle rss dataset and helped with the tabletop robot experiments featured true gradsim differentiable simulation for system identification and visuomotor control krishna murthy jatavallabhula miles macklin florian golemo vikram voleti linda petrini martin weiss breandan considine jerome parent levesque kevin xie kenny erleben liam paull florian shkurti derek nowrouzezahrai and sanja fidler iclr 2021 pdf webpage code abs bib video tl dr contributions in this paper we tackle the problem of estimating object physical properties such as mass friction and elasticity directly from video sequences such a system identification problem is fundamentally ill posed due to the loss of information during image formation current best solutions to the problem require precise 3d labels which are labor intensive to gather and infeasible to create for many systems such as deformable solids or cloth in this work we present gradsim a framework that overcomes the dependence on 3d supervision by combining differentiable multiphysics simulation and differentiable rendering to jointly model the evolution of scene dynamics and image formation this unique combination enables backpropagation from pixels in a video sequence through to the underlying physical attributes that generated them furthermore our unified computation graph across dynamics and rendering engines enables the learning of challenging visuomotor control tasks without relying on state based 3d supervision while obtaining performance competitive to better than techniques that require precise 3d labels differentiable models of time varying dynamics and image formation pipelines result in highly accurate physical parameter estimation from video and visuomotor control this idea was jointly conceived in a meeting which included me derek breandan martin bhairav mehta and maxime chevalier boisvert martin prototyped an initial differentiable billiards engine i implemented the first rigid body engine integrated it with a differentiable renderer and setup sys id experiments miles and i then joined forces with him focusing on the physics engine and me focusing on the physics rendering combination and overall systems integration i ran all of the experiments for this paper florian golemo and vikram created the datasets designed experiments and also helped with code and the manuscript all authors participated in writing the manuscript and the author response phase florian shkurti derek and sanja nearly equally co advised on this effort inproceedings gradsim title gradsim differentiable simulation for system identification and visuomotor control author jatavallabhula krishna murthy and macklin miles and golemo florian and voleti vikram and petrini linda and weiss martin and considine breandan and parent levesque jerome and xie kevin and erleben kenny and paull liam and shkurti florian and nowrouzezahrai derek and fidler sanja year 2021 booktitle iclr featured true gradslam dense slam meets automatic differentiation krishna murthy jatavallabhula ganesh iyer and liam paull icra 2020 pdf webpage code abs bib video tl dr contributions the question of representation is central in the context of dense simultaneous localization and mapping slam newer learning based approaches have the potential to leverage data or task performance to directly inform the choice of representation however learning representations for slam has been an open question because traditional slam systems are not end to end differentiable in this work we present gradslam a differentiable computational graph take on slam leveraging the automatic differentiation capabilities of computational graphs gradslam enables the design of slam systems that allow for gradient based learning across each of their components or the system as a whole this is achieved by creating differentiable alternatives for each non differentiable component in a typical dense slam system specifically we demonstrate how to design differentiable trust region optimizers surface measurement and fusion schemes as well as differentiate over rays without sacrificing performance this amalgamation of dense slam with computational graphs enables us to backprop all the way from 3d maps to 2d pixels opening up new possibilities in gradient based learning for slam we present end to end differentiable dense slam systems that open up new possibilites for integrating deep learning and slam i came up with the idea and led this work ganesh iyer was instrumental with implementing the surfel and point based fusion pipelines inproceedings gradslam title gradslam dense slam meets automatic differentiation author jatavallabhula krishna murthy and iyer ganesh and paull liam year 2020 booktitle icra featured true copyright 2026 krishna murthy jatavallabhula hand rolled with astro last updated july 3 2026