Meta tags:
description=;

Headings (most frequently used words):

fine, dataset, you, the, same, why, referit3d, for, object, in, video, contrast, objects, grained, which, one, language, are, neural, listeners, grained3d, identification, real, world, scenes, eccv, 2020, oral, abstract, overview, intuitions, method, referit3dnet, qualitative, results, citation, benchmark, challenges, acknowledgements, paper, code, benchmarks, explicitly, of, class, only, ii, at, least, distractor, exists, scene, iii, collected, large, nr3d, natural, bother, making, also, synthetic, focusing, on, spatial, relations, iv, all, contrasting, contexts, and, produced, qualitatively, about, or, there, important, intrinsic, differences, among, them, browsers,

Text of the page (most frequently used words):
the (73), and (31), object (30), you (11), scene (11), fine (10), grained (10), that (10), are (9), reference (9), for (8), #language (8), which (8), nr3d (8), sr3d (8), with (7), can (7), #neural (7), utterances (7), from (6), listeners (6), class (6), referred (6), natural (6), stanford (5), this (5), scenes (5), same (5), objects (5), datasets (4), identification (4), our (4), referit3d (4), each (4), color (4), processed (4), one (4), armchair (4), contrast (4), lamp (4), target (4), fig (4), view (4), only (4), spatial (4), university (4), grant (3), research (3), kaust (3), they (3), also (3), their (3), not (3), results (3), problem (3), via (3), use (3), method (3), your (3), find (3), real (3), world (3), shown (3), two (3), network (3), dgcn (3), classifier (3), here (3), dataset (3), among (3), making (3), between (3), examples (3), because (3), why (3), free (3), form (3), contains (3), linguistic (3), wish (2), amazon (2), want (2), help (2), last (2), least (2), introduced (2), different (2), either (2), please (2), benchmark (2), work (2), achlioptas (2), panos (2), abdelreheem (2), ahmed (2), xia (2), fei (2), elhoseiny (2), mohamed (2), guibas (2), leonidas (2), eccv (2), 2020 (2), four (2), boxes (2), distractors (2), referential (2), text (2), under (2), inside (2), probabilities (2), referit3dnet (2), point (2), its (2), utterance (2), representations (2), graph (2), centric (2), context (2), scannet (2), browser (2), them (2), easy (2), hard (2), above (2), enough (2), all (2), references (2), front (2), about (2), independent (2), contexts (2), discoverable (2), contrasting (2), data (2), relations (2), collected (2), large (2), simple (2), classification (2), refrigerator (2), human (2), office (2), chair (2), task (2), focus (2), underlying (2), video (2), using (2), identify (2), develop (2), authors, acknowledge, support, vannevar, bush, faculty, fellowship, samsung, gro, program, sail, toyota, center, nsf, iis, 1763268, bas, 1685, gift, web, services, thank, iro, armeni, angel, chang, jiayun, wang, inspiring, discussions, bringing, project, fruition, but, express, gratitude, wonderful, turkers, mechanical, turk, whose, curating, was, paramount, acknowledgements, approaches, tackling, new, let, know, attained, aggregating, page, add, aggregate, highlight, challenges, useful, consider, citing, inproceedings, achlioptas2020referit_3d, title, author, booktitle, 16th, european, conference, computer, vision, year, citation, successful, cases, applying, top, images, failure, ones, bottom, targets, intra, displayed, image, predictions, dashed, circles, along, inferred, omit, inter, ease, presentation, yellow, red, green, qualitative, represented, cloud, containing, xyz, coordinates, rgb, encoded, visual, encoder, pointnet, shared, weights, simultaneously, describing, recurrent, rnn, resulting, fused, together, dynamic, convolution, creates, aware, representation, per, output, mlp, estimates, likelihood, auxiliary, losses, modulate, unfused, before, these, respectively, next, whiteboard, explore, browsers, download, 7mb, 19mb, 20mb, instance, might, single, compare, figure, elements, allow, another, person, robot, item, pair, cene, iscoverable, see, have, tick, uttering, having, specific, mind, right, beds, implies, listener, needs, face, bed, like, closer, white, theory, requires, pinpoint, without, caring, back, views, showing, difference, dependent, produced, qualitatively, there, important, intrinsic, differences, verified, reasoning, left, etc, focuses, aspect, nicely, other, properties, such, shape, even, naively, adding, training, improves, performance, comprehending, disentangling, ubiquitous, types, bother, synthetic, focusing, iii, similarly, forces, good, beyond, distractor, exists, enables, speakers, details, disambiguate, fostering, production, put, trivially, utter, furthemore, inclusion, bounding, surround, helps, annotators, especially, since, reconstructions, far, noise, explicit, efficient, minimal, coded, distinguishes, corresponding, marked, against, distracting, explicitly, overview, intuitions, common, challenging, setup, where, belongs, instances, due, scarcity, unsuitability, existent, oriented, resources, first, scale, complementary, visio, template, based, leveraging, classes, localize, deploying, player, game, recognize, high, resp, accuracy, tapping, novel, comprehend, key, technical, contribution, designing, approach, combining, geometric, information, clouds, creating, multi, modal, importantly, show, architectures, promote, communication, networks, outperform, unaware, alternatives, bottleneck, assisted, significant, directly, multiple, introduce, abstract, benchmarks, code, paper, oral,

Text of the page (random words):
referit3d referit3d neural listeners for fine grained 3d object identification in real world scenes eccv 2020 oral panos achlioptas stanford university ahmed abdelreheem kaust fei xia stanford university mohamed elhoseiny stanford university kaust leonidas guibas stanford university paper video code dataset benchmarks abstract in this work we introduce the problem of using referential language to identify common objects in real world 3d scenes we focus on a challenging setup where the referred object belongs to a fine grained object class and the underlying scene contains multiple object instances of that class due to the scarcity and unsuitability of existent 3d oriented linguistic resources for this task we first develop two large scale and complementary visio linguistic datasets i sr3d which contains 83 5k template based utterances leveraging spatial relations among fine grained object classes to localize a referred object in a scene and ii nr3d which contains 41 5k natural free form utterances collected by deploying a 2 player object reference game in 3d scenes using utterances of either datasets human listeners can recognize the referred object with high 86 92 resp accuracy by tapping on the introduced data we develop novel neural listeners that can comprehend object centric natural language and identify the referred object directly in a 3d scene a key technical contribution is designing an approach for combining linguistic and geometric information in the form of 3d point clouds and creating multi modal 3d neural listeners importantly we show that architectures which promote object to object communication via graph neural networks outperform context unaware alternatives and that fine grained object classification is a significant bottleneck for language assisted 3d object identification video overview intuitions i you contrast explicitly objects of the same fine grained object class only why fig 1 examples of natural free form utterances each color coded utterance distinguishes the corresponding object marked with same color against a distracting object in the underlying scene because this enables the human or neural speakers to use minimal details to disambiguate the target object fostering the production of efficient fine grained references put it simple if you contrast an armchair to a target office chair you can trivially utter the office chair furthemore the inclusion of explicit bounding boxes that surround the contrasting objects helps our annotators focus on the task especially since the scannet 3d reconstructions are far from noise free ii you contrast objects for which at least one same fine grained distractor exists in the scene why because similarly to the above this forces the reference to go beyond fine grained or simple object classification i e if the target is the only refrigerator of the scene the reference the refrigerator is good enough no iii you collected a large dataset nr3d which natural language why bother making also a synthetic one focusing on spatial relations fig 2 examples of spatial reference types of sr3d because as we verified in nr3d spatial reasoning left of between etc is ubiquitous in natural reference sr3d focuses on that aspect only disentangling the reference problem nicely from other object properties such as their color or shape also even naively adding sr3d to nr3d in the training data improves the listeners performance in comprehending natural language iv are all contrasting contexts and produced language qualitatively about the same or there are important intrinsic differences among them fig 3 examples of nr3d showing the difference between the easy and hard contexts scene discoverable sd and not scene discoverable utterances and view independent vi and view dependent utterances no they are not the same for instance you might contrast a lamp with a single lamp or with four of them compare easy vs hard in above figure or you can use in your language enough elements that allow another person or a robot to find the target item among all objects of the scene making the reference object pair s cene d iscoverable in fig see references that have a tick under sd last it is different from uttering an object reference having a specific view in mind e g the lamp on the right in between the beds which implies that the listener needs to find the front face of the bed vs making a reference like the lamp closer to the white armchair which in theory only requires one to pinpoint the armchair without caring about its front back views i e it is view independent vi dataset you can download nr3d here 10 7mb and sr3d sr3d here 19mb 20mb browsers you can explore the nr3d and sr3d utterances inside the 3d scannet scenes here nr3d browser sr3d browser method referit3dnet each object of a 3d scene represented as a 6d point cloud containing its xyz coordinates and rgb color is encoded by a visual encoder e g pointnet with shared weights simultaneously the utterance describing the referred object e g the armchair next to the whiteboard is processed by a recurrent neural network rnn the resulting representations are fused together and processed by a dynamic graph convolution network dgcn which creates an object centric and scene context aware representation per object the output of the dgcn is processed by an mlp classifier that estimates the likelihood of each object to be the referred one two auxiliary losses modulate the unfused representations before these are processed by the dgcn via an object class classifier and a text classifier respectively qualitative results successful cases of applying referit3dnet are shown in the top four images and failure ones in the bottom two targets are shown in green boxes intra class distractors in red and the referential text is displayed under each image the network predictions are shown inside dashed yellow circles along with the inferred probabilities we omit the probabilities of inter class distractors to ease the presentation citation if you find our work useful in your research please consider citing inproceedings achlioptas2020referit_3d title referit3d neural listeners for fine grained 3d object identification in real world scenes author achlioptas panos and abdelreheem ahmed and xia fei and elhoseiny mohamed and guibas leonidas j booktitle 16th european conference on computer vision eccv year 2020 referit3d benchmark challenges we wish to aggregate and highlight results from different approaches tackling the problem of fine grained 3d object identification via language if you use either of our datasets with a new method please let us know so we can add your method and attained results in our benchmark aggregating page acknowledgements the authors wish to acknowledge the support of a vannevar bush faculty fellowship a grant from the samsung gro program and the stanford sail toyota research center nsf grant iis 1763268 kaust grant bas 1 1685 01 01 and a research gift from amazon web services they also want to thank iro armeni angel x chang and jiayun wang for inspiring discussions and their help in bringing this project to fruition last but not least they want to express their gratitude to the wonderful turkers of amazon mechanical turk whose help in curating the introduced datasets was paramount