Meta tags:
description=VA2CONTACT: Visual-auditory Extrinsic Contact Estimation;

Headings (most frequently used words):

to, va2contact, visual, extrinsic, contact, active, audio, sensing, sim, real, transfer, and, auditory, estimation, icra, 2026, video, demonstration, abstract, method, key, features, results, team, occlusion, handling, xili, yi, jayjun, lee, nima, fazeli, tl, dr, integrates, accurately, detect, contacts, between, grasped, object, the, environment, even, in, occluded, or, ambiguous, scenarios, with, zero, shot,

Text of the page (most frequently used words):
#contact (30), the (21), and (20), real (18), audio (13), world (11), #contacts (9), va2contact (8), scenarios (8), with (8), for (7), visual (7), that (7), extrinsic (7), vision (6), estimation (6), our (6), sim (6), michigan (5), transfer (5), simulation (5), detect (5), hallucination (5), university (4), jayjun (4), performance (4), policy (4), only (4), zero (4), shot (4), from (4), ambiguous (4), information (4), even (4), occluded (4), method (4), technique (4), labels (4), are (4), signals (4), through (4), sensing (4), grasped (4), object (4), nima (3), lee (3), xili (3), umich (3), edu (3), department (3), robotics (3), manipulation (3), tasks (3), near (3), active (3), using (3), accurately (3), between (3), environment (3), auditory (3), project (2), about (2), fazeli (2), downstream (2), aware (2), diffusion (2), wiping (2), training (2), where (2), detection (2), methods (2), ability (2), key (2), demonstrate (2), multimodal (2), approach (2), significantly (2), data (2), robust (2), when (2), conduction (2), speakers (2), microphones (2), occlusions (2), rich (2), often (2), model (2), its (2), dense (2), maps (2), difficult (2), train (2), entirely (2), ground (2), truth (2), bridge (2), gap (2), introduce (2), acoustic (2), simulated (2), system (2), paper (2), integrates (2), video (2), coming (2), soon (2), 2025, questions, please, nfz, yixili, team, improved, task, achieved, success, rate, compared, baseline, successful, without, additional, better, insufficient, accurate, heavily, fail, higher, recall, scores, general, indicating, superior, true, fewer, false, negatives, findings, include, evaluated, variety, environments, comparing, against, im2contact, ablation, studies, experiments, outperforms, especially, challenging, results, enables, eliminating, need, while, maintaining, maintains, high, points, visually, thanks, complementary, provided, can, travel, solid, objects, occlusion, handling, uses, static, present, providing, dynamics, invisible, sensors, features, freeze, use, predictions, along, camera, input, observations, predicts, probability, map, depth, optical, flow, spectrogram, since, obtain, acoustics, overcome, challenge, simulating, collect, paired, labeled, types, teleoperated, over, various, surfaces, generate, randomly, sample, spectrograms, corresponding, their, type, bridging, addresses, challenges, combining, strengths, multiple, modalities, consists, three, main, components, work, together, localize, hinges, robot, perceive, surrounding, however, these, observe, alone, due, limited, resolution, states, this, propose, global, scene, local, cues, obtained, equips, robotic, gripper, enabling, emit, receive, external, perception, pipeline, injecting, samples, into, scenes, resulting, estimates, both, location, size, across, range, cluttered, further, explicit, prediction, improves, learning, abstract, demo, your, browser, does, not, support, tag, demonstration, code, equal, contribution, ¹university, fazeli¹, icra, 2026,

Text of the page (random words):
va2contact visual auditory extrinsic contact estimation va2contact visual auditory extrinsic contact estimation icra 2026 xili yi ¹ jayjun lee ¹ nima fazeli¹ ¹university of michigan equal contribution paper code coming soon data coming soon video demonstration your browser does not support the video tag contact estimation demo tl dr va2contact integrates visual and active audio sensing to accurately detect extrinsic contacts between a grasped object and the environment even in occluded or ambiguous scenarios with zero shot sim to real transfer abstract robust manipulation often hinges on a robot s ability to perceive extrinsic contacts contacts between a grasped object and its surrounding environment however these contacts are difficult to observe through vision alone due to occlusions limited resolution and ambiguous near contact states in this paper we propose a visual auditory method for extrinsic contact estimation that integrates global scene information from vision with local contact cues obtained through active audio sensing our approach equips a robotic gripper with contact microphones and conduction speakers enabling the system to emit and receive acoustic signals through the grasped object to detect external contacts we train our perception pipeline entirely in simulation and zero shot transfer to the real world to bridge the sim to real gap we introduce a real to sim audio hallucination technique injecting real world audio samples into simulated scenes with ground truth contact labels the resulting multimodal model accurately estimates both the location and size of extrinsic contacts across a range of cluttered and occluded scenarios we further demonstrate that explicit contact prediction significantly improves policy learning for downstream contact rich manipulation tasks method our method addresses the challenges of extrinsic contact estimation by combining the strengths of multiple sensing modalities the system consists of three main components that work together to accurately detect and localize contacts between a grasped object and the environment 1 real to sim audio hallucination since dense contact maps are difficult to obtain in the real world we train entirely in simulation using ground truth contact labels and bridge the gap to real world acoustics using an audio hallucination technique to overcome the challenge of simulating audio we introduce an audio hallucination technique in the real world we collect audio signals paired with labeled contact types using teleoperated wiping over various surfaces in simulation we generate contact maps and randomly sample real spectrograms corresponding to their type bridging real world acoustic signals and simulated labels 2 va2contact va2contact that predicts a dense contact probability map from depth optical flow and an audio spectrogram 3 real world contact aware policy we freeze the va2contact model and use its contact predictions along with camera input as observations for a diffusion policy key features active audio sensing uses conduction speakers and contact microphones to detect contacts even in static scenarios and when visual occlusions are present providing rich information about contact dynamics that is often invisible to visual sensors occlusion handling maintains high performance even when contact points are visually occluded or in ambiguous near contact scenarios thanks to the complementary information provided by audio signals that can travel through solid objects sim to real transfer our real to sim audio hallucination technique enables zero shot transfer from simulation to real world scenarios eliminating the need for real world training data with contact labels while maintaining robust performance results we evaluated va2contact on a variety of contact estimation tasks in real world environments comparing it against the vision only im2contact method and ablation studies our experiments demonstrate that our multimodal approach significantly outperforms vision only methods especially in challenging scenarios key findings include higher recall and f1 scores in general contact detection indicating superior ability to detect true contacts with fewer false negatives accurate contact detection even in heavily occluded scenarios where vision only methods fail better performance in ambiguous near contact scenarios where visual information is insufficient successful zero shot transfer from simulation to real world without additional training improved performance in downstream manipulation tasks a contact aware diffusion policy for a wiping task achieved 8 10 success rate compared to 4 10 for a vision only baseline team xili yi department of robotics university of michigan yixili umich edu jayjun lee department of robotics university of michigan jayjun umich edu nima fazeli department of robotics university of michigan nfz umich edu contact for questions about the project please contact xili yi jayjun lee or nima fazeli 2025 va2contact project university of michigan