Meta tags:
description=SAFE: Multitask Failure Detection for Vision-Language-Action Models;
keywords=SAFE, Vision-Language-Action Model, VLA, Failure Detection, Failure Estimation, Robotics, AI;

Headings (most frequently used words):

failure, detection, for, safe, multitask, models, vla, the, and, failures, detector, how, do, vision, language, action, latent, feature, analysis, experiments, neural, information, processing, systems, neurips, 2025, we, introduce, problem, propose, that, can, detect, unseen, tasks, zero, shot, achieve, state, of, art, performance, abstract, bibtex, well, detectors, distinguish, from, successes, accuracy, time, trade, off, using, functional, cp, are, detected, aligned, with, human, intuition,

Text of the page (most frequently used words):
the (76), failure (44), and (42), safe (20), #detection (15), robot (14), that (13), success (12), vla (11), are (10), tasks (10), for (9), models (8), #detector (8), this (7), multitask (7), scores (7), time (7), from (6), task (6), following (6), with (6), different (6), based (6), feature (6), features (6), action (5), while (5), threshold (5), conformal (5), prediction (5), single (5), website (4), code (4), which (4), vision (4), language (4), information (4), gets (4), stuck (4), when (4), functional (4), failures (4), can (4), accuracy (4), using (4), performance (4), internal (4), unseen (4), vlas (4), you (3), fails (3), grasp (3), stop (3), exhibits (3), behavior (3), state (3), predicted (3), varying (3), level (3), trade (3), off (3), figure (3), uncertainty (3), roc (3), computed (3), score (3), best (3), experiments (3), calibration (3), diverse (3), baselines (3), latent (3), propose (3), such (3), across (3), environments (3), institute (3), ask (2), nerfies (2), source (2), under (2), qiao (2), yuanliang (2), sun (2), shengxiang (2), gilitschenski (2), igor (2), nishimura (2), haruki (2), itkina (2), masha (2), shkurti (2), florian (2), neural (2), processing (2), systems (2), repeatedly (2), carrot (2), multiple (2), attempts (2), note (2), increasing (2), after (2), finishes (2), picking (2), signal (2), subsequently (2), dangerous (2), show (2), few (2), successful (2), failed (2), rollouts (2), red (2), alert (2), has (2), gives (2), between (2), how (2), auc (2), metric (2), results (2), all (2), detectors (2), real (2), world (2), fast (2), openvla (2), also (2), stac (2), only (2), sample (2), consistency (2), embedding (2), distribution (2), token (2), same (2), out (2), set (2), predicts (2), these (2), vector (2), scalar (2), indicating (2), likelihood (2), find (2), shown (2), generalize (2), high (2), about (2), have (2), achieve (2), novel (2), policies (2), trained (2), detect (2), paper (2), introduce (2), problem (2), both (2), art (2), uoft (2), means, free, borrow, just, link, back, footer, please, remember, remove, analytics, included, header, not, want, your, adapted, templates, licensed, creative, commons, attribution, sharealike, international, license, article, gu2026safe, title, author, journal, advances, volume, pages, 40041, 40076, year, 2026, bibtex, despite, handle, lid, attempting, triggering, misses, insertion, attempt, unstable, place, bowl, unexpected, knocks, down, tomato, sauce, grasping, its, initial, alphabet, soup, raises, example, together, green, shaded, region, indicates, determined, video, frames, border, mean, been, raised, step, detected, aligned, human, intuition, significance, used, control, conservativeness, plot, balanced, tpr, tnr, against, average, values, llm, quantification, literature, report, area, curve, max, each, rollout, averages, over, possible, thresholds, reflecting, overall, regardless, selection, table, highlighted, respectively, orange, second, well, distinguish, successes, conduct, benchmarks, franka, simplerenv, libero, adopt, uses, per, timestep, actions, sampled, inconsistency, among, samples, distances, wise, probability, entropy, evaluate, use, framework, determine, determines, hold, exceeds, during, testing, confidently, detects, deployment, sequentially, processes, mlp, lstm, backbone, layers, reduce, overfitting, improve, generalization, learning, extracts, last, layer, model, ablate, ways, extracting, aggregate, them, into, extraction, above, observation, learns, main, components, below, failing, even, though, fall, motivates, efficient, zone, capture, general, analysis, promising, robotic, behaviors, manipulation, they, limited, rates, deployed, box, allow, safely, interact, their, need, timely, backtrack, help, however, existing, tested, one, specific, require, generalist, analyze, space, sufficient, knowledge, generic, insight, design, learn, predict, evaluated, compatible, policy, architectures, test, simulated, extensively, compare, achieves, abstract, zero, shot, arxiv, toyota, research, tri, robotics, university, toronto, neurips, 2025,

Text of the page (random words):
safe multitask failure detection for vision language action models safe multitask failure detection for vision language action models neural information processing systems neurips 2025 qiao gu 1 2 3 yuanliang ju 1 2 3 shengxiang sun 1 2 igor gilitschenski 1 2 3 haruki nishimura 4 masha itkina 4 florian shkurti 1 2 3 1 university of toronto uoft 2 uoft robotics institute 3 vector institute 4 toyota research institute tri paper arxiv code we introduce the multitask failure detection problem for vla models and propose safe a failure detector that can detect failures for unseen tasks zero shot and achieve state of the art performance abstract while vision language action models vlas have shown promising robotic behaviors across a diverse set of manipulation tasks they achieve limited success rates when deployed on novel tasks out of the box to allow these policies to safely interact with their environments we need a failure detector that gives a timely alert such that the robot can stop backtrack or ask for help however existing failure detectors are trained and tested only on one or a few specific tasks while vlas require the detector to generalize and detect failures also in unseen tasks and novel environments in this paper we introduce the multitask failure detection problem and propose safe a failure detector for generalist robot policies such as vlas we analyze the vla feature space and find that vlas have sufficient high level knowledge about task success and failure which is generic across different tasks based on this insight we design safe to learn from vla internal features and predict a single scalar indicating the likelihood of task failure safe is trained on both successful and failed rollouts and is evaluated on unseen tasks safe is compatible with different policy architectures we test it on openvla π 0 and π 0 fast in both simulated and real world environments extensively we compare safe with diverse baselines and show that safe achieves state of the art failure detection performance and the best trade off between accuracy and detection time using conformal prediction vla latent feature analysis we find that the vla s internal features capture high level information about task success and failure and such information is general across different tasks as shown in the figure below when a vla is failing even though from different tasks the features fall in the same failure zone this motivates safe an efficient multitask failure detector that is based on vla internal features and can generalize to unseen tasks safe multitask failure detector for vla models based on the above observation we propose safe a failure detector that learns from vla internal features and predicts a single scalar indicating the likelihood of task failure safe has 3 main components feature extraction safe extracts the latent feature from the last layer of a vla model in experiments we ablate different ways of extracting features and aggregate them into a single feature vector learning failure detector safe sequentially processes the latent feature and predicts a failure score using an mlp or an lstm backbone these models are of 1 or 2 layers to reduce overfitting and improve generalization calibration and deployment safe determines a time varying threshold using functional conformal prediction cp on a hold out calibration set if the predicted score exceeds the threshold during testing safe confidently detects a failure experiments we evaluate the following diverse baselines all the baselines use the same conformal prediction framework as safe to determine the time varying threshold token uncertainty failure scores are computed based on token wise uncertainty probability and entropy embedding distribution failure scores are computed based on the embedding distances to the calibration distribution sample consistency multiple actions are sampled and failure scores are the inconsistency among the samples action consistency we adopt stac scores and also stac single that only uses a single sample per timestep we conduct experiments on openvla π 0 and π 0 fast vla models on libero simplerenv benchmarks and a real world franka robot how well do failure detectors distinguish failures from successes following the llm uncertainty quantification literature we report the area under the roc curve roc auc metric in the following figure roc auc results are computed based on the max predicted failure score in each rollout this metric averages the performance over all possible thresholds reflecting the overall failure detection performance regardless of threshold selection in the following table the best and second best results are highlighted in red and orange respectively how do detection accuracy and detection time trade off using functional cp by varying the significance level α used in functional conformal prediction cp we can control the conservativeness of failure detection which gives a trade off between detection accuracy and detection time in the following figure we plot the balanced accuracy tpr tnr 2 against the average detection time for different α values are the detected failures aligned with human intuition in the following we show a few example successful and failed rollouts together with the failure scores predicted by safe the green shaded region indicates the failure detection threshold determined by functional conformal prediction video frames with red border mean the a failure alert has been raised at that time step success failure when the robot gets stuck while picking up alphabet soup it raises a failure signal success failure the robot gets stuck in its initial state success failure the robot knocks down the tomato sauce and fails grasping subsequently exhibits dangerous behavior success failure when the robot attempts to place the bowl it exhibits unexpected dangerous behavior success failure the robot misses the insertion attempt and subsequently exhibits unstable behavior success failure the robot gets stuck while attempting to grasp triggering a failure signal success failure the robot repeatedly fails to grasp the carrot success note that failure scores stop increasing after the robot finishes the task failure the robot gets stuck while picking up the handle of the lid success note that failure scores stop increasing after the robot finishes the task failure the robot repeatedly fails to grasp the carrot despite multiple attempts bibtex article gu2026safe title safe multitask failure detection for vision language action models author gu qiao and ju yuanliang and sun shengxiang and gilitschenski igor and nishimura haruki and itkina masha and shkurti florian journal advances in neural information processing systems volume 38 pages 40041 40076 year 2026 this website adapted from the nerfies templates which is licensed under a creative commons attribution sharealike 4 0 international license this means you are free to borrow the source code of this website we just ask that you link back to the nerfies source code in the footer please remember to remove the analytics code included in the header of the website which you do not want on your website