Meta tags:
description=Describe Anything: Detailed Localized Image and Video Captioning;

Headings (most frequently used words):

captioning, detailed, dlc, localized, describe, anything, and, dam, image, video, model, of, for, sdp, bench, iccv, 2025, citation, introducing, the, highly, instruction, controlled, zero, shot, regional, qa, architecture, semi, supervised, data, pipeline, benchmark, advantages, comparison, conclusion,

Text of the page (most frequently used words):
the (82), and (79), with (40), detailed (28), dam (27), its (25), #localized (24), model (24), #captioning (23), descriptions (19), dlc (19), image (18), for (18), our (17), from (16), describe (14), video (14), caption (14), anything (13), region (12), data (11), bench (11), this (10), can (10), slightly (10), models (9), specified (8), benchmark (8), that (8), context (8), credit (7), llm (7), regions (7), images (7), accurate (7), features (7), legs (7), videos (6), based (6), both (6), global (6), cow (6), has (6), monkey (6), design (5), previous (5), are (5), dataset (5), captions (5), more (5), rich (5), details (5), not (5), masks (5), user (5), sequence (5), long (4), liu (4), use (4), zero (4), level (4), provide (4), hallucination (4), reference (4), detail (4), attributes (4), supervised (4), quality (4), regional (4), into (4), boxes (4), focal (4), brown (4), coat (4), movements (4), forward (4), body (4), black (4), food (4), goldfish (4), such (3), citation (3), annotation (3), tasks (3), support (3), shot (3), method (3), language (3), open (3), trained (3), videorefer (3), description (3), outperforms (3), only (3), specific (3), new (3), without (3), relying (3), query (3), datasets (3), semi (3), learning (3), high (3), human (3), pipeline (3), sdp (3), points (3), introduce (3), like (3), training (3), using (3), generate (3), architecture (3), about (3), users (3), intricate (3), subtle (3), patterns (3), tag (3), appearance (3), scribbles (3), tail (3), each (3), sense (3), motion (3), back (3), throughout (3), composed (3), vertical (3), iris (3), light (3), white (3), out (3), dog (3), project (2), lian (2), yifan (2), ding (2), yunhao (2), sifei (2), hanzi (2), mao (2), boyi (2), marco (2), pavone (2), ming (2), trevor (2), darrell (2), adam (2), yala (2), yin (2), cui (2), arxiv (2), 2025 (2), generates (2), various (2), component (2), code (2), future (2), research (2), see (2), will (2), area (2), evaluation (2), achieves (2), average (2), relative (2), improvement (2), short (2), metrics (2), best (2), paco (2), class (2), excels (2), particularly (2), objects (2), parts (2), keyword (2), performance (2), thinking (2), mode (2), second (2), proposed (2), existing (2), producing (2), general (2), those (2), pred (2), scorer (2), positive (2), correct (2), scalable (2), diverse (2), unlabeled (2), question (2), referring (2), curation (2), vlm (2), vision (2), backbone (2), prompt (2), advantages (2), then (2), judge (2), questions (2), objects365 (2), uses (2), evaluate (2), approach (2), large (2), aware (2), full (2), target (2), beyond (2), interactive (2), cases (2), varying (2), style (2), sam (2), materials (2), also (2), describing (2), how (2), over (2), time (2), paper (2), your (2), browser (2), does (2), frames (2), generating (2), within (2), task (2), which (2), below (2), initially (2), seen (2), head (2), calm (2), progresses (2), steady (2), rhythmic (2), indicating (2), relaxed (2), posture (2), sturdy (2), maintains (2), rectangular (2), featuring (2), series (2), structure (2), creating (2), predominantly (2), giving (2), eye (2), thick (2), gray (2), darker (2), face (2), holding (2), interacts (2), closer (2), mouth (2), hands (2), expression (2), eyes (2), closed (2), balance (2), fluid (2), cat (2), orange (2), ears (2), ornate (2), railing (2), floral (2), symmetrical (2), horizontal (2), metal (2), table (2), flat (2), base (2), four (2), chair (2), fin (2), fins (2), pointed (2), page, inspired, academic, pages, grounded, diffusion, article, lian2025describe, title, author, journal, preprint, 2504, 16072, year, you, work, find, helpful, please, consider, citing, utilized, applications, ranging, serving, intermediate, downstream, plan, publicly, release, excited, community, explore, potential, hope, serve, useful, resource, conclusion, ref, respectively, flickr30k, entities, against, phrase, lvis, benchmarks, challenging, requires, distinguishing, between, analysis, scores, quantitative, results, sections, domain, 700k, regard, sourcing, panda, 70m, stvg, api, source, vlms, underlined, accuracies, solutions, less, surpasses, designed, reasoning, setting, standard, comparison, assessment, negative, incorrect, penalty, present, similarity, web, scale, limited, fully, leverage, precise, annotated, keywords, reframe, referred, expansion, mask, imprecise, ssl, contextful, allowing, multi, granular, providing, already, lost, feature, extraction, provided, extracting, solution, problem, practice, prompted, generated, evaluated, querying, assigned, deducted, response, show, example, instead, simple, text, overlap, checks, absence, errors, offers, metric, measuring, because, lack, devised, two, stage, first, expand, labels, segmentation, apply, self, form, refine, builds, extensive, integrates, aligned, spatially, gated, cross, attention, layers, fuse, local, cues, parameters, initialized, preserving, pre, capabilities, yields, richer, zoomed, view, ensures, sees, fine, while, retaining, result, reflect, bigger, picture, smallest, nuances, answer, extra, ask, draws, understanding, driven, answers, capability, enhances, natural, according, instructions, guide, produce, whether, brief, summary, narrative, needed, adapt, output, flexibility, benefits, rapid, labeling, depth, expert, analyses, instruction, controlled, compared, prior, works, balancing, clarity, highlight, changing, textures, far, what, provides, highly, perform, changes, specifying, sufficient, mose, single, frame, extends, naturally, change, must, track, across, capturing, evolving, interactions, transformations, performs, accepts, inputs, specification, including, clicks, markus, trienke, comprehensive, unlike, traditional, summarizes, entire, scene, broad, strokes, dives, deeper, finer, goal, capture, object, name, category, but, texture, color, shape, notable, any, visually, distinctive, powerful, multimodal, specify, contextual, watch, intro, learn, introducing, lighter, patch, rump, depicted, lowered, suggesting, demeanor, begins, move, extending, gait, tufted, end, sways, gently, step, adding, fluidity, remains, mostly, upright, arched, well, defined, carry, purpose, consistent, pace, smooth, unhurried, embodying, serene, presence, modern, skyscraper, sleek, evenly, spaced, windows, building, stepped, section, smaller, than, one, tiered, effect, facade, reflective, glass, panels, contemporary, polished, dark, almost, pupil, surrounded, irregularly, shaped, patches, mottled, outer, edge, jagged, uneven, dynamic, appears, reaching, container, right, hand, piece, yellow, hunched, focus, intent, brings, manipulate, facial, suggests, concentration, enjoyment, partially, shifts, maintaining, continues, eat, purposeful, showcasing, dexterity, agility, final, depict, bringing, lowering, possibly, preparing, take, another, bite, pink, nose, soft, fur, balcony, wrought, iron, geometric, includes, motifs, central, flanked, scrollwork, bars, grid, decorative, elements, round, green, circular, top, cylindrical, band, around, middle, supported, blue, wooden, straight, backrest, three, slats, seat, vibrant, striking, pattern, spots, glides, gracefully, through, water, elongated, streamlined, prominent, dorsal, arches, elegantly, along, fans, displaying, delicate, translucent, ripples, movement, swims, undulates, smoothly, propelling, gentle, pectoral, flutter, subtly, aiding, navigation, curious, expressive, seems, observe, surroundings, keen, interest, journey, exuding, elegance, aquatic, environment, medium, sized, reddish, underbelly, bushy, red, collar, silver, showing, teeth, tongue, hanging, running, pose, front, extended, stretched, behind, marked, demo, nvidia, berkeley, ucsf, iccv, overview,

Text of the page (random words):
describe anything detailed localized image and video captioning overview dlc task architecture data benchmark citation describe anything detailed localized image and video captioning iccv 2025 long lian 1 2 yifan ding 1 yunhao ge 1 sifei liu 1 hanzi mao 1 boyi li 1 2 marco pavone 1 ming yu liu 1 trevor darrell 2 adam yala 2 3 yin cui 1 1 nvidia 2 uc berkeley 3 ucsf paper video interactive demo code models data benchmark citation tl dr our describe anything model dam generates detailed descriptions for user specified regions in images and videos marked by points boxes scribbles or masks we introduce dlc bench to evaluate such region based descriptions caption from dam a medium sized dog with a thick reddish brown coat and a white underbelly and legs the dog has a bushy tail pointed ears and a red collar with a silver tag its mouth is open showing its teeth and its tongue is hanging out the dog is in a running pose with its front legs extended forward and its back legs stretched out behind caption from dam a vibrant goldfish with a striking pattern of orange and black spots glides gracefully through the water its body is elongated and streamlined with a prominent dorsal fin that arches elegantly along its back the tail fin fans out displaying a delicate translucent quality that ripples with each movement as it swims the goldfish s body undulates smoothly propelling it forward with a gentle rhythmic motion the fins particularly the pectoral fins flutter subtly aiding in navigation and balance the goldfish s head is slightly pointed with a curious expressive eye that seems to observe its surroundings with keen interest throughout its journey the goldfish maintains a steady fluid motion exuding a sense of calm and elegance as it interacts with the aquatic environment caption from dam a blue wooden chair with a straight backrest featuring three vertical slats the seat is flat and rectangular and the chair has four sturdy legs caption from dam a round green metal table with a flat circular top the table has a cylindrical base with a horizontal black band around its middle the base is supported by four black metal legs caption from dam the balcony features an ornate wrought iron railing with intricate floral and geometric patterns the design includes symmetrical motifs with a central floral design flanked by symmetrical ornate scrollwork the railing is composed of vertical and horizontal bars creating a grid like structure with decorative elements caption from dam a white cat with light orange ears and a pink nose the cat has a relaxed expression with slightly closed eyes and a soft white fur coat caption from dam a monkey with a light brown coat and a slightly darker face is seen in a series of dynamic movements initially it appears to be reaching into a container with its right hand which is holding a piece of yellow food the monkey s posture is slightly hunched indicating focus and intent as it interacts with the food as the sequence progresses the monkey brings the food closer to its mouth using both hands to manipulate it its facial expression suggests concentration and enjoyment with its eyes partially closed the monkey s body shifts slightly maintaining balance as it continues to eat throughout the sequence the monkey s movements are fluid and purposeful showcasing its dexterity and agility the final frames depict the monkey holding the food with both hands bringing it closer to its face and then lowering it slightly possibly preparing to take another bite caption from dam the eye has a dark almost black pupil surrounded by a thick irregularly shaped iris the iris is predominantly light gray with patches of darker gray and brown giving it a mottled appearance the outer edge of the iris is slightly jagged and uneven caption from dam a modern skyscraper with a sleek rectangular design featuring a series of vertical evenly spaced windows the building has a stepped structure with each section slightly smaller than the one below it creating a tiered effect the facade is predominantly composed of reflective glass panels giving it a contemporary and polished appearance caption from dam a cow with a rich brown coat and a lighter patch on its rump is depicted in a sequence of movements initially the cow is seen with its head slightly lowered suggesting a calm demeanor as the sequence progresses the cow begins to move forward its legs extending in a steady rhythmic gait the tail with its tufted end sways gently with each step adding a sense of fluidity to its motion the cow s body remains mostly upright with its back slightly arched indicating a relaxed posture the legs sturdy and well defined carry the cow forward with a sense of purpose throughout the sequence the cow maintains a consistent pace its movements smooth and unhurried embodying a serene and composed presence introducing the describe anything model dam the describe anything model dam is a powerful multimodal large language model that can generate detailed descriptions for specific regions in images or videos users can specify regions using points boxes scribbles or masks and dam will provide rich contextual descriptions of those regions watch our intro video below to learn more about dam your browser does not support the video tag detailed localized captioning dlc detailed localized captioning dlc is the task of generating comprehensive and context aware descriptions of specific regions within an image unlike traditional image captioning which summarizes the entire scene in broad strokes dlc dives deeper into the finer details of a user specified area the goal is to capture not only the object s name or category but also subtle attributes such as texture color patterns shape notable parts and any visually distinctive features describe anything model dam performs detailed localized captioning dlc generating detailed and localized descriptions for user specified regions within images dam accepts various user inputs for region specification including clicks scribbles boxes and masks image credit markus trienke cc by sa 2 0 dlc extends naturally to videos by describing how a specified region s appearance and context change over time models must track the target across frames capturing evolving attributes interactions and subtle transformations your browser does not support the video tag describe anything model dam can also perform detailed localized video captioning describing how a specified region changes over time for localized video descriptions specifying the region on only a single frame is sufficient see our paper for full descriptions video credit mose dataset cc by nc sa 4 0 highly detailed image and video captioning our method excels at producing detailed descriptions of objects in both images and videos by balancing the clarity of a focal region with global context the model can highlight subtle features like intricate patterns or changing textures far beyond what general image level captioning provides compared to prior works the description from our describe anything model dam is more detailed and accurate image credit sam materials cc by sa 4 0 instruction controlled captioning users can guide our model to produce descriptions of varying detail and style whether a brief summary or a long intricate narrative is needed the model can adapt its output this flexibility benefits diverse use cases from rapid labeling tasks to in depth expert analyses describe anything model dam can generate descriptions of varying detail and style according to user instructions image credit sam materials cc by sa 4 0 zero shot regional qa beyond descriptions our model can answer questions about a specified region without extra training data users can ask about the region s attributes and the model draws on its localized understanding to provide accurate context driven answers this capability enhances natural interactive use cases architecture of describe anything model dam our architecture uses a focal prompt to provide both the full image and a zoomed in view of the target region this approach ensures the model sees fine details while retaining global context the result is detailed accurate captions that reflect both the bigger picture and the smallest nuances image credit objects365 dataset cc by 4 0 we introduce a localized vision backbone that integrates global and focal features images and masks are aligned spatially and gated cross attention layers fuse detailed local cues with global context new parameters are initialized to zero preserving pre trained capabilities this design yields richer more context aware descriptions semi supervised data pipeline for detailed localized captioning dlc sdp because existing datasets lack detailed localized descriptions we devised a two stage pipeline first we use a vlm to expand short class labels from segmentation datasets into rich descriptions second we apply self training as a form of semi supervised learning on unlabeled images using our model to generate and refine new captions this scalable approach builds large high quality training data without relying on extensive human annotation dlc bench a benchmark for detailed localized captioning we introduce dlc bench a benchmark that uses an llm based judge to evaluate a model s region based descriptions instead of relying on simple text overlap dlc bench checks for correct details and the absence of errors this offers a more accurate and human like metric for measuring dlc performance in dlc bench a captioning model is prompted to describe a specified image region the generated description is then evaluated by querying an llm judge points are assigned or deducted based on the llm s response the question we show is an example of positive questions image credit objects365 dataset cc by 4 0 advantages of dam dlc sdp and dlc bench component previous practice problem our solution advantages describe anything model dam extracting regional features from global image features regional details already lost in image feature extraction and not provided to the llm providing focal prompt to proposed localized vision backbone detail rich contextful features allowing for accurate multi granular localized descriptions ssl data pipeline dlc sdp query a data curation vlm with referring boxes and global image captions imprecise referring to data curation model reframe the query into a mask referred keyword expansion question leverage high quality precise human annotated regional masks and keywords fully supervised learning limited data with high annotation quality semi supervised learning scalable to diverse web scale unlabeled datasets benchmark dlc bench pred caption reference gt caption language based similarity metrics or llm scorer incorrect hallucination penalty for correct details not present in the reference caption pred caption query for positive negative attributes llm scorer accurate detail and hallucination assessment without relying on reference captions comparison on dlc bench our model outperforms existing solutions by producing more detailed and accurate localized descriptions with less hallucination it surpasses models trained for general image level tasks and those designed for localized reasoning setting a new standard for detailed context rich captioning accuracies on detailed localized captioning in our proposed dlc bench dam outperforms previous api only models open source models and region specific vlms on detailed localized captioning underlined the second best method models with thinking mode detailed localized video captioning on hc stvg dam outperforms previous models on detailed localized video captioning models with thinking mode performance on detailed localized video description on videorefer bench d we provide analysis on hallucination scores hd in our quantitative results and reference captions sections trained on in domain videorefer 700k with regard to videorefer bench both sourcing videos from panda 70m lvis and paco open class keyword level captioning benchmarks dam excels particularly in the challenging paco benchmark that requires distinguishing between objects and parts zero shot evaluation on phrase level dataset flickr30k entities our model achieves 7 34 average relative improvement against previous best zero shot evaluation on the detailed captioning dataset ref l4 our method achieves 39 5 and 13 1 average relative improvement on the short long language based captioning metrics respectively conclusion our describe anything model dam generates detailed descriptions for specified regions in images and videos and can be utilized in various applications ranging from data annotation to serving as an intermediate component in downstream tasks we plan to publicly release our code models data and benchmark to support future research we are excited to see the community explore the potential of detailed localized captioning we hope this benchmark and model will serve as a useful resource for future research in this area citation if you use this work or find it helpful please consider citing article lian2025describe title describe anything detailed localized image and video captioning author long lian and yifan ding and yunhao ge and sifei liu and hanzi mao and boyi li and marco pavone and ming yu liu and trevor darrell and adam yala and yin cui journal arxiv preprint arxiv 2504 16072 year 2025 credit the design of this project page is inspired by previous academic project pages such as llm grounded diffusion