Meta tags:

Headings (most frequently used words):

boundary, boundaries, attention, to, junction, space, output, of, jump, model, details, overview, representing, per, patch, in, training, data, emergent, properties, generalization, natural, images, low, light, detection, evolution, spatial, affinities, embedding, bibtex, contact, info, visual, comparison, more, examples,

Text of the page (most frequently used words):
and (48), the (45), boundaries (29), boundary (19), model (17), that (14), #junction (14), output (14), its (13), are (13), our (12), #attention (11), patch (11), for (10), input (10), map (10), bottom (9), with (9), spatial (9), junctions (9), windowing (9), can (8), each (8), noise (8), any (7), space (7), this (7), affinities (7), distance (7), local (7), resolution (6), pixel (6), image (6), colors (6), field (6), images (6), find (5), embeddings (5), times (5), these (5), unsigned (5), global (5), high (5), fine (5), structures (5), data (5), function (5), also (4), embedding (4), smooth (4), top (4), row (4), from (4), point (4), gamma (4), mathbb (4), way (4), they (4), during (4), iterations (4), patches (4), low (4), functions (4), training (4), trained (4), synthetic (4), end (4), size (4), train (4), parameters (4), corners (4), learning (3), arxiv (3), bibtex (3), learned (3), figure (3), between (3), neighbors (3), ones (3), their (3), overlapping (3), them (3), every (3), both (3), simple (3), natural (3), coarse (3), properties (3), those (3), which (3), details (3), variety (3), including (3), three (3), partitions (3), harvard (2), contact (2), info (2), polansky (2), mia (2), gaia (2), herrmann (2), charles (2), hur (2), junhwa (2), sun (2), deqing (2), verbin (2), dor (2), zickler (2), todd (2), comparison (2), shows (2), interpolation (2), delta (2), learns (2), angles (2), vertex (2), visualize (2), through (2), compute (2), query (2), sum (2), implied (2), spatially (2), evolution (2), implies (2), over (2), slice (2), non (2), linearity (2), call (2), unrasterized (2), thickness (2), parameter (2), right (2), produces (2), levels (2), sensor (2), windows (2), but (2), more (2), window (2), light (2), detection (2), smoothed (2), features (2), human (2), annotations (2), very (2), geometric (2), across (2), varying (2), amounts (2), classical (2), methods (2), based (2), color (2), allows (2), produce (2), emergent (2), precision (2), operations (2), iteratively (2), using (2), specialized (2), neighborhood (2), text (2), show (2), order (2), represent (2), patterns (2), bars (2), visualized (2), partition (2), stride (2), parametric (2), maps (2), aware (2), smoothing (2), overview (2), form (2), all (2), associated (2), introduce (2), infers (2), contours (2), dataset (2), code (2), always, happy, chat, miapolansky, edu, copy, article, mia2023boundaries, author, title, faint, journal, year, 2023, visualizes, equally, spaced, samples, euclidean, sense, gamma_i, starting, gamma_a, ending, gamma_b, comparable, analytically, designed, interestingly, associate, nearly, equal, close, center, another, pairwise, imply, affinity, normalized, segments, contain, respect, serve, filtering, kernels, convert, noisy, probe, intermediate, exploratory, unstructured, early, become, consistent, later, thicker, thinner, upsampled, naïve, bilinear, windowed, surrounding, get, simply, wise, average, applying, amplify, zero, set, definition, rendered, below, render, super, left, change, adjusting, canny, sigma, structured, forests, pidinet, hed, edter, ours, crisp, photographs, success, part, due, inferred, general, smaller, good, recovering, situations, cause, false, conversely, larger, provide, resilience, cannot, recover, automatically, inferring, combines, benefits, distances, examples, down, visual, despite, being, provides, reasonable, combinations, quite, stable, exposure, conditions, have, qualitative, somewhat, different, models, match, purely, include, finer, than, systems, inference, sizes, unlike, many, use, single, everywhere, generalization, predict, true, visible, known, machine, sufficient, consisting, circles, squares, shape, uniformly, colored, augmented, perturbing, types, uses, eight, total, some, weight, sharing, includes, two, free, gather, perform, rasterizations, foldings, specific, entire, invariant, discrete, shifts, applies, inputs, fairly, compact, only, 207k, vector, fields, updating, variant, dot, product, simultaneously, updates, embedded, method, associates, how, modulated, mathbf, p_1, p_2, p_3, convex, weights, dictionary, binary, pillboxes, modification, vary, according, context, walks, uniformity, absence, edges, moving, slider, bar, describe, referred, comprising, orientation, theta, relative, omega_1, omega_2, omega_3, representing, per, level, special, unfolds, into, operates, partitioning, defines, effective, shown, clockwise, jump, core, self, useful, state, suggests, has, inductive, bias, since, small, then, deploy, aspect, ratio, outputs, you, scroll, segment, various, accumulations, lead, lightweight, represented, encode, express, elements, thin, expresses, without, rasterization, unlimited, unfolded, google, research, university, paper, curves, grouping, misc, links, introduction, return, navigate,

Text of the page (random words):
boundary attention navigate return to top 1 introduction 2 model details a overview b junction space c boundary attention d training data 3 emergent properties a natural images b low light boundary detection c output boundaries d evolution of boundaries e output spatial affinities f embedding of junction space 4 misc links arxiv code dataset bibtex contact info boundary attention learning curves corners junctions and grouping paper arxiv code dataset bibtex mia gaia polansky 1 2 charles herrmann 1 junhwa hur 1 deqing sun 1 dor verbin 1 todd zickler 1 1 google research 2 harvard university we introduce a form of local attention that infers unrasterized boundaries including contours corners and junctions from the bottom up input smooth colors boundaries spatial affinities unfolded input patches partitions colors partitions boundaries windowing functions we introduce a lightweight bottom up model that infers color based boundaries with high precision output boundaries are represented by a field of embeddings that encode three way partitions and associated windowing functions of every stride 1 patch in an image this output can express a variety of boundary elements including contours thin bars corners t junctions and y junctions it expresses them without rasterization and so with unlimited resolution the figure s bottom row shows these outputs as you scroll with each patch s partition visualized by its boundaries its segment colors and its associated windowing function various global accumulations of these overlapping patches lead to the pixel resolution output maps in the figure s top row a boundary aware smoothing of the input colors a global boundary map and the spatial affinities between each pixel and all of its neighbors the core of our model is a specialized form of neighborhood self attention that we call boundary attention we find that we can train it to a useful state using very simple synthetic images which suggests it has an inductive bias for boundaries also since all of its operations are local and bottom up we can train it on small images and then deploy it at any image size and aspect ratio jump to model details overview at a high level our model learns a field of special geometric embeddings for every patch in an image the input image unfolds into stride 1 patches and boundary attention operates iteratively on their embeddings to produce for each patch bf i a parametric three way partitioning and bf ii a parametric windowing function that defines its effective patch size this output field implies a variety of global maps shown in clockwise order a boundary aware smoothing of the input colors an unsigned boundary distance map a boundary map and a map of spatial affinities between any query point and its neighbors representing per patch boundaries in junction space we describe each patch s three way partition referred to as a junction by parameters g in mathbb r 2 times mathbb s 1 times delta 2 comprising a vertex u v orientation theta and relative angles omega_1 omega_2 omega_3 that sum to 2 pi walks in junction space are spatially smooth and can represent a variety of local boundary patterns including uniformity i e absence of boundaries edges bars corners t junctions and y junctions these local boundary patterns can be visualized by moving the slider bar in text a to the right we show each junction s implied unsigned distance map our method also associates each junction with a learned spatial windowing function in text b we show how a junction is modulated through its windowing function the windowing parameters mathbf p p_1 p_2 p_3 are convex weights over a dictionary of binary pillboxes this modification allows our model to vary boundaries according to local context in order to represent both fine and coarse details boundary attention our model produces the output vector fields by learning an embedding gamma of junction parameters g and iteratively updating the pixel resolution field of these embeddings gamma n using a specialized variant of neighborhood dot product attention it simultaneously updates a field of embedded windowing functions pi n the model uses eight iterations of boundary attention in total with some weight sharing across iterations it includes two parameter free operations gather and slice that perform rasterizations and foldings which are specific to junctions the entire model is invariant to discrete spatial shifts and so applies to inputs of any size it is also fairly compact with only 207k parameters training data we find it sufficient to train the model on simple synthetic data consisting of overlapping circles and squares each shape is uniformly colored and the images are augmented during training by perturbing them with varying types and amounts of noise we train our model to predict the unsigned distance function for the true visible boundaries which is known up to machine precision emergent properties generalization to natural images despite being trained on very simple synthetic data the model provides reasonable boundaries for natural images the boundaries are combinations of fine geometric structures and coarse ones and they are quite stable across exposure conditions that have varying amounts of sensor noise the qualitative properties of our model s boundaries are somewhat different from those of classical bottom up methods and from those of learned end to end models that are trained to match human annotations its boundaries are based purely on color and so include finer structures than those of end to end systems and its inference of local window sizes allows it to produce both fine and coarse structures unlike many classical bottom up methods that use a single patch size everywhere visual comparison top down trained on human annotations bottom up trained on synthetic data bottom up no training more examples input distances boundaries smoothed features input boundaries smoothed features low light boundary detection our model produces crisp boundaries for photographs with high levels of sensor noise its success is in part due to the inferred local windowing functions in general smaller windows are good for recovering fine structures in low noise situations but can cause false boundaries at high noise levels conversely larger windows provide more resilience to noise but cannot recover fine structures by automatically inferring a window for every patch our model combines the benefits of both input ours edter hed pidinet structured forests field of junctions 17 canny sigma 2 low noise high noise output boundaries the output junction g n at each pixel n implies an unsigned distance map over the windowed patch surrounding it to get a global unsigned distance map for the image we simply compute the pixel wise average slice of the overlapping patches we visualize this global map by applying a non linearity to amplify its zero distance set and we call this the output boundary map this definition of output boundaries is unrasterized and so can be rendered at any resolution and thickness below we render them at super resolution left and change their thickness by adjusting our non linearity s parameter right input distance map boundaries our 2 times upsampled boundaries vs naïve bilinear interpolation thinner thicker evolution of boundaries at any point during the iterations we can probe the intermediate junctions and their implied boundaries we find they are exploratory and unstructured during early iterations and become spatially consistent during later ones output spatial affinities another way to visualize our output g n is through the pairwise affinities they imply we compute the spatial affinity between any query pixel n and its neighbors n as the normalized sum of junction segments that contain n these spatial affinities respect boundaries and they serve as the filtering kernels that convert the noisy input image colors to the output smooth ones embedding of junction space we also find that the model s learned embedding of junction space is smooth the top row of this figure visualizes equally spaced samples in the euclidean sense of embeddings gamma_i from starting point gamma_a to gamma 0 and to ending point gamma_b for comparison the bottom row shows a comparable analytically designed interpolation in junction space mathbb r 2 times mathbb s 1 times delta 2 interestingly the model learns to associate gamma 0 with nearly equal angles and a vertex close to the patch center bibtex article mia2023boundaries author polansky mia gaia and herrmann charles and hur junhwa and sun deqing and verbin dor and zickler todd title boundary attention learning to find faint boundaries at any resolution journal arxiv year 2023 copy contact info always happy to chat at miapolansky at g harvard edu