Meta tags:

Headings (most frequently used words):

and, in, learning, deep, the, spain, herr, strathmann, two, thousand, nineteen, kernels, for, exponential, family, densities, job, opportunity, with, shogun, ati, ds3, summer, school, paris, self, organization, interpretable, discrete, representation, on, time, series, bleau, swanage, val, di, mello, eiger, mönch, jungfrau, my, phd, thesis, is, online, cracks, at, millstone, posts, navigation, coffee, climbing, jazz, math, building, van, costa, blanca, el, chorro, southern, la, pedriza, galayos, torozo, central, pyrenees, asturias, yosemite,

Text of the page (most frequently used words):
the (101), and (42), for (22), #kernel (19), with (16), 2018 (12), shogun (12), comment (11), leave (11), model (11), this (11), karlnapf (10), deep (10), are (10), one (9), that (9), paper (9), log (9), june (8), see (8), density (8), 2019 (7), using (7), two (7), gradient (7), form (7), project (6), there (6), learning (6), get (6), will (6), has (6), exponential (6), family (6), would (6), may (5), involved (5), more (5), https (5), open (5), source (5), hmc (5), gradients (5), network (5), these (5), data (5), closed (5), score (5), can (5), com (4), here (4), work (4), time (4), practical (4), testing (4), kernels (4), github (4), community (4), our (4), optional (4), etc (4), such (4), could (4), quite (4), have (4), not (4), exp (4), parameters (4), matching (4), phi (4), gaussian (4), vert (4), many (4), april (3), through (3), interested (3), fit (3), openml (3), number (3), details (3), how (3), stage (3), least (3), october (3), motivated (3), was (3), step (3), where (3), funnel (3), models (3), things (3), blog (3), post (3), half (3), use (3), loss (3), practice (3), left (3), smoothness (3), infinite (3), spain (3), climbing (3), website (2), required (2), write (2), wordpress (2), cracks (2), just (2), via (2), help (2), phd (2), cool (2), about (2), building (2), discrete (2), interpretable (2), latent (2), space (2), self (2), organising (2), series (2), myself (2), session (2), slides (2), sample (2), across (2), even (2), into (2), good (2), had (2), arthur (2), gretton (2), ds3 (2), summer (2), school (2), paris (2), numfocus (2), org (2), toolbox (2), getting (2), computing (2), library (2), machine (2), methods (2), core (2), pilot (2), extension (2), you (2), better (2), read (2), part (2), partly (2), opportunity (2), out (2), code (2), experience (2), coreml (2), able (2), start (2), working (2), based (2), perform (2), due (2), their (2), cases (2), like (2), densities (2), very (2), mathsf (2), normal (2), other (2), instead (2), auto (2), diff (2), error (2), training (2), procedure (2), first (2), coefficients (2), then (2), make (2), need (2), solution (2), well (2), descent (2), objective (2), always (2), swiss (2), army (2), knife (2), but (2), learn (2), typically (2), nice (2), guarantees (2), simple (2), single (2), sigma (2), right (2), softplus (2), some (2), invariant (2), when (2), mode (2), z_i (2), points (2), dimensional (2), assumptions (2), come (2), same (2), they (2), kevin (2), yosemite (2), pyrenees (2), pedriza (2), galayos (2), torozo (2), costa (2), blanca (2), chorro (2), van (2), hard (2), all (2), climb (2), year (2), been (2), months (2), ago (2), told (2), believed (2), november (2), coffee (2), jazz (2), math (2), herr (2), strathmann (2), name, email, loading, comments, theme, penscratch, proudly, powered, previous, posts, navigation, millstone, found, accidentally, google, scholar, increase, download, counter, thesis, online, maybe, day, eiger, mönch, jungfrau, val, mello, bleau, swanage, got, mildly, lead, along, variational, autoencoder, thus, admits, topological, neighbourhood, structure, map, represent, dynamics, lab, modelling, also, built, markov, transition, put, version, arxiv, gunnar, rätsch, francesco, locatello, matthias, hüser, vincent, fortuin, ethz, group, organization, representation, notebook, covered, implementation, basics, independence, goodness, examples, including, quality, gan, samples, detecting, dependence, translated, documents, managed, sneak, fun, overall, pleasure, run, actually, representing, comparing, probabilities, following, lecture, thanks, zoltan, szabo, polytechnique, issues, wiki, apple, coremltools, www, http, nonprofit, dedicated, supporting, scientific, aiming, offer, unified, efficient, its, written, interfaces, large, modern, languages, vibrant, diverse, international, fiscally, sponsored, contact, entrance, task, send, pull, request, currently, planning, after, successful, option, please, touch, developers, mailing, list, funded, located, supervised, development, team, person, remotely, great, oldest, libraries, your, hands, dirty, huge, base, tipping, alan, turing, institute, contributed, before, computational, sciences, stats, build, management, dev, ops, tools, git, cmake, travis, buildbot, linux, docker, knowledge, background, designing, software, gcc, valgrind, flexible, enough, spend, full, london, looking, developer, paid, month, march, improving, meta, capabilities, ideal, candidate, highly, msc, postdoc, desire, movement, who, around, job, ati, historically, used, benchmark, mcmc, algorithms, among, others, second, order, much, ability, adapt, sizes, local, scaling, something, new, wonder, arise, context, say, abc, otherwise, intractable, those, adaptive, sampler, improve, bit, girolami, 2011, mighty, naturally, worked, exact, available, whether, useful, expressive, case, comes, mind, fact, experiments, distribution, picture, prod_, x_n, stan, neal, 2003, experimentally, compared, approaches, likelihoods, normalizing, flows, results, positive, rid, sub, sampling, partitioning, which, type, wittawat, employed, conceptually, cross, validation, update, dataset, extremely, helpful, propagate, quadratic, style, seems, usual, trickery, applies, take, away, each, involves, linear, solve, solutions, reliably, algorithm, nitty, gritty, naive, doing, overfit, made, arbitrarily, moving, towards, point, making, zero, stochastic, might, indeed, produce, unstable, updates, however, while, does, original, optimizing, obviously, makes, way, complicated, whereas, grid, search, black, box, optimizer, parameter, approach, fails, locally, adapts, neural, networks, construct, non, stationary, location, dependent, top, fully, connected, nonlinearity, gives, properties, defined, functions, normalizable, proposition, spatially, corresponding, uniform, assumption, domain, although, sufficient, consistency, limit, induced, fail, finite, datasets, especially, takes, differently, scaled, shapes, different, parts, figure, illustrates, problem, fitting, mixture, gaussians, correct, bandwidths, broad, narrow, translation, must, pick, average, between, any, choice, yield, poor, consider, lite, estimator, from, sum_, alpha_ik, inducing, span, input, intuitively, means, simply, set, layered, bumps, infinitely, smooth, equal, degrees, variation, everywhere, puts, free, tackles, most, limiting, characteristics, standard, blessing, curse, allow, strong, statistical, yet, drastically, restrict, expressiveness, estimating, icml, together, dougal, sutherland, 2020, asturias, central, southern, summarise, experiences, words, firsts, achievements, failures, never, again, moments, below, pictures, finally, confronting, lurking, fear, captain, stating, goal, felt, somewhat, ridiculous, almost, joke, hindsight, did, follow, andy, kirkpatrick, advice, don, down, last, certainly, intense, roughly, stopped, converted, drove, climbed, loads, adventure, trad, sport, slabs, off, widths, granite, corner, monstrous, alpine, classics, amazing, limestone, canalizos, picos, europa, sentence, few, soon, either, six, capitan, within, thousand, nineteen, science, menu, skip, content,

Text of the page (random words):
herr strathmann coffee climbing jazz math skip to content herr strathmann coffee climbing jazz math menu blog science about two thousand and nineteen november 29 2019 november 30 2019 karlnapf leave a comment if six months ago i was told that i would climb el capitan within the year i would not have believed it first sentence of a blog post i read a few months ago had i been told i would write the same myself soon i would not have believed it either the last year and a half has certainly been intense roughly i stopped working converted a van drove to spain and climbed loads adventure trad in costa blanca sport in el chorro slabs and off widths in la pedriza granite corner cracks in galayos and torozo monstrous alpine classics in the pyrenees amazing limestone canalizos in the picos de europa and then finally yosemite confronting lurking fear vi 5 7 c2 on the captain stating this as a goal always felt somewhat ridiculous almost like a joke but in hindsight all we did was to start climbing and follow andy kirkpatrick s advice when things get hard don t come down it is hard to summarise all these experiences in words many firsts many achievements many failures and many i will never climb again moments so i will not below are some pictures building the van costa blanca and el chorro in southern spain la pedriza 3 galayos and torozo in central spain pyrenees and asturias yosemite learning deep kernels for exponential family densities june 29 2019 april 19 2020 karlnapf leave a comment an icml 2019 paper together with kevin li dougal sutherland and arthur gretton code by kevin more score matching for estimating gradients using the infinite dimensional kernel exponential family e g for gradient free hmc this paper tackles one of the most limiting practical characteristics of using the kernel infinite dimensional exponential family model in practice the smoothness assumptions that come with the standard swiss army knife gaussian kernel these smoothness assumptions are blessing and curse at the same time they allow for strong statistical guarantees yet they can quite drastically restrict the expressiveness of the model to see this consider a log density model of the form the lite estimator from our hmc paper log p x sum_ i 1 n alpha_ik x z_i for inducing points z_i the points that span the model could be e g the input data and the gaussian kernel k x y exp left vert x y vert 2 sigma right intuitively this means that the log density is simply a set of layered gaussian bumps infinitely smooth with equal degrees of variation everywhere as the paper puts it these kernels are typically spatially invariant corresponding to a uniform smoothness assumption across the domain although such kernels are sufficient for consistency in the infinite sample limit the induced models can fail in practice on finite datasets especially if the data takes differently scaled shapes in different parts of the space figure 1 left illustrates this problem when fitting a simple mixture of gaussians here there are two correct bandwidths one for the broad mode and one for the narrow mode a translation invariant kernel must pick a single one e g an average between the two and any choice will yield a poor fit on at least part of the density how can we learn a kernel that locally adapts to the density deep neural networks we construct a non stationary i e location dependent kernel using a deep network phi x on top a gaussian kernel i e k x y exp left vert phi x phi y vert 2 sigma right the network phi x is fully connected with softplus nonlinearity i e log 1 e x softplus gives us some nice properties such as well defined loss functions and a normalizable density model see proposition 2 in the paper for details however we need to learn the parameters of the network while kernel methods typically have nice closed form solution with guarantees and so does the original kernel exponential family model see my post optimizing the parameters of ϕ x ϕ x ϕ x ϕ x obviously makes things way more complicated whereas we could use a simple grid search or black box optimizer for a single kernel parameter this approach here fails due to the number of parameters in ϕ x ϕ x 1 2 p 0 x x log p x x log p 0 x 2 d x ϕ x could we use naive gradient descent doing so on our score matching objective 1 2 p 0 x x log p x x log p 0 x 2 d x with log p x n i 1 α i k z i x and k x y exp x y 2 σ will always overfit to the training data as the score gradient error loss can be made arbitrarily good by moving z i zi towards a data point and making σ go to zero stochastic gradient descent the swiss army knife of deep learning on the score matching objective might help but would indeed produce very unstable updates instead we employed a two stage training procedure that is conceptually motivated by cross validation we first do a closed form update for the kernel model coefficients α i α i α i αi on one half of the dataset then we perform a gradient step on the parameters of the deep kernel on the other half we make use of auto diff extremely helpful here as we need to propagate gradients through a quadratic form style score matching loss the closed form kernel solution and the network this seems to work quite well in practice the usual deep trickery to make it work applies take away by using a two stage procedure where each gradient step involves a closed form linear solve solutions for the kernel coefficients α i α i αi we can fit this model reliably see algorithm 1 in the paper for more nitty gritty details a cool extension of the paper would be to get rid of the sub sampling partitioning of the data and instead auto diff through the leave one out error which is closed form for these type of kernel models see e g wittawat s blog post we experimentally compared the deep kernel exponential family to a number of other approaches based on likelihoods normalizing flows etc and the results are quite positive see the paper naturally as i have worked on using these gradients in hmc where the exact gradients are not available i am very interested to see whether and how useful such a more expressive density model is the case that comes to mind and in fact motivated one of the experiments in the this paper is the funnel distribution e g neal 2003 picture by stan e g p y x mathsf normal y 0 3 prod_ n 1 9 mathsf normal x_n 0 exp y 2 the mighty funnel this density historically was used as a benchmark for mcmc algorithms among others hmc with second order gradients girolami 2011 perform much better due to their ability to adapt their step sizes to the local scaling of the density something that our new deep kernel exponential family is able to model so i wonder are there cases where such funnel like densities arise in the context of say abc or otherwise intractable gradient models for those cases an adaptive hmc sampler with the deep network kernel could improve things quite a bit job opportunity with shogun and the ati october 17 2018 june 28 2019 karlnapf leave a comment we the community around the shogun ml 1 open source machine learning library are looking for a developer for a paid 6 month pilot project october 18 march 19 on improving meta learning capabilities openml coreml the ideal candidate is a highly motivated msc phd postdoc with the desire to get involved in the open source movement who is based on london is able to start working in october is flexible enough to spend full time or at least 50 on the project has a background in designing software in c gcc valgrind c 11 etc optional has knowledge of openml 2 and coreml 3 optional has experience with build management and dev ops tools git cmake travis buildbot linux docker etc optional has experience in computational sciences ml stats etc optional has contributed to open source before the project is funded by with the alan turing institute and at least part of the work will be located there you will be supervised by the shogun core development team partly in person and partly remotely this is a great opportunity to get involved in one of the oldest ml libraries out there getting your hands dirty on a huge code base and tipping into the open source community the project is currently in planning stage after a successful pilot there is the option for an extension if you are interested please get in touch via the developers the mailing list or even better read how to get involved 4 and send us a pull request for an entrance task 5 on github see our website for contact details shogun is a library aiming to offer unified and efficient machine learning methods its core is written in c and it interfaces to a large number of modern computing languages the shogun community is vibrant diverse and international shogun is a fiscally sponsored project of numfocus a nonprofit dedicated to supporting the open source scientific computing community 1 http shogun ml 2 https www openml org 3 https github com apple coremltools 4 https github com shogun toolbox shogun wiki getting involved 5 https github com shogun toolbox shogun issues 6 https numfocus org ds3 summer school in paris june 30 2018 june 30 2018 karlnapf leave a comment i had the pleasure to run a actually two practical on representing and comparing probabilities using kernels at the ds3 summer school at the polytechnique in paris following a lecture by arthur gretton thanks to zoltan szabo for organising the session we covered the implementation basics of two sample testing independence testing and goodness of fit testing with examples including testing the quality of gan samples detecting dependence across translated documents and more i even managed to sneak shogun into the practical good fun overall slides 1 slides 2 and practical session notebook deep self organization interpretable discrete representation learning on time series june 7 2018 june 28 2019 karlnapf leave a comment i got mildly involved in a cool project with the ethz group lead by vincent fortuin and matthias hüser along with francesco locatello myself and gunnar rätsch the work is about building a variational autoencoder with a discrete and thus interpretable latent space that admits topological neighbourhood structure through using a self organising map to represent latent dynamics the lab is interested in time series modelling there also is a built in markov transition model we just put a version on arxiv bleau swanage june 5 2018 karlnapf leave a comment val di mello may 19 2018 may 23 2018 karlnapf leave a comment eiger mönch jungfrau may 5 2018 may 23 2018 karlnapf leave a comment maybe one day my phd thesis is online april 23 2018 june 28 2019 karlnapf leave a comment here it is just found it accidentally via google scholar help me increase the download counter cracks at millstone april 21 2018 may 24 2018 karlnapf leave a comment posts navigation previous proudly powered by wordpress theme penscratch by wordpress com loading comments write a comment email required name required website