Meta tags:
description=Trying to keep track of what I stumble upon;

Headings (most frequently used words):

the, share, this, of, theorem, for, you, jordan, form, way, fundamental, douglas, rachford, regularize, proving, existence, svd, directly, without, using, spectral, selfadjoint, compact, operators, why, should, calculate, normal, anyway, cannot, compute, canonical, numerically, handout, lecture, notes, lineare, algebra, an, index, free, to, take, gradient, neural, network, optimal, transport, best, stepsizes, lipschitz, strongly, montone, sweet, spot, iteration, subgradients, cyclical, monotonicity, and, calculus, mixing, norms, luxemburg, recent, posts, archives, blogroll, rss, feed, categories, email, subscription, tags,

Text of the page (most frequently used words):
the (381), and (152), that (118), for (87), this (75), new (63), opens (60), window (60), share (50), matrix (45), not (37), with (37), norm (36), which (35), are (34), function (30), all (29), monotone (28), one (26), can (26), email (25), have (25), there (25), #theorem (23), form (23), transport (22), jordan (21), but (21), let (21), linear (20), reddit (20), print (20), linkedin (20), facebook (20), you (19), here (19), some (19), has (19), from (19), may (18), map (18), optimal (17), what (17), canonical (17), like (17), set (17), more (16), just (16), get (16), consider (15), different (15), luxemburg (15), since (15), where (15), douglas (14), rachford (14), holds (14), monotonicity (14), diagonalizable (14), convex (13), functions (13), 2018 (13), under (13), does (13), matrices (13), now (12), subgradient (12), algebra (12), math (12), way (12), also (12), norms (12), two (12), simple (12), then (12), stepsize (12), mass (12), 00000 (12), loading (11), tags (11), june (11), 2012 (11), fundamental (11), even (11), about (11), optimization (10), link (10), friend (10), mixed (10), balls (10), same (10), above (10), note (10), first (10), posted (10), dirk (10), such (10), derivative (10), product (10), comment (9), functional (9), uncategorized (9), 2011 (9), 2013 (9), should (9), operators (9), again (9), show (9), its (9), construction (9), use (9), every (9), any (9), point (9), solution (9), concave (9), plan (9), ans (9), basis (8), another (8), july (8), sets (8), how (8), case (8), these (8), actually (8), example (8), proof (8), indeed (8), post (8), definition (8), see (8), superdifferential (8), exponential (8), other (7), measure (7), machine (7), september (7), february (7), march (7), 2014 (7), sweet (7), spot (7), take (7), neural (7), numerically (7), calculate (7), using (7), vector (7), hence (7), calculus (7), look (7), right (7), cyclically (7), constant (7), maps (7), eigenvalues (7), convergence (6), non (6), inverse (6), december (6), 2016 (6), 2017 (6), 2019 (6), iteration (6), lipschitz (6), strongly (6), index (6), network (6), compute (6), why (6), compact (6), second (6), only (6), both (6), conditions (6), continuous (6), sum (6), points (6), shows (6), means (6), cost (6), fact (6), problem (6), diagonal (6), write (5), regularize (5), blog (5), method (5), lecture (5), notes (5), problems (5), teaching (5), space (5), matlab (5), october (5), 2015 (5), existence (5), animated (5), level (5), equal (5), taking (5), possible (5), thing (5), into (5), general (5), each (5), say (5), out (5), supremum (5), cyclical (5), notion (5), mapping (5), factor (5), leave (5), marginals (5), assume (5), singular (5), kronecker (5), precision (5), 0000e (5), numbers (5), entry (5), comments (4), already (4), principle (4), integral (4), analysis (4), algorithms (4), learning (4), august (4), november (4), best (4), free (4), lineare (4), three (4), exponents (4), respective (4), while (4), last (4), functionals (4), could (4), would (4), something (4), defined (4), still (4), gives (4), little (4), did (4), equation (4), aim (4), exists (4), down (4), simply (4), single (4), contraction (4), small (4), optimality (4), item (4), related (4), feasible (4), scalar (4), want (4), tensor (4), vectorization (4), given (4), close (4), value (4), sign (3), subscribe (3), wordpress (3), com (3), weak (3), regularization (3), inclusion (3), processing (3), convexity (3), posts (3), rss (3), april (3), stepsizes (3), gradient (3), handout (3), anyway (3), svd (3), without (3), spectral (3), recent (3), images (3), radii (3), standard (3), they (3), shape (3), exponent (3), true (3), dimensional (3), parts (3), course (3), than (3), powers (3), equivalent (3), similar (3), allows (3), construct (3), variables (3), previous (3), coordinate (3), define (3), wrote (3), increasing (3), path (3), over (3), hand (3), side (3), clearly (3), corresponding (3), application (3), recover (3), additive (3), result (3), subgradients (3), number (3), multivalued (3), domain (3), itself (3), says (3), our (3), sense (3), finding (3), rate (3), numerical (3), plot (3), converges (3), respect (3), really (3), conjugate (3), measures (3), full (3), usual (3), need (3), start (3), following (3), leads (3), carry (3), products (3), loss (3), straightforward (3), identity (3), will (3), rewrite (3), pretty (3), usually (3), try (3), quite (3), indistinguishable (3), eigenvectors (3), floating (3), impossible (3), promise (3), invertible (3), decomposition (3), quote (3), limit (3), remember (3), sequence (3), orthonormal (3), required (2), view (2), log (2), subscribed (2), join (2), 138 (2), subscribers (2), sparsity (2), shapes (2), choice (2), papers (2), control (2), job (2), imaging (2), image (2), conference (2), pursuit (2), address (2), your (2), yet (2), january (2), 2025 (2), montone (2), cannot (2), normal (2), proving (2), directly (2), selfadjoint (2), page (2), cross (2), polytopes (2), octahedra (2), makes (2), their (2), scale (2), directions (2), geometry (2), induce (2), mixing (2), constructions (2), term (2), often (2), used (2), flexibility (2), part (2), bit (2), split (2), group (2), saw (2), add (2), converge (2), simplicity (2), denote (2), taken (2), integers (2), pairs (2), lower (2), finally (2), choose (2), remarkable (2), somehow (2), depends (2), rockafellar (2), stronger (2), words (2), maximally (2), claim (2), connected (2), follows (2), quoc (2), role (2), find (2), minimize (2), moursi (2), vandenberghe (2), remark (2), cumbersome (2), plots (2), expression (2), optimize (2), few (2), preprint (2), splitting (2), turns (2), matters (2), lot (2), too (2), very (2), siam (2), journal (2), before (2), decreasing (2), depending (2), variable (2), short (2), together (2), implication (2), concavity (2), prove (2), fulfill (2), moreover (2), correct (2), negative (2), fulfills (2), respectively (2), continuity (2), observe (2), back (2), differs (2), well (2), comes (2), natural (2), when (2), monotonically (2), condition (2), technicalities (2), integrability (2), boundedness (2), notions (2), probably (2), support (2), constraints (2), whole (2), good (2), distribution (2), instead (2), going (2), work (2), enough (2), takes (2), represent (2), represents (2), chain (2), rule (2), produce (2), layer (2), parameters (2), otherwise (2), concatenation (2), applied (2), weight (2), die (2), distinct (2), inv (2), warning (2), rcond (2), 11022e (2), 5036e (2), exist (2), rank (2), 2204e (2), row (2), dense (2), bunch (2), entries (2), ask (2), eigenvalue (2), unique (2), dependence (2), issue (2), commute (2), introductory (2), class (2), blue (2), upper (2), initial (2), explicit (2), always (2), complex (2), direct (2), subsequence (2), restriction (2), trying (2), keep (2), track (2), stumble (2), upon (2), website, name, collapse, bar, manage, subscriptions, site, reader, report, content, privacy, account, young, wavelets, uncertainty, tikhonov, pde, parameter, offer, advertisement, ivanov, ismp, transforms, ill, posed, fourier, transform, compressed, sensing, chi, denoising, enter, receive, notifications, subscription, software, statistics, signal, categories, feed, blogging, mathematician, steve, imge, pursuits, null, nuit, blanche, slice, pizza, mathblogging, org, mathbabe, loren, art, janlo, bandit, gower, weblog, francisco, blanco, silva, clément, mouhot, chapterzero, 0xde, blogroll, archives, next, ball, keeps, changes, scaling, stays, radius, observation, differently, pictures, picture, additional, summing, counts, partition, give, observations, differences, partitions, top, namely, among, things, behave, groups, shall, treated, homogeneous, satisfies, thus, normed, spaces, riemannian, increase, inside, equals, affine, semicontinuous, everywhere, proper, reconstruct, trick, base, basepoint, 1966, converse, property, called, terms, integer, obtain, hint, imply, properties, inequalities, symmetric, recall, hard, sweep, rug, counterexample, singlevalued, represented, writing, interchanged, roles, inequality, completely, suitable, higher, dimensions, select, dimension, fix, interval, relates, via, precise, wonder, hope, adaptive, proposed, basically, amounts, valued, swapped, paper, able, experimentally, knows, strong, guaranteed, explain, involves, root, polynomial, degree, below, surprisingly, complicated, nice, days, ago, walaa, lieven, published, derived, rates, special, mention, title, goes, iterates, operator, practice, large, lead, slow, kind, folk, wisdom, tran, dinh, investigated, visualization, tweet, introduce, svaiter, benar, fux, 280, 287, treatment, context, lions, pierre, louis, bertrand, mercier, nonlinear, 1979, 964, 979, been, clarified, hilbert, resolvent, reflector, concisely, written, blogged, solve, inclusions, comparing, pair, smallest, quantities, interesting, values, acts, time, follow, supported, corollary, assumption, contradiction, mine, statement, contained, denotes, signed, pushforward, better, shift, around, messy, probability, projections, onto, smaller, implies, neighborhoods, contrary, permutation, sketch, medskip, supergradient, wikipedia, concept, reversed, signs, infimum, resembles, permutations, uses, independent, latter, strange, argue, being, euclidean, seems, familiar, subset, real, line, moves, move, similarly, unclear, clarifications, involve, found, ambrosio, gigli, explanation, simpler, user, guide, contains, marginal, satisfied, characterizes, topic, understanding, demand, reach, formulated, distribute, looks, stands, run, target, according, world, theory, suggests, curry, thet, moved, serious, happens, situations, sits, least, end, flexible, model, kinds, stated, many, forms, formulation, kantorovich, measurable, coming, assigns, unit, placed, vice, versa, ways, formulate, mathematically, variations, final, relation, tensors, explains, tensorflow, reshaped, tupel, gradients, transpose, obtained, representation, particularly, implement, languages, provide, make, sure, advantage, sparse, memory, computer, flooded, zeros, wise, jacobian, calculated, rewritten, helpful, build, marvelous, compatible, size, particular, come, handy, columnwise, former, joint, computation, less, bias, input, activation, applies, training, networks, results, expressions, cluttered, indices, wintersemster, habe, ich, vorlesung, gehalten, hier, dazu, however, original, 100, accurate, recombining, although, diagonalize, heard, mean, representative, known, certain, contain, exactly, was, diagonalized, closer, promised, fulfilled, happening, putcome, diagonalizability, seem, diag, octave, simplest, speaking, shown, uniqueness, limits, forming, defining, double, eigenspace, spanned, extend, solving, quotes, underlying, reasons, teach, year, got, thinking, issues, depend, continuously, instructor, said, fairly, big, approximation, asymptotics, curve, component, exact, initialized, red, corresponds, yellow, expected, nothing, else, difference, quotient, far, away, triangular, light, arbitrarily, replacing, error, presented, classroom, blocks, block, nil, potent, exponentials, easy, diagonalization, notice, fibonicci, appear, clear, thought, fibonacci, treat, them, series, challenge, favorite, system, explicitly, help, illustrated, reason, most, play, bother, computing, prime, solutions, systems, odes, later, trefethen, bau, present, generalize, argument, idea, know, initially, planned, avoid, couldn, claimed, infinite, compactness, cauchy, inner, lagrange, multipliers, subspace, orthogonal, restrict, process, ends, proceed, iteratively, maximizing, bounded, generality, bugged, had, rely, self, adjoint, lectures, decided, mathematics, home,

Text of the page (random words):
onormal basis of numbers such that for all and for all proof we consider the problem let be a maximizing sequence and since the sequence is bounded we can without loss of generality assume that it has a weak limit since is compact we have that converges to and it holds that we set now we proceed iteratively assume that and has already be defined set i e the subspace that is orthogonal to all the we restrict to i e we define as the restriction of to and observe that is again compact if the process ends and otherwise we get as above the existence of with and we set and by construction we have now let us show that the are orthonormal again by construction using lagrange multipliers for the problem we have the optimality condition taking the inner product of the first equation with we get using that is a restriction of which shows that with this we get for that this also shows that by construction we have if the is infinite dimensional then by compactness of and boundedness of have that has a subsequence that is a cauchy sequence this means that and hence we have that since we can start with any subsequence this shows that finally we show that is a basis of assume that for all and then and by construction we have for hence if for all then i e as claimed initially i planned to avoid all use of eigenvalues but i couldn t see how to do this later i saw that in trefethen and bau s numerical linear algebra the present a direct proof of the existence of the svd of complex matrices that does not use eigenvalues all together theorem 4 1 but i could not see how to generalize the argument to operators yet if you have any idea how to do this let me know share this share on facebook opens in new window facebook share on x opens in new window x share on linkedin opens in new window linkedin email a link to a friend opens in new window email print opens in new window print more share on reddit opens in new window reddit like loading july 3 2019 why should you calculate the jordan normal form anyway posted by dirk under math teaching uncategorized tags eigenvalues eigenvectors jordan canonical form linear algebra leave a comment in my previous post i illustrated why it is not possible to compute the jordan canonical form numerically i e in floating point numbers the simple reason for every matrix and every there is a matrix which differs from by at most e g in every entry but all norms for matrices are equivalent so this does not really play a role such that is diagonalizable so why should you bother about computing the jordan canonical form anyway or even learning or teaching it well the prime application of the jordan canonical form is to calculate solutions of linear systems of odes the equation with matrix and initial value both could also be complex this system has a unique solution which can be given explicitly with the help of the matrix exponential as where the matrix exponential is it is not always simple to work out the matrix exponential by hand the straightforward way would be to calculate all the powers of weight them by and sum the series this may be a challenge even for simple matrices my favorite example is the matrix its first powers are you may notice that the fibonicci numbers appear and this is pretty clear on a second thought so finding a explicit form for leads us to finding an explicit form for the th fibonacci number which is possible but i will not treat this here another way is diagonalization if is diagonalizable i e there is an invertible matrix and a diagonal matrix such that you see that and the matrix exponential of a diagonal matrix is simply the exponential function applied to the diagonal entries but not all matrices are diagonalizable the solution that is usually presented in the classroom is to use the jordan canonical form instead and to compute the matrix exponential of jordan blocks using that you can split a jordan block into the sum of a diagonal matrix and a nil potent matrix and since and commute one can calculate and both matrix exponentials are quite easy to compute but in light of the fact that there are a diagonalizable matrices arbitrarily close to any matrix on may ask what about replacing a non diagonalizable matrix with a diagonalizable one with a small error and then use this one let s try this on a simple example we consider which is not diagonalizable the linear initial value problem has the solution and the matrix exponential is so we get the solution let us take a close by matrix which is diagonalizable for some small we choose since is upper triangular it has its eigenvalues on the diagonal since there are two distinct eigenvalues and hence is diagonalizable indeed with we get the matrix exponential of is hence the solution of is how is this related to the solution of how far is it away of course the lower right entry of converges to for but what about the upper right entry note that the entry is nothing else that the negative difference quotient for the derivative of the function at hence and we get as expected it turns out that a fairly big is already enough to get a quite good approximation and even the correct asymptotics the blue curve it first component of the exact solution initialized with the second standard basis vector the red one corresponds and the yellow on pretty close to the blue on is for to e share this share on facebook opens in new window facebook share on x opens in new window x share on linkedin opens in new window linkedin email a link to a friend opens in new window email print opens in new window print more share on reddit opens in new window reddit like loading june 19 2019 you cannot compute the jordan canonical form numerically posted by dirk under math teaching uncategorized tags eigenvalues eigenvectors jordan canonical form linear algebra 7 comments i remember from my introductory class in linear algebra that my instructor said it is impossible to calculate the jordan canonical form of a matrix numerically another quote i remember is the jordan canonical form does not depend continuously on the matrix for both quotes i did not remember the underlying reasons and since i do teach an introductory class on linear algebra this year i got thinking about these issues again here is a very simple example for the fact in the second quote consider the matrix for this matrix has as a double eigenvalue but the corresponding eigenspace is one dimensional and spanned by to extend this vector to a basis we calculate a principle vector by solving which leads to defining we get the jordan canonical form of as so we have but is not the jordan canonical form of so in short the jordan canonical form of the limit of is not the limit of the jordan canonical form of in other words taking limits does not commute with forming the jordan canonical form a side note of course the jordan canonical form is not even unique in general so speaking of dependence on the matrix is an issue what we have shown is that there is no way to get continuous dependence on the matrix even if non uniqueness is not an issue like in the example above what about the first quote why is it impossible to compute the jordan canonical form numerically let s just try we start with the simplest non diagonalizable matrix if we ask matlab or octave to do the eigenvalue decomposition we get v d diag a v 1 00000 1 00000 0 00000 0 00000 d 1 0 0 1 we see that does not seem to be invertible and indeed we get rank v ans 1 what is happening matlab did not promise the produce an invertible and it does not promise that the putcome would fulfill which is my definition of diagonalizability it does promise that and in fact a v ans 1 00000 1 00000 0 00000 0 00000 v d ans 1 00000 1 00000 0 00000 0 00000 a v v d ans 0 0000e 00 2 2204e 16 0 0000e 00 0 0000e 00 so the promised identity is fulfilled up to machine precision which is actually equal and we denote it by from here on how did matlab diagonalize this matrix here is the thing the diagonalizable matrices are dense in you probably have heard that before what does that mean numerically any matrix that you represent in floating point numbers is actually a representative of a whole bunch of matrices each entry is only known up to a certain precision but this bunch of matrices does contain some matrix which is diagonalizable this is exactly what it means to be a dense set so it is impossible to say if a matrix given in floating point numbers is actually diagonalizable or not so what matrix was diagonalized by matlab let us have closer look at the matrix the entries in the first row are in fact and v 1 ans 1 1 in the second row we have v 2 1 ans 0 v 2 2 ans 2 2204e 16 and there we have it the inverse of does exist although the matrix has numerical rank and it is inv v warning matrix singular to machine precision rcond 1 11022e 16 ans 1 0000e 00 4 5036e 15 0 0000e 00 4 5036e 15 and note that is indeed just the inverse of the machine precision so this inverse is actually 100 accurate recombining gives inv v d v warning matrix singular to machine precision rcond 1 11022e 16 ans 1 0 0 1 which is not even close to our original how is that here is a solution the matrix is indistinguishable from numerically however it has two distinct eigenvalues so it is diagonalizable indeed a basis of eigenvectors is which is indistinguishable from above and it holds that which is indistinguishable from share this share on facebook opens in new window facebook share on x opens in new window x share on linkedin opens in new window linkedin email a link to a friend opens in new window email print opens in new window print more share on reddit opens in new window reddit like loading march 27 2019 handout lecture notes lineare algebra 1 posted by dirk under math teaching uncategorized tags lecture notes linear algebra leave a comment im wintersemster 2018 19 habe ich die vorlesung lineare algebra 1 gehalten hier die lecture notes dazu handout lineare algebra 1 share this share on facebook opens in new window facebook share on x opens in new window x share on linkedin opens in new window linkedin email a link to a friend opens in new window email print opens in new window print more share on reddit opens in new window reddit like loading december 13 2018 an index free way to take the gradient of a neural network posted by dirk under machine learning math optimization uncategorized tags chain rule kronecker product machine learning neural network optimization vectorization 1 comment taking the derivative of the loss function of a neural network can be quite cumbersome even taking the derivative of a single layer in a neural network often results in expressions cluttered with indices in this post i d like to show an index free way to do it consider the map where is the weight matrix is the bias is the input and is the activation function usually represents both a scalar function i e mapping and the function mapping which applies in each coordinate in training neural networks we would try to optimize for best parameters and so we need to take the derivative with respect to and so we consider the map this map is a concatenation of the map and and since the former map is linear in the joint variable the derivative of should be pretty simple what makes the computation a little less straightforward is the fact the we are usually not used to view matrix vector products as linear maps in but in so let s rewrite the thing there are two particular notions which come in handy here the kronecker product of matrices and the vectorization of matrices vectorization takes some given columnwise and maps it by the kronecker product of matrices and is a matrix in we will build on the following marvelous identity for matrices of compatible size we have that why is this helpful it allows us to rewrite so we can also rewrite so our map mapping can be rewritten as mapping since is just a concatenation of applied coordinate wise and a linear map now given as a matrix the derivative of i e the jacobian a matrix in is calculated simply as while this representation of the derivative of a single layer of a neural network with respect to its parameters is not particularly simple it is still index free and moreover straightforward to implement in languages which provide functions for the kronecker product and vectorization if you do this make sure to take advantage of sparse matrices for the identity matrix and the diagonal matrix as otherwise the memory of your computer will be flooded with zeros now let s add a scalar function e g to produce a scalar loss that we can minimize i e we consider the map the derivative is obtained by just another application of the chain rule if we want to take gradients we just transpose the expression and get note that the right hand side is indeed vector in and hence can be reshaped to a tupel of an matrix and an vector a final remark the kronecker product is related to tensor products if and represent linear maps and respectively then represents the tensor product of the maps this relation to tensor products and tensors explains where the tensor in tensorflow comes from share this share on facebook opens in new window facebook share on x opens in new window x share on linkedin opens in new window linkedin email a link to a friend opens in new window email print opens in new window print more share on reddit opens in new window reddit like loading september 13 2018 the fundamental theorem of optimal transport posted by dirk under math optimization regularization tags calculus of variations optimal transport leave a comment the problem of optimal transport of mass from one distribution to another can be stated in many forms here is the formulation going back to kantorovich we have two measurable sets and coming with two measures and we also have a function which assigns a transport cost i e is the cost that it takes to carry one unit of mass from point to what we want is a plan that says where the mass in should be placed in or vice versa there are different ways to formulate this mathematically a simple way is to look for a map which says that thet mass in should be moved to while natural there is a serious problem with this what if not all mass at should go to the same point in this happens in simple situations where all mass in sits in just one point but there are at least two different points in where mass should end up this is not going to work with a map as above so the map is not flexible enough to model all kinds of transport we may need what we want is a way to distribute mass from one point in to the whole set this looks like we want maps which map points in to functions on i e something like where stands for some set of functions on we can de curry this function to some by that s good in principle be we still run into problems when the target mass distribution is singular in the sense that is a continuous set and there are single points in that carry some mass according to since ...