Meta tags:

Headings (most frequently used words):

gradient, descent, solution, of, linear, system, and, contents, description, non, comments, modifications, extensions, theoretical, properties, examples, see, also, references, further, reading, external, links, an, analogy, for, understanding, choosing, the, step, size, direction, geometric, behavior, residual, orthogonality, fast, methods, momentum, or, heavy, ball, method,

Text of the page (most frequently used words):
the (279), mathbf (174), displaystyle (123), #gradient (108), #descent (82), and (82), for (63), learning (61), algorithm (56), method (54), that (48), eta (42), optimization (34), nabla (32), search (31), function (29), methods (28), direction (27), this (26), neural (25), with (24), can (23), edit (21), right (21), machine (20), step (20), left (20), algorithms (19), local (18), linear (17), convex (17), convergence (17), steepest (16), bmatrix (16), end (16), network (15), point (15), doi (15), then (15), are (15), top (15), text (14), first (14), artificial (14), size (14), which (14), minimum (13), isbn (13), used (13), loop (13), minimization (12), one (12), solution (12), system (12), begin (12), using (11), momentum (11), they (11), from (10), conjugate (10), line (10), stochastic (10), arxiv (10), networks (10), problems (10), value (10), qquad (10), exp (10), cos (10), use (9), hill (9), wolfe (9), intelligence (9), vector (9), objective (9), such (9), programming (8), quadratic (8), newton (8), unconstrained (8), list (8), negative (8), matrix (8), based (8), optimal (8), also (8), example (8), small (8), repeat (8), wikipedia (7), may (7), non (7), links (7), differentiable (7), general (7), computer (7), self (7), analysis (7), regression (7), term (7), applied (7), 978 (7), more (7), case (7), heavy (7), not (7), number (7), would (7), aligned (7), frac (7), policy (6), path (6), tree (6), nonlinear (6), deep (6), clustering (6), iterative (6), data (6), systems (6), large (6), model (6), has (6), press (6), journal (6), nesterov (6), positive (6), preconditioning (6), properties (6), assumptions (6), when (6), ball (6), iteration (6), slow (6), down (6), iterations (6), find (6), where (6), inequality (6), result (6), choose (6), mountain (6), sequence (6), steepness (6), toggle (5), under (5), page (5), description (5), order (5), mathematical (5), flow (5), minima (5), other (5), fletcher (5), conditions (5), engineering (5), automated (5), vision (5), factor (5), diffusion (5), random (5), field (5), reasoning (5), problem (5), control (5), models (5), image (5), supervised (5), series (5), new (5), pdf (5), further (5), springer (5), rate (5), smooth (5), fast (5), 2nd (5), between (5), two (5), lipschitz (5), update (5), each (5), will (5), constant (5), equations (5), any (5), its (5), sufficiently (5), global (5), geq (5), move (5), contents (4), about (4), commons (4), wikidata (4), articles (4), external (4), all (4), retrieved (4), set (4), frank (4), powell (4), symmetric (4), bfgs (4), functions (4), planning (4), related (4), feature (4), proximal (4), memory (4), reinforcement (4), backpropagation (4), recurrent (4), reduction (4), logic (4), rule (4), theta (4), cost (4), generative (4), residual (4), john (4), people (4), 2023 (4), ascent (4), york (4), introduction (4), university (4), 2016 (4), january (4), s2cid (4), 1016 (4), information (4), issn (4), 1137 (4), siam (4), theory (4), least (4), see (4), converges (4), theoretical (4), given (4), trying (4), analogy (4), simple (4), converge (4), modifications (4), space (4), should (4), main (4), changes (4), 306 (4), 456 (4), guess (4), 472 (4), sin (4), tfrac (4), current (4), orthogonality (4), square (4), exit (4), return (4), note (4), real (4), optimising (4), max (4), choosing (4), requires (4), some (4), instrument (4), measure (4), get (4), hide (4), sidebar (4), view (3), terms (3), wikimedia (3), was (3), link (3), wayback (3), dead (3), short (3), software (3), annealing (3), climbing (3), evolutionary (3), maximum (3), graph (3), bound (3), dynamic (3), principal (3), successive (3), mirror (3), quasi (3), outline (3), open (3), source (3), quantum (3), glossary (3), alphago (3), game (3), language (3), processing (3), time (3), state (3), action (3), chaining (3), error (3), map (3), perceptron (3), generalized (3), selection (3), dimensionality (3), means (3), structured (3), bayes (3), genetic (3), approach (3), decision (3), classification (3), weighted (3), forest (3), filter (3), forward (3), backward (3), applications (3), variational (3), autoencoder (3), transformer (3), andrew (3), stephen (3), knowledge (3), human (3), agent (3), context (3), symbolic (3), datasets (3), parameter (3), history (3), math (3), archived (3), october (3), 2017 (3), how (3), online (3), derivatives (3), vandenberghe (3), lieven (3), 2004 (3), cambridge (3), 521 (3), 2015 (3), complexity (3), 2018 (3), cite (3), 001 (3), optimized (3), basic (3), mathematics (3), squares (3), computing (3), distance (3), barzilai (3), borwein (3), yang (3), cauchy (3), examples (3), assumed (3), lead (3), sophisticated (3), suitable (3), zig (3), zag (3), being (3), called (3), proposed (3), known (3), decrease (3), mathcal (3), have (3), solving (3), while (3), possible (3), generally (3), every (3), level (3), sets (3), like (3), steps (3), found (3), evaluating (3), leq (3), 3x_ (3), 4x_ (3), 625x_ (3), 2x_ (3), 20x_ (3), 625 (3), consider (3), solve (3), shows (3), condition (3), because (3), definite (3), most (3), both (3), finding (3), defined (3), good (3), brackets (3), their (3), observation (3), tools (3), subsection (3), languages (2), table (2), safety (2), contact (2), privacy (2), last (2), 2026 (2), hidden (2), categories (2), category (2), deprecated (2), parameters (2), simulated (2), metaheuristics (2), ford (2), dijkstra (2), paradigms (2), active (2), simplex (2), plane (2), constrained (2), levenberg (2), marquardt (2), gauss (2), rank (2), davidon (2), broyden (2), goldfarb (2), shanno (2), hall (2), nelder (2), mead (2), motion (2), lists (2), journals (2), muzero (2), alphazero (2), zero (2), play (2), scale (2), ransac (2), parser (2), outlier (2), temporal (2), difference (2), skill (2), constructing (2), trees (2), actor (2), critic (2), organizing (2), rprop (2), embedding (2), sparse (2), pca (2), component (2), alignment (2), kernel (2), association (2), optics (2), hierarchical (2), dbscan (2), support (2), relevance (2), naive (2), logistic (2), majority (2), boosting (2), markov (2), monte (2), carlo (2), expectation (2), maximization (2), statistical (2), inference (2), multimodal (2), differential (2), deepening (2), depth (2), best (2), beam (2), impact (2), visual (2), art (2), video (2), education (2), economic (2), principle (2), act (2), gan (2), mamba (2), rnn (2), convolutional (2), gru (2), lstm (2), turing (2), architectures (2), david (2), alex (2), fei (2), geoffrey (2), hinton (2), simon (2), semantic (2), ibm (2), recognition (2), speech (2), synthesis (2), alexnet (2), protocol (2), hypothetical (2), weak (2), neuro (2), rlhf (2), training (2), sarsa (2), bias (2), variance (2), tradeoff (2), loss (2), projects (2), via (2), learn (2), 83378 (2), boyd (2), reading (2), combettes (2), eds (2), fixed (2), science (2), part (2), lecture (2), course (2), coursera (2), uses (2), help (2), citeseerx (2), 151 (2), pmid (2), 0036 (2), review (2), 2019 (2), yurii (2), computational (2), practical (2), numerical (2), recipes (2), scientific (2), 2021 (2), improve (2), adam (2), accelerated (2), society (2), 2020 (2), philip (2), 235 (2), vol (2), jonathan (2), 141 (2), functional (2), polyak (2), curry (2), haskell (2), 1944 (2), 1090 (2), 1943 (2), hadamard (2), jacques (2), des (2), references (2), backtracking (2), mills (2), guarantees (2), require (2), strongly (2), special (2), euclidean (2), divergence (2), constraints (2), projection (2), extensions (2), technique (2), extension (2), calculate (2), pattern (2), minimized (2), through (2), next (2), previous (2), same (2), fgm (2), acceleration (2), faster (2), been (2), since (2), decreases (2), there (2), textstyle (2), neighborhood (2), even (2), saddle (2), applying (2), although (2), rather (2), explicit (2), techniques (2), better (2), alternatives (2), but (2), extremely (2), hessian (2), take (2), different (2), directions (2), shape (2), expensive (2), however (2), works (2), dimensions (2), inner (2), product (2), vectors (2), variable (2), proportional (2), typically (2), comments (2), approximate (2), 0075 (2), 002 (2), 20944 (2), might (2), gives (2), now (2), must (2), 209 (2), thus (2), minimize (2), let (2), define (2), cases (2), due (2), residuals (2), overshoot (2), toward (2), reason (2), taken (2), kappa (2), orthogonal (2), geometric (2), behavior (2), much (2), hbox (2), twice (2), implies (2), traditional (2), follows (2), locally (2), following (2), assuming (2), clever (2), setting (2), way (2), need (2), over (2), second (2), amount (2), off (2), measures (2), angle (2), settings (2), downhill (2), mathematically (2), too (2), choices (2), idea (2), slope (2), represents (2), travel (2), before (2), illustrated (2), stuck (2), mountains (2), fog (2), were (2), want (2), understanding (2), here (2), contour (2), bowl (2), multi (2), studied (2), minimizing (2), curve (2), net (2), anomaly (2), detection (2), prediction (2), shot (2), appearance (2), upload (2), file (2), read (2), article (2), log (2), create (2), account (2), donate (2), menu (2), add, topic, mobile, cookie, statement, statistics, developers, code, conduct, legal, contacts, disclaimers, available, additional, apply, site, you, agree, registered, trademark, profit, organization, foundation, inc, creative, attribution, sharealike, license, rendered, parsoid, edited, september, utc, webarchive, template, march, cs1, errors, matches, https, org, index, php, title, gradient_descent, oldid, 1373673523, tabu, spiral, parallel, push, relabel, fulkerson, edmonds, karp, dinic, flows, floyd, warshall, spfa, bellman, shortest, kruskal, prim, borůvka, spanning, cut, branch, integer, greedy, approximation, combinatorial, pivoting, lemke, criss, cross, revised, dantzig, exchange, basis, projective, karmarkar, ellipsoid, khachiyan, affine, scaling, interior, subgradient, reduced, cutting, sequential, augmented, lagrangian, penalty, barrier, computes, maxima, hessians, truncated, dog, leg, sr1, berndt, hausman, trust, region, gradients, parabolic, interpolation, golden, section, heuristics, alphatensor, alphaevolve, alphadev, turboquant, satplan, gammon, invariant, transform, grabcut, canny, edge, detector, inside, outside, earley, cocke, younger, kasami, byte, pair, encoding, natural, idistance, warping, vattention, pagedattention, reward, pvlv, prefrontal, cortex, basal, ganglia, working, driven, wake, sleep, quickprop, leabra, quantization, growing, hebbian, generec, almeida, pineda, distributed, neighbor, factorization, redundancy, manifold, fastica, growth, eclat, apriori, lloyd, linde, buzo, gray, medoids, winnow, knn, ripper, classifier, nearest, neighbors, id3, production, dominance, rough, cn2, alternating, randomized, lpboost, logitboost, brownboost, bootstrap, aggregating, adaboost, ensemble, histogram, rapidly, exploring, probabilistic, roadmap, graphplan, window, bug, pathfinding, viterbi, chain, mcmc, kalman, belief, propagation, baum, welch, walksat, resolution, dpll, rete, particle, swarm, ant, colony, bio, inspired, evolution, computation, binary, bayesian, alopex, sss, minimax, expectiminimax, alpha, beta, pruning, uniform, solver, breadth, stack, workplace, warfare, military, games, marketing, chatbot, psychosis, healthcare, fiction, architecture, engine, explainable, environmental, competition, arms, race, anthropomorphism, winter, slop, literacy, infrastructure, effect, center, bubble, boom, social, opposition, centers, propaganda, virtual, politician, regulation, precautionary, nationalism, ethics, elections, takeover, government, cold, war, political, gnn, adversarial, vae, highway, cnn, multilayer, mlp, echo, gated, unit, long, vit, françois, chollet, daniel, kokotajlo, jan, leike, mustafa, suleyman, schulman, aidan, gomez, noam, shazeer, ashish, vaswani, andrej, karpathy, silver, demis, hassabis, ian, goodfellow, quoc, oriol, vinyals, ilya, sutskever, krizhevsky, james, goodnight, graves, grossberg, lotfi, zadeh, yoshua, bengio, yann, lecun, jürgen, schmidhuber, hopfield, paul, werbos, seppo, linnainmaa, seymour, papert, joseph, weizenbaum, bernard, widrow, rosenblatt, oliver, selfridge, herbert, cliff, shaw, allen, newell, nathaniel, rochester, mccarthy, marvin, minsky, takeo, kanade, kunihiko, fukushima, shun, ichi, amari, claude, shannon, christopher, manning, von, neumann, walter, pitts, warren, sturgis, mcculloch, alan, yago, dbpedia, conceptnet, bases, opencog, lida, clarion, soar, cognitive, reasoners, procedural, programs, engines, expert, deductive, classifiers, robot, autogpt, driving, car, openai, five, decisional, watsonx, watson, project, debater, oasis, genie, world, udio, suno, riffusion, music, generation, veo, seedance, sora, kling, hailuo, runway, gen, dream, stable, recraft, midjourney, imagen, ideogram, gpt, flux, firefly, dall, aurora, alphafold, facial, whisper, elevenlabs, ocr, hwr, wavenet, audio, implementations, physical, agent2agent, superintelligence, asi, agi, lethal, autonomous, weapons, laws, humanity, exam, companion, intelligent, nmt, playing, theorem, proving, situated, sovereign, blended, vibe, coding, word, hallucination, recursive, improvement, reflection, llm, post, uncanny, valley, rag, adversary, autoregression, latent, imitation, prompt, augmentation, regularization, weight, initialization, gating, rectifier, sigmoid, softmax, activation, batchnorm, normalization, convolution, attention, sgd, overfitting, double, hyperparameter, representation, constraint, satisfaction, concepts, proprietary, institutions, companies, timeline, garrigos, guillaume, gower, robert, handbook, theorems, 2301, 11235, youtube, 3blue1brown, ghostarchive, book, teaching, khan, academy, videos, discusses, boost, ublas, media, himmelblau, 1972, procedures, mcgraw, 132, 028921, chong, edwin, żak, stanislaw, 2013, fourth, hoboken, wiley, 160, 118, 27901, 131, 520, 457, bubeck, sébastien, 1405, 4980, pesquet, 2011, splitting, signal, bauschke, elser, luke, wolkowicz, 212, 4419, 9568, 0912, 3522, 185, inverse, burachik, nitish, srivastava, kevin, swersky, 2014, willamette, adaptation, qian, ning, 1999, 2783597, 12662723, s0893, 6080, 00116, 5612, 145, drori, yoel, exact, 205861966, jco, 1606, 01424, kim, fessler, 107, 207055414, 27765996, 5067109, pmc, 1007, s10107, 015, 0949, 1406, 5468, walkington, noel, 562, 1445, 21m1390037, 539, notes, ee236c, ucla, 4020, 7553, introductory, lectures, ross, july, 127649426, cam, 044, 354, strutz, vieweg, 658, 11455, fitting, uncertainty, beyond, vetterling, 1992, 43108, flannery, teukolsky, xie, zeke, yuan, zhu, zhanxing, sugiyama, masashi, pmlr, 11458, 2103, 17182, 11448, proceedings, 38th, international, conference, manipulating, noise, generalization, kingma, diederik, jimmy, 1412, 6980, meyer, gerard, november, 1974, 663, 1402, 0312050, 655, diakonikolas, jelena, jordan, michael, 944, 1052, 6234, 20m1322716, 1906, 00436, 915, hamiltonian, perspective, abdulkadirov, ruslan, lyakhov, pavel, nagornov, nikolay, 2466, 2227, 7390, 3390, math11112466, survey, modern, holmes, 031, 22429, bouwmeester, henricus, dougherty, knyazev, 285, procs, 241, 1212, 6680, 276, procedia, nonsymmetric, saad, yousef, 2003, philadelphia, industrial, 89871, 534, 195, haykin, adaptive, pearson, india, 2008, 108, 142, 217, 242, bernstein, jeremy, vahdat, arash, yue, yisong, liu, ming, stability, 2002, 03432, april, 1969, 1011036, 226, 2005, teo, boston, 256, 387, 24254, 1988, 148, 1093, imanum, ima, akilov, 1982, pergamon, 023036, kantorovich, 1987, boris, 261, qam, 10667, 258, quart, appl, courant, s0002, 9904, 07818, bulletin, american, equilibrium, vibrations, 1908, mémoire, sur, problème, analyse, relatif, équilibre, plaques, élastiques, encastrées, mémoires, présentés, par, divers, savants, éstrangers, académie, sciences, institut, france, lemaréchal, 2012, documenta, mathematica, 1st, ems, 254, original, 936609, 4171, dms, 251, stories, grötschel, 1017, cbo9780511804441, neuroevolution, continuous, cls, formula, delta, seiberg, witten, higgs, depend, variant, made, affect, proven, linearly, looser, either, weaker, squared, bregman, extended, handle, including, onto, only, feasible, efficiently, computable, specific, monotone, inclusions, includes, inequalities, train, updating, adds, property, weights, break, sliding, surface, values, mass, movement, medium, remembers, determines, asymptotically, combination, conservative, force, viscous, newtonian, dynamics, fpgm, modification, enables, agm, specifically, generated, nevertheless, opportunity, reducing, ogm, reduces, bounded, develops, subsequent, iterates, progress, resulting, multiple, address, these, deficiencies, viewed, turn, equation, derived, controller, feedback, form, ordinary, euler, sometimes, substitute, family, relies, than, exploration, inversion, fewer, higher, consists, calculating, multiplied, into, combined, issues, dominate, limited, instead, modified, momentums, exponential, moving, averages, optimizers, diffgrad, yogi, adabelief, etc, many, compute, required, very, geometry, cures, computationally, concentric, circles, curvature, accuracy, finite, seen, consequence, magnitude, dot, dimension, maximized, independent, adjustments, partial, colinear, schwarz, spaces, infinite, dimensional, ones, latter, calculates, determine, fréchet, derivative, sizable, reduce, until, yields, done, variety, simply, 8x_, 1250x_, 1250, jacobian, know, attempt, initial, introduce, associated, animation, showing, surfaces, arrows, show, isosurfaces, below, three, unknown, variables, shown, slowly, high, forces, undo, zigzags, inefficiency, preferred, alternate, extreme, axes, elongated, produces, characteristic, poor, conditioning, primary, cause, reinforces, alternation, langle, rangle, across, rarely, popular, commonly, spectral, ratio, determined, root, benefit, less, preconditioner, eigenvalues, avoid, multiplying, per, performed, analytically, formulas, norm, reformulated, wiener, usually, above, guaranteed, stronger, advanced, convexity, build, custom, estimate, approx, forgo, benefits, satisfies, rates, experimental, results, does, gamma, could, extra, evaluations, undesirable, ways, around, sure, decreased, depends, trade, quickly, along, thought, letting, denote, say, fairly, assumption, continuously, prove, important, advocated, practice, deviates, seem, counter, intuitive, smaller, compensated, sustained, longer, represent, explore, aligns, taking, another, measurement, differentiation, intuition, behind, scenario, visibility, low, therefore, visible, involves, looking, position, proceeding, proceed, uphill, eventually, possibly, hole, lake, assume, immediately, obvious, happen, moment, takes, quite, sunset, difficulty, frequency, track, process, adjacent, picture, blue, curves, regions, red, arrow, originating, going, leads, bottom, minimal, lines, satisfying, guarantee, certain, particular, those, include, desired, allowed, change, cdots, monotonic, enough, words, subtracted, against, mind, starts, considers, ldots, mathbb, goes, fastest, illustration, serves, today, attributed, who, suggested, 1847, independently, similar, 1907, becoming, increasingly, well, decades, augustin, louis, particularly, useful, repeated, opposite, conversely, stepping, trajectory, maximizes, procedure, confused, research, jmlr, ijcai, iclr, icml, neurips, iccv, emnlp, ecml, pkdd, eccv, cvpr, aaai, conferences, topological, pac, occam, empirical, risk, machines, foundations, roc, confusion, coefficient, determination, diagnostics, mechanistic, interpretability, crowdsourcing, humans, ecram, electrochemical, ram, memtransistor, spiking, physics, informed, radiance, deepdream, lenet, som, restricted, boltzmann, reservoir, esn, feedforward, isolation, conditional, graphical, sdl, sne, pgd, nmf, lda, ica, cca, exploratory, mean, shift, fuzzy, cure, birch, svm, rvm, bagging, ensembles, apprenticeship, ontology, grammar, induction, rules, automl, cleaning, density, estimation, modeling, neuromorphic, curriculum, batch, few, meta, transfer, semi, unsupervised, mining, multivariate, analytical, free, encyclopedia, item, printable, version, download, print, export, switch, legacy, shortened, url, permanent, what, actions, english, talk, tiếng, việt, українська, ไทย, svenska, српски, srpski, русский, português, polski, മലയാളം, lietuvių, 한국어, 日本語, italiano, bahasa, indonesia, עברית, galego, français, فارسی, español, deutsch, čeština, català

Text of the page (random words):
sion perceptron relevance vector machine rvm support vector machine svm clustering birch cure hierarchical k means fuzzy expectation maximization em dbscan optics mean shift dimensionality reduction factor analysis exploratory cca ica lda nmf pca pgd t sne sdl structured prediction graphical models bayes net conditional random field hidden markov anomaly detection ransac k nn local outlier factor isolation forest neural networks autoencoder deep learning feedforward neural network recurrent neural network lstm gru esn reservoir computing boltzmann machine restricted gan diffusion model som convolutional neural network u net lenet alexnet deepdream neural field neural radiance field physics informed neural networks transformer vision mamba spiking neural network memtransistor electrochemical ram ecram reinforcement learning q learning policy gradient sarsa temporal difference td multi agent self play learning with humans active learning crowdsourcing human in the loop mechanistic interpretability rlhf model diagnostics coefficient of determination confusion matrix learning curve roc curve mathematical foundations kernel machines bias variance tradeoff computational learning theory empirical risk minimization occam learning pac learning statistical learning vc theory topological deep learning journals and conferences aaai cvpr eccv ecml pkdd emnlp iccv neurips icml iclr ijcai ml jmlr related articles glossary of artificial intelligence list of datasets for machine learning research list of datasets in computer vision and image processing outline of machine learning v t e the idea is to take repeated steps in the opposite direction of the gradient or approximate gradient of the function at the current point because this is the direction of steepest descent conversely stepping in the direction of the gradient will lead to a trajectory that maximizes that function the procedure is then known as gradient ascent gradient descent should not be confused with local search algorithms although both are iterative methods for optimization gradient descent is particularly useful in machine learning and artificial intelligence for minimizing the cost or loss function 1 gradient descent is generally attributed to augustin louis cauchy who first suggested it in 1847 2 jacques hadamard independently proposed a similar method in 1907 3 4 its convergence properties for non linear optimization problems were first studied by haskell curry in 1944 5 with the method becoming increasingly well studied and used in the following decades 6 7 a simple extension of gradient descent stochastic gradient descent serves as the most basic algorithm used for training most deep networks today description edit illustration of gradient descent on a series of level sets negative gradient in direction of steepest descent gradient descent is based on the observation that if the multi variable function f x displaystyle f mathbf x is defined and differentiable in a neighborhood of a point a displaystyle mathbf a then f x displaystyle f mathbf x decreases fastest if one goes from a displaystyle mathbf a in the direction of the negative gradient of f displaystyle f at a i e f a displaystyle mathbf a i e nabla f mathbf a it follows that if a n 1 a n η f a n displaystyle mathbf a _ n 1 mathbf a _ n eta nabla f mathbf a _ n for a small enough step size or learning rate η r displaystyle eta in mathbb r _ then f a n f a n 1 displaystyle f mathbf a_ n geq f mathbf a_ n 1 in other words the term η f a displaystyle eta nabla f mathbf a is subtracted from a displaystyle mathbf a because we want to move against the gradient toward the local minimum with this observation in mind one starts with a guess x 0 displaystyle mathbf x _ 0 for a local minimum of f displaystyle f and considers the sequence x 0 x 1 x 2 displaystyle mathbf x _ 0 mathbf x _ 1 mathbf x _ 2 ldots such that x n 1 x n η n f x n n 0 displaystyle mathbf x _ n 1 mathbf x _ n eta _ n nabla f mathbf x _ n n geq 0 we have a monotonic sequence f x 0 f x 1 f x 2 displaystyle f mathbf x _ 0 geq f mathbf x _ 1 geq f mathbf x _ 2 geq cdots so the sequence x n displaystyle mathbf x _ n converges to the desired local minimum note that the value of the step size η displaystyle eta is allowed to change at every iteration it is possible to guarantee the convergence to a local minimum under certain assumptions on the function f displaystyle f for example f displaystyle f convex and f displaystyle nabla f lipschitz and particular choices of η displaystyle eta those include the sequence η n x n x n 1 f x n f x n 1 f x n f x n 1 2 displaystyle eta _ n frac left left mathbf x _ n mathbf x _ n 1 right top left nabla f mathbf x _ n nabla f mathbf x _ n 1 right right left nabla f mathbf x _ n nabla f mathbf x _ n 1 right 2 as in the barzilai borwein method 8 9 or a sequence η n displaystyle eta _ n satisfying the wolfe conditions which can be found by using line search when the function f displaystyle f is convex all local minima are also global minima so in this case gradient descent can converge to the global solution this process is illustrated in the adjacent picture here f displaystyle f is assumed to be defined on the plane and that its graph has a bowl shape the blue curves are the contour lines that is the regions on which the value of f displaystyle f is constant a red arrow originating at a point shows the direction of the negative gradient at that point note that the negative gradient at a point is orthogonal to the contour line going through that point we see that gradient descent leads us to the bottom of the bowl that is to the point where the value of the function f displaystyle f is minimal an analogy for understanding gradient descent edit fog in the mountains the basic intuition behind gradient descent can be illustrated by a hypothetical scenario people are stuck in the mountains and are trying to get down i e trying to find the global minimum there is heavy fog such that visibility is extremely low therefore the path down the mountain is not visible so they must use local information to find the minimum they can use the method of gradient descent which involves looking at the steepness of the hill at their current position then proceeding in the direction with the steepest descent i e downhill if they were trying to find the top of the mountain i e the maximum then they would proceed in the direction of steepest ascent i e uphill using this method they would eventually find their way down the mountain or possibly get stuck in some hole i e local minimum or saddle point like a mountain lake however assume also that the steepness of the hill is not immediately obvious with simple observation but rather it requires a sophisticated instrument to measure which the people happen to have at that moment it takes quite some time to measure the steepness of the hill with the instrument thus they should minimize their use of the instrument if they want to get down the mountain before sunset the difficulty then is choosing the frequency at which they should measure the steepness of the hill so as not to go off track in this analogy the people represent the algorithm and the path taken down the mountain represents the sequence of parameter settings that the algorithm will explore the steepness of the hill represents the slope of the function at that point the instrument used to measure steepness is differentiation the direction they choose to travel in aligns with the gradient of the function at that point the amount of time they travel before taking another measurement is the step size choosing the step size and descent direction edit since using a step size η displaystyle eta that is too small would slow convergence and a η displaystyle eta too large would lead to overshoot and divergence finding a good setting of η displaystyle eta is an important practical problem philip wolfe also advocated using clever choices of the descent direction in practice 10 while using a direction that deviates from the steepest descent direction may seem counter intuitive the idea is that the smaller slope may be compensated for by being sustained over a much longer distance to reason about this mathematically consider a direction p n displaystyle mathbf p _ n and step size η n displaystyle eta _ n and consider the more general update a n 1 a n η n p n displaystyle mathbf a _ n 1 mathbf a _ n eta _ n mathbf p _ n finding good settings of p n displaystyle mathbf p _ n and η n displaystyle eta _ n requires some thought first of all we would like the update direction to point downhill mathematically letting θ n displaystyle theta _ n denote the angle between f a n displaystyle nabla f mathbf a_ n and p n displaystyle mathbf p _ n this requires that cos θ n 0 displaystyle cos theta _ n 0 to say more we need more information about the objective function that we are optimising under the fairly weak assumption that f displaystyle f is continuously differentiable we may prove that 11 f a n 1 f a n η n f a n 2 p n 2 cos θ n max t 0 1 f a n t η n p n f a n 2 f a n 2 displaystyle f mathbf a _ n 1 leq f mathbf a _ n eta _ n nabla f mathbf a _ n _ 2 mathbf p _ n _ 2 left cos theta _ n max _ t in 0 1 frac nabla f mathbf a _ n t eta _ n mathbf p _ n nabla f mathbf a _ n _ 2 nabla f mathbf a _ n _ 2 right 1 this inequality implies that the amount by which we can be sure the function f displaystyle f is decreased depends on a trade off between the two terms in square brackets the first term in square brackets measures the angle between the descent direction and the negative gradient the second term measures how quickly the gradient changes along the descent direction in principle inequality 1 could be optimized over p n displaystyle mathbf p _ n and η n displaystyle eta _ n to choose an optimal step size and direction the problem is that evaluating the second term in square brackets requires evaluating f a n t η n p n displaystyle nabla f mathbf a _ n t eta _ n mathbf p _ n and extra gradient evaluations are generally expensive and undesirable some ways around this problem are forgo the benefits of a clever descent direction by setting p n f a n displaystyle mathbf p _ n nabla f mathbf a_ n and use line search to find a suitable step size γ n displaystyle gamma _ n such as one that satisfies the wolfe conditions a more economic way of choosing learning rates is backtracking line search a method that has both good theoretical guarantees and experimental results note that one does not need to choose p n displaystyle mathbf p _ n to be the gradient any direction that has positive inner product with the gradient will result in a reduction of the function value for a sufficiently small value of η n displaystyle eta _ n assuming that f displaystyle f is twice differentiable use its hessian 2 f displaystyle nabla 2 f to estimate f a n t η n p n f a n 2 t η n 2 f a n p n displaystyle nabla f mathbf a _ n t eta _ n mathbf p _ n nabla f mathbf a _ n _ 2 approx t eta _ n nabla 2 f mathbf a _ n mathbf p _ n then choose p n displaystyle mathbf p _ n and η n displaystyle eta _ n by optimising inequality 1 assuming that f displaystyle nabla f is lipschitz use its lipschitz constant l displaystyle l to bound f a n t η n p n f a n 2 l t η n p n displaystyle nabla f mathbf a _ n t eta _ n mathbf p _ n nabla f mathbf a _ n _ 2 leq lt eta _ n mathbf p _ n then choose p n displaystyle mathbf p _ n and η n displaystyle eta _ n by optimising inequality 1 build a custom model of max t 0 1 f a n t η n p n f a n 2 f a n 2 displaystyle max _ t in 0 1 frac nabla f mathbf a _ n t eta _ n mathbf p _ n nabla f mathbf a _ n _ 2 nabla f mathbf a _ n _ 2 for f displaystyle f then choose p n displaystyle mathbf p _ n and η n displaystyle eta _ n by optimising inequality 1 under stronger assumptions on the function f displaystyle f such as convexity more advanced techniques may be possible usually by following one of the recipes above convergence to a local minimum can be guaranteed when the function f displaystyle f is convex all local minima are also global minima so in this case gradient descent can converge to the global solution solution of a linear system edit the steepest descent algorithm applied to the wiener filter 12 gradient descent can be used to solve a system of linear equations a x b 0 displaystyle mathbf a mathbf x mathbf b 0 reformulated as a quadratic minimization problem if the system matrix a displaystyle mathbf a is real symmetric and positive definite an objective function is defined as the quadratic function with minimization of f x x a x 2 x b displaystyle f mathbf x mathbf x top mathbf a mathbf x 2 mathbf x top mathbf b so that f x 2 a x b displaystyle nabla f mathbf x 2 mathbf a mathbf x mathbf b for a general real matrix a displaystyle mathbf a linear least squares define f x a x b 2 displaystyle f mathbf x left mathbf a mathbf x mathbf b right 2 in traditional linear least squares for real a displaystyle mathbf a and b displaystyle mathbf b the euclidean norm is used in which case f x 2 a a x b displaystyle nabla f mathbf x 2 mathbf a top mathbf a mathbf x mathbf b the line search minimization finding the locally optimal step size η displaystyle eta on every iteration can be performed analytically for quadratic functions and explicit formulas for the locally optimal η displaystyle eta are known 6 13 for example for real symmetric and positive definite matrix a displaystyle mathbf a a simple algorithm can be as follows 6 repeat in the loop r b a x η r r r a r x x η r if r r is sufficiently small then exit loop end repeat loop return x as the result displaystyle begin aligned text repeat in the loop qquad mathbf r mathbf b mathbf ax qquad eta mathbf r top mathbf r mathbf r top mathbf ar qquad mathbf x mathbf x eta mathbf r qquad hbox if mathbf r top mathbf r text is sufficiently small then exit loop text end repeat loop text return mathbf x text as the result end aligned to avoid multiplying by a displaystyle mathbf a twice per iteration we note that x x η r displaystyle mathbf x mathbf x eta mathbf r implies r r η a r displaystyle mathbf r mathbf r eta mathbf ar which gives the traditional algorithm 14 r b a x repeat in the loop η r r r a r x x η r if r r is sufficiently small then exit loop r r η a r end repeat loop return x as the result displaystyle begin aligned mathbf r mathbf b mathbf ax text repeat in the loop qquad eta mathbf r top mathbf r mathbf r top mathbf ar qquad mathbf x mathbf x eta mathbf r qquad hbox if mathbf r top mathbf r text is sufficiently small then exit loop qquad mathbf r mathbf r eta mathbf ar text end repeat loop text return mathbf x text as the result end aligned convergence path of steepest descent method for a 2 2 2 3 the method is rarely used for solving linear equations with the conjugate gradient method being one of the most popular alternatives the number of gradient descent iterations is commonly proportional to the spectr...