Meta tags:

Headings (most frequently used words):

stochastic, gradient, descent, variants, contents, background, iterative, method, linear, regression, history, notable, applications, extensions, and, approximations, in, continuous, time, see, also, notes, references, further, reading, external, links, implicit, updates, isgd, momentum, averaging, adagrad, rmsprop, adam, sign, based, backtracking, line, search, second, order, methods,

Text of the page (most frequently used words):
the (274), and (112), learning (111), displaystyle (107), gradient (101), #stochastic (82), for (64), #descent (62), with (43), this (37), neural (35), eta (35), optimization (31), machine (31), that (29), method (28), edit (26), rate (26), from (25), doi (25), algorithm (24), function (24), adam (23), sum (23), can (21), text (20), where (20), training (19), left (18), right (18), parameter (17), momentum (17), arxiv (17), adaptive (17), frac (17), old (17), gradients (16), convergence (16), nabla (16), are (16), search (15), articles (15), methods (15), sgd (15), networks (15), new (15), was (14), network (14), regression (14), squares (14), approximation (14), rmsprop (14), has (14), using (13), least (13), information (13), pdf (13), may (12), all (12), data (12), models (12), s2cid (12), update (12), used (12), hat (12), retrieved (11), based (11), large (11), first (11), order (11), linear (11), line (11), only (11), mathbb (11), textstyle (11), been (11), when (11), each (11), statistics (10), 2023 (10), hinton (10), artificial (10), deep (10), isbn (10), also (10), such (10), not (10), which (10), vector (10), set (10), bmatrix (10), applications (9), geoffrey (9), systems (9), journal (9), ieee (9), second (9), estimation (9), beta (9), parameters (9), adagrad (9), step (9), wikipedia (8), june (8), 2018 (8), convex (8), john (8), model (8), weight (8), 2021 (8), mean (8), 2019 (8), problems (8), its (8), one (8), since (8), iteration (8), factor (8), tau (8), min (8), links (7), algorithms (7), david (7), 978 (7), 1109 (7), bibcode (7), 2016 (7), processing (7), back (7), empirical (7), 1997 (7), minimization (7), backtracking (7), given (7), objective (7), rates (7), variants (7), but (7), article (7), unsourced (6), james (6), loss (6), spall (6), 2012 (6), mathematical (6), scale (6), robbins (6), procedure (6), cite (6), batch (6), introduced (6), average (6), online (6), random (6), through (6), time (6), however (6), max (6), more (6), constant (6), sqrt (6), does (6), standard (6), form (6), risk (6), hessian (6), over (6), updates (6), single (6), called (6), between (6), isgd (6), implicit (6), problem (6), converges (6), non (5), use (5), page (5), statements (5), april (5), 2020 (5), list (5), external (5), statistical (5), ilya (5), control (5), self (5), image (5), general (5), intelligence (5), supervised (5), backpropagation (5), clustering (5), functions (5), history (5), 2017 (5), bottou (5), léon (5), springer (5), efficient (5), next (5), beyond (5), vol (5), field (5), issn (5), sign (5), january (5), help (5), nesterov (5), university (5), 2014 (5), lecture (5), running (5), sample (5), press (5), optimizers (5), propagation (5), maximum (5), than (5), have (5), small (5), needed (5), both (5), evaluations (5), exponential (5), include (5), iterations (5), main (5), mini (5), batches (5), classical (5), ast (5), logistic (5), very (5), begin (5), end (5), summand (5), contents (4), policy (4), additional (4), dead (4), references (4), short (4), approximations (4), estimators (4), computational (4), generative (4), perceptron (4), recurrent (4), vision (4), computer (4), sutskever (4), reasoning (4), diffusion (4), speech (4), newton (4), bias (4), variance (4), further (4), modified (4), dynamics (4), equations (4), 1214 (4), annals (4), version (4), monro (4), simultaneous (4), singer (4), magnitude (4), link (4), requires (4), converge (4), without (4), analysis (4), international (4), conference (4), recent (4), jmlr (4), 2011 (4), 1986 (4), 323 (4), means (4), some (4), feature (4), pytorch (4), icassp (4), likelihood (4), denotes (4), changes (4), example (4), see (4), consider (4), there (4), then (4), iterative (4), direct (4), summands (4), another (4), matrix (4), above (4), simple (4), often (4), citation (4), other (4), case (4), even (4), while (4), improves (4), many (4), get (4), forgetting (4), square (4), current (4), published (4), size (4), delta (4), high (4), ldots (4), arg (4), samples (4), minimum (4), true (4), partial (4), evaluating (4), hide (4), move (4), sidebar (4), toggle (3), view (3), code (3), under (3), terms (3), last (3), 2026 (3), hidden (3), cs1 (3), multiple (3), errors (3), needing (3), july (3), different (3), wikidata (3), graph (3), autoencoder (3), long (3), transformer (3), alex (3), fei (3), bengio (3), yann (3), lecun (3), frank (3), rosenblatt (3), herbert (3), knowledge (3), rule (3), language (3), generation (3), recognition (3), human (3), agent (3), context (3), automated (3), theorem (3), approach (3), symbolic (3), reinforcement (3), engineering (3), datasets (3), october (3), learn (3), introduction (3), estimates (3), icml (3), natural (3), efficiently (3), ruppert (3), 2013 (3), perturbation (3), 2009 (3), transactions (3), decay (3), 1904 (3), amsgrad (3), whye (3), 2022 (3), advances (3), representations (3), divide (3), subgradient (3), polyak (3), averaging (3), pmid (3), 1038 (3), rumelhart (3), williams (3), finite (3), fast (3), mit (3), full (3), sources (3), tensorflow (3), libraries (3), almost (3), estimate (3), element (3), product (3), reduction (3), rather (3), scaling (3), leq (3), respect (3), sense (3), behavior (3), solution (3), continuous (3), setting (3), developed (3), determining (3), matrices (3), practice (3), practical (3), particular (3), condition (3), change (3), loop (3), cost (3), account (3), better (3), norm (3), performance (3), done (3), work (3), well (3), decreasing (3), gamma (3), calculated (3), idea (3), described (3), every (3), per (3), few (3), sparse (3), averaged (3), 1980s (3), sometimes (3), several (3), related (3), alpha (3), estimated (3), most (3), observations (3), equation (3), selected (3), too (3), later (3), extensions (3), approaches (3), were (3), remains (3), local (3), computing (3), tools (3), languages (2), table (2), safety (2), contact (2), about (2), privacy (2), you (2), organization (2), categories (2), expanded (2), maint (2), names (2), authors (2), missing (2), 2024 (2), clarification (2), description (2), org (2), impact (2), visual (2), video (2), principle (2), act (2), gan (2), mamba (2), rnn (2), convolutional (2), gru (2), memory (2), lstm (2), differentiable (2), turing (2), architectures (2), daniel (2), ian (2), goodfellow (2), stephen (2), yoshua (2), amari (2), christopher (2), manning (2), semantic (2), classifiers (2), ibm (2), stable (2), synthesis (2), alexnet (2), implementations (2), protocol (2), neuro (2), open (2), source (2), recursive (2), improvement (2), rlhf (2), sarsa (2), regularization (2), initialization (2), normalization (2), quasi (2), tradeoff (2), projects (2), glossary (2), paper (2), distill (2), works (2), original (2), youtube (2), archived (2), how (2), 2003 (2), 471 (2), 33052 (2), wiley (2), tricks (2), trade (2), backprop (2), fundamentals (2), 168 (2), reading (2), sebastian (2), february (2), research (2), foundations (2), nonlinear (2), jacobian (2), 1998 (2), raphson (2), tac (2), automatic (2), 2000 (2), byrd (2), hansen (2), nocedal (2), 1137 (2), siam (2), tran (2), access (2), proof (2), reddi (2), sashank (2), kale (2), satyen (2), kumar (2), sanjiv (2), 09237 (2), book (2), peter (2), powerpropagation (2), fasfa (2), optimizer (2), into (2), rules (2), neurips (2), iclr (2), kingma (2), diederik (2), jimmy (2), 1412 (2), 6980 (2), toronto (2), 2025 (2), his (2), coursera (2), class (2), march (2), duchi (2), hazan (2), elad (2), yoram (2), 2159 (2), 2121 (2), boris (2), optim (2), acceleration (2), accelerated (2), langevin (2), simulated (2), annealing (2), adadelta (2), martens (2), heavy (2), ball (2), ronald (2), propagating (2), 6088 (2), 536 (2), 205001834 (2), 323533a0 (2), 1986natur (2), 533r (2), 533 (2), nature (2), properties (2), 1990 (2), ijcnn (2), results (2), permanent (2), martin (2), inversion (2), geophysics (2), berlin (2), heidelberg (2), 2008 (2), conditional (2), 1007 (2), review (2), mining (2), bilmes (2), jeff (2), chin (2), chee (2), demmel (2), phipac (2), speed (2), error (2), munich (2), germany (2), 4153 (2), 4156 (2), 604861 (2), acoustics (2), signal (2), asanovic (2), krste (2), probabilistic (2), kiefer (2), wolfowitz (2), aoms (2), 1951 (2), fit (2), interpolation (2), optimizing (2), siegmund (2), series (2), bousquet (2), olivier (2), tradeoffs (2), 1982 (2), jstor (2), association (2), cambridge (2), wise (2), odot (2), notes (2), coordinate (2), sde (2), point (2), motion (2), sdes (2), dimensional (2), noise (2), flow (2), dots (2), lfloor (2), rfloor (2), exists (2), db_ (2), sigma (2), tfrac (2), fluctuations (2), proposed (2), expectation (2), subject (2), following (2), sufficiently (2), smooth (2), let (2), asymptotically (2), optimal (2), uses (2), possible (2), require (2), less (2), differences (2), instead (2), these (2), requiring (2), values (2), achievable (2), resulting (2), effective (2), mentioned (2), known (2), armijo (2), allow (2), need (2), lipschitz (2), special (2), adding (2), needs (2), though (2), aforementioned (2), being (2), taken (2), considering (2), significantly (2), lower (2), rprop (2), adamw (2), squared (2), adamax (2), examples (2), initial (2), averages (2), would (2), zero (2), scalar (2), prevent (2), moments (2), varepsilon (2), moment (2), shown (2), updated (2), historical (2), borrowed (2), solve (2), root (2), invented (2), successfully (2), after (2), now (2), written (2), previous (2), diagonal (2), ones (2), settings (2), ordinary (2), same (2), 2010s (2), name (2), physics (2), traveling (2), combined (2), minimizes (2), determines (2), earlier (2), combination (2), generalized (2), implemented (2), follows (2), closed (2), wide (2), range (2), specifically (2), suppose (2), features (2), will (2), numerically (2), normalized (2), comparison (2), filter (2), lms (2), relatively (2), mild (2), assumptions (2), diverge (2), within (2), shortened (2), solved (2), note (2), largely (2), evaluated (2), visualizing (2), improvements (2), basic (2), recognized (2), cause (2), makes (2), bfgs (2), popular (2), community (2), graphical (2), support (2), machines (2), notable (2), derived (2), techniques (2), applying (2), optimize (2), benefits (2), vectorization (2), 1950s (2), underparameterized (2), overparameterized (2), pseudocode (2), become (2), difference (2), surely (2), perform (2), because (2), until (2), randomly (2), cases (2), expensive (2), sums (2), subset (2), here (2), arise (2), estimating (2), background (2), theory (2), curve (2), net (2), forest (2), anomaly (2), detection (2), bayes (2), structured (2), prediction (2), dimensionality (2), classification (2), shot (2), citations (2), appearance (2), upload (2), file (2), read (2), log (2), create (2), donate (2), menu (2), add, topic, mobile, cookie, statement, developers, conduct, legal, contacts, disclaimers, available, apply, site, agree, registered, trademark, profit, wikimedia, foundation, inc, creative, commons, attribution, sharealike, license, rendered, parsoid, edited, september, utc, periodical, november, permanently, 2015, https, index, php, title, stochastic_gradient_descent, oldid, 1376343264, category, workplace, warfare, military, art, games, marketing, chatbot, psychosis, healthcare, fiction, education, architecture, engine, explainable, environmental, competition, arms, race, anthropomorphism, winter, slop, literacy, infrastructure, effect, center, bubble, boom, social, economic, opposition, centers, propaganda, virtual, politician, regulation, precautionary, nationalism, ethics, elections, takeover, alignment, government, cold, war, political, gnn, adversarial, variational, vae, highway, residual, cnn, multilayer, mlp, echo, state, gated, unit, term, vit, françois, chollet, kokotajlo, jan, leike, mustafa, suleyman, schulman, aidan, gomez, noam, shazeer, ashish, vaswani, andrej, karpathy, silver, demis, hassabis, quoc, oriol, vinyals, krizhevsky, andrew, goodnight, graves, grossberg, lotfi, zadeh, jürgen, schmidhuber, hopfield, paul, werbos, seppo, linnainmaa, seymour, papert, joseph, weizenbaum, bernard, widrow, oliver, selfridge, simon, cliff, shaw, allen, newell, nathaniel, rochester, mccarthy, marvin, minsky, takeo, kanade, kunihiko, fukushima, shun, ichi, claude, shannon, von, neumann, walter, pitts, warren, sturgis, mcculloch, alan, people, yago, dbpedia, conceptnet, bases, opencog, lida, clarion, soar, cognitive, reasoners, procedural, logic, programs, inference, engines, expert, deductive, robot, autogpt, action, selection, muzero, driving, car, openai, five, alphazero, alphago, decisional, watsonx, watson, project, debater, oasis, genie, world, udio, suno, riffusion, music, veo, seedance, sora, kling, hailuo, runway, gen, dream, recraft, midjourney, imagen, ideogram, gpt, flux, firefly, dall, aurora, alphafold, facial, whisper, elevenlabs, ocr, hwr, wavenet, audio, physical, agent2agent, hypothetical, superintelligence, asi, agi, weak, lethal, autonomous, weapons, laws, humanity, exam, companion, intelligent, nmt, game, playing, proving, actor, critic, situated, sovereign, blended, vibe, coding, word, embedding, hallucination, reflection, llm, post, uncanny, valley, rag, adversary, autoregression, latent, imitation, prompt, augmentation, gating, rectifier, sigmoid, softmax, activation, batchnorm, convolution, attention, conjugate, overfitting, double, hyperparameter, representation, constraint, satisfaction, planning, concepts, lists, software, proprietary, institutions, companies, timeline, interactive, explaining, goh, 23915, 00006, why, really, via, 3blue1brown, orr, genevieve, 642, 35288, müller, klaus, robert, buduma, nikhil, locascio, nicholas, reilly, 9781491925584, designing, 2004, lnai, 3176, 540, 23122, 146, advanced, lectures, gess, benjamin, kassing, konarovskyi, vitalii, flows, limits, math, 2302, 07125, qianxiao, tai, cheng, weinan, 1533, 7928, 1811, 01558, brust, 2107, 05598, workshop, 276, 207585383, 1162, 089976698300017746, 251, computation, 1985, 245, aos, 1176346589, 236, multivariate, bhatnagar, prasad, prashanth, london, 4471, 4284, feedback, weighting, mechanisms, improving, 1229, 3564529, 2019793, 2009itac, 1216s, 1216, 1839, 1853, 880982, 2000itac, 1839s, 1031, 12396034, 140954362, 1401, 7020, 1008, 569, 560, signsgd, compressed, optimisation, balles, lukas, hennig, philipp, dissecting, loshchilov, hutter, decoupled, 1711, 05101, phuong, thi, phong, trieu, 61716, 2169, 3536, 2916341, 2019ieeea, 761706t, 03590, 61706, overview, yuzheng, lin, licong, tang, shange, 1912, 09926, schwarz, jonathan, jayakumar, siddhant, pascanu, razvan, latham, teh, yee, 1333722169, oclc, sparsity, inducing, reparameterisation, naveen, philip, 36227, techrxiv, 20427852, novel, dozat, incorporating, 70293087, zhang, yushun, chen, congliang, shi, naichen, sun, ruoyu, luo, zhi, quan, any, modification, 2208, 09632, rubio, martínez, master, thesis, oxford, 6th, event, occurs, dropout, credited, effectiveness, various, deepai, gupta, maya, samy, weston, jason, 1492, 1461, highly, multiclass, juditsky, anatoli, 1992, 855, 3548228, 0330046, 838, papers, explained, borysenko, oleksandr, byshkin, maksym, 10705, 34021212, 8139967, pmc, s41598, 021, 90144, 2021natsr, 1110705b, 2005, 14605, scientific, reports, coolmomentum, zeiler, matthew, 1212, 5701, dahl, george, sanjoy, dasgupta, mcallester, proceedings, 30th, atlanta, 1147, 1139, importance, toulis, panos, airoldi, edoardo, asymptotic, 1727, 10279395, aos1506, 1408, 2923, 1694, hoboken, sections, simulation, cited, darken, christian, moody, int, joint, conf, 137720, courville, aaron, 291, 0262035613, avi, pfeffer, harvard, cs181, perceptrons, krebs, jerome, anderson, hinkley, neelamani, ramesh, lee, sunwoong, baumstein, anatoly, lacasse, wavefield, seismic, encoded, wcc188, 1190, 3230502, wcc177, jenny, rose, finkel, kleeman, proc, annual, meeting, acl, parsing, module, keras, nguyen, giang, dlugolinsky, stefan, bobák, viet, garcía, álvaro, heredia, ignacio, malík, hluchý, ladislav, 124, 254236976, s10462, 018, 09679, frameworks, survey, torch, documentation, 1476, 4687, peng, xinyu, wang, yue, accelerating, minibatch, typicality, sampling, 4659, 73728964, 31899442, tnnls, 2957003, 2020itnnl, 4649p, 1903, 04192, 4649, 1958, storage, brain, 408, 12781225, 13602029, 1037, h0042519, 386, psychological, 1952, 466, 1177729392, 462, 400, 1177729586, belkin, mikhail, 248, 0962, 4929, 1017, s0962492921000039, 2105, 14368, 203, acta, numerica, fear, remarkable, phenomena, prism, 1971, negative, supermartingales, rustagi, jagdish, academic, 604550, kiwiel, krzysztof, 2001, efficiency, quasiconvex, 10043417, 1819784, 0025, 5610, pl00011414, programming, murphy, kevin, 161, inconsistent, 380, 834, 2287314, 1080, 01621459, 10477894, 831, american, ferguson, thomas, 521, 65263, sra, suvrit, nowozin, wright, eds, 368, 262, 01646, 351, hill, climbing, classifier, differentially, private, broken, law, approximates, infinite, precise, brownian, ito, integral, dw_, capture, around, solutions, limiting, objects, precisely, differential, taking, choice, indices, scheme, valid, horizon, assume, coefficients, test, viewed, discretization, ode, analogue, deterministic, provides, near, measurements, directly, required, theoretically, sound, versions, others, perturbations, replacing, fisher, transforms, usual, either, inputs, optimality, calculation, predictive, structure, exploited, 2nd, variant, below, sourced, goldstein, manner, check, unknown, advance, hand, guarantee, property, enjoys, globally, chosen, section, expansion, goes, researchers, tried, simplify, removing, communication, transferring, workers, server, serves, compress, having, comparable, past, adamx, infinity, varying, interpretations, adasqrt, enhanced, nadam, popularity, inspired, enhancements, establishing, incomplete, subsequent, revealed, objectives, despite, continues, due, strong, moving, initialized, towards, compensate, division, 999, factors, respectively, squaring, rooting, combining, indexes, indexed, good, adaptation, seen, generalization, capable, opposed, concept, storing, diminishing, gradually, influence, phd, students, group, like, adapted, magnitudes, unusually, merely, designed, applied, essentially, stores, dimension, formula, gives, rise, applies, denominator, derivatives, extreme, dampened, receive, higher, mathrm, diag, mathsf, still, base, multiplied, elements, outer, informally, increases, decreases, strategy, informative, sparser, takes, place, bar, independently, late, records, keeps, track, mid, predicted, yurii, stems, analogy, thought, particle, space, incurs, unlike, tends, keep, direction, preventing, oscillations, scientists, decades, closely, underdamped, force, relative, contribution, proposals, appeared, soviet, mathematician, 1964, solving, functional, remembers, leads, found, regular, thus, bounds, bisection, simply, equivalent, depends, write, depend, except, obeys, instance, poisson, remain, virtually, similar, nlms, uniformly, sampled, although, theoretical, happens, quite, unstable, misspecified, absolute, eigenvalues, probability, contrast, wish, indicates, inner, could, intercept, proceeds, appears, sides, proximal, generally, sensitive, induce, numerical, instability, whereby, iterate, perspective, projection, problematic, low, slow, conceptually, extension, number, giving, fine, tuning, schedules, macqueen, guidance, choosing, schedule, competes, widely, 1960, originally, adaline, including, reported, waveform, fwi, facto, vowpal, wabbit, dominated, far, predecessors, classic, partially, supports, device, setups, groups, limited, had, already, added, assumed, fixed, branches, hyperparameters, across, soon, substituted, explored, paving, way, balancing, layers, earliest, preceding, building, year, close, demonstrating, applicability, central, jacob, jack, sutton, behaves, differently, distance, starting, forall, geq, want, straight, corresponding, responses, minimized, specific, key, batched, leftarrow, 2x_, analyzed, theories, briefly, decrease, appropriate, global, otherwise, fact, consequence, pseudoconvex, compromise, compute, against, make, separately, bunch, mode, result, smoother, computed, repeat, approximate, obtained, shuffle, choose, presented, approximated, sweeps, performs, passes, made, shuffled, pass, cycles, typical, total, steps, enormous, formulas, exist, becomes, economize, enables, inexpensive, economical, families, minimize, denoted, variable, arises, value, independent, minimizers, restrictive, therefore, contemporary, theorists, zeros, derivative, score, stationary, points, typically, associated, observation, minimizing, behind, traced, today, important, abbreviated, suitable, regarded, replaces, actual, entire, thereof, especially, reduces, achieving, faster, exchange, burden, subdifferentiable, smoothness, outline, ijcai, iccv, emnlp, ecml, pkdd, eccv, cvpr, aaai, journals, conferences, topological, pac, occam, kernel, roc, confusion, coefficient, determination, diagnostics, mechanistic, interpretability, crowdsourcing, active, humans, play, multi, temporal, ecram, electrochemical

Text of the page (random words):
istic regression q x i w y i s x i w displaystyle q x_ i w y_ i s x_ i w where s u e u 1 e u displaystyle s u e u 1 e u is the logistic function in poisson regression q x i w y i e x i w displaystyle q x_ i w y_ i e x_ i w and so on in such settings isgd is simply implemented as follows let f ξ η q x i w old ξ x i 2 displaystyle f xi eta q x_ i w text old xi x_ i 2 where ξ displaystyle xi is scalar then isgd is equivalent to w new w old ξ x i where ξ f ξ displaystyle w text new w text old xi ast x_ i text where xi ast f xi ast the scaling factor ξ r displaystyle xi ast in mathbb r can be found through the bisection method since in most regular models such as the aforementioned generalized linear models function q displaystyle q is decreasing and thus the search bounds for ξ displaystyle xi ast are min 0 f 0 max 0 f 0 displaystyle min 0 f 0 max 0 f 0 momentum edit further proposals include the momentum method or the heavy ball method which in ml context appeared in rumelhart hinton and williams paper on backpropagation learning 30 and borrowed the idea from soviet mathematician boris polyak s 1964 article on solving functional equations 31 stochastic gradient descent with momentum remembers the update δ w at each iteration and determines the next update as a linear combination of the gradient and the previous update 32 33 δ w α δ w η q i w displaystyle delta w alpha delta w eta nabla q_ i w w w δ w displaystyle w w delta w that leads to w w η q i w α δ w displaystyle w w eta nabla q_ i w alpha delta w where the parameter w displaystyle w which minimizes q w displaystyle q w is to be estimated η displaystyle eta is a step size sometimes called the learning rate in machine learning and α displaystyle alpha is an exponential decay factor between 0 and 1 that determines the relative contribution of the current gradient and earlier gradients to the weight change the name momentum stems from an analogy to momentum in physics the weight vector w displaystyle w thought of as a particle traveling through parameter space 30 incurs acceleration from the gradient of the loss force unlike in classical stochastic gradient descent it tends to keep traveling in the same direction preventing oscillations momentum has been used successfully by computer scientists in the training of artificial neural networks for several decades 34 the momentum method is closely related to underdamped langevin dynamics and may be combined with simulated annealing 35 in mid 1980s the method was modified by yurii nesterov to use the gradient predicted at the next point and the resulting so called nesterov accelerated gradient was sometimes used in ml in the 2010s 36 averaging edit averaged stochastic gradient descent invented independently by ruppert and polyak in the late 1980s is ordinary stochastic gradient descent that records an average of its parameter vector over time that is the update is the same as for ordinary stochastic gradient descent but the algorithm also keeps track of 37 w 1 t i 0 t 1 w i displaystyle bar w frac 1 t sum _ i 0 t 1 w_ i when optimization is done this averaged parameter vector takes the place of w adagrad edit adagrad for adaptive gradient algorithm is a modified stochastic gradient descent algorithm with per parameter learning rate first published in 2011 38 informally this increases the learning rate for sparser parameters clarification needed and decreases the learning rate for ones that are less sparse this strategy often improves convergence performance over standard stochastic gradient descent in settings where data is sparse and sparse parameters are more informative examples of such applications include natural language processing and image recognition 38 it still has a base learning rate η but this is multiplied with the elements of a vector g j j which is the diagonal of the outer product matrix g τ 1 t g τ g τ t displaystyle g sum _ tau 1 t g_ tau g_ tau mathsf t where g τ q i w displaystyle g_ tau nabla q_ i w the gradient at iteration τ the diagonal is given by g j j τ 1 t g τ j 2 displaystyle g_ j j sum _ tau 1 t g_ tau j 2 this vector essentially stores a historical sum of gradient squares by dimension and is updated after every iteration the formula for an update is now a w w η d i a g g 1 2 g displaystyle w w eta mathrm diag g frac 1 2 odot g or written as per parameter updates w j w j η g j j g j displaystyle w_ j w_ j frac eta sqrt g_ j j g_ j each g i i gives rise to a scaling factor for the learning rate that applies to a single parameter w i since the denominator in this factor g i τ 1 t g τ 2 textstyle sqrt g_ i sqrt sum _ tau 1 t g_ tau 2 is the ℓ 2 norm of previous derivatives extreme parameter updates get dampened while parameters that get few or small updates receive higher learning rates 34 while designed for convex problems adagrad has been successfully applied to non convex optimization 39 rmsprop edit rmsprop for root mean square propagation is a method invented in 2012 by james martens and ilya sutskever at the time both phd students in geoffrey hinton s group in which the learning rate is like in adagrad adapted for each of the parameters the idea is to divide the learning rate for a weight by a running average of the magnitudes of recent gradients for that weight 40 unusually it was not published in an article but merely described in a coursera lecture citation needed 41 42 so first the running average is calculated in terms of means square v w t γ v w t 1 1 γ q i w 2 displaystyle v w t gamma v w t 1 left 1 gamma right left nabla q_ i w right 2 where γ displaystyle gamma is the forgetting factor the concept of storing the historical gradient as sum of squares is borrowed from adagrad but forgetting is introduced to solve adagrad s diminishing learning rates in non convex problems by gradually decreasing the influence of old data citation needed and the parameters are updated as w w η v w t q i w displaystyle w w frac eta sqrt v w t nabla q_ i w rmsprop has shown good adaptation of learning rate in different applications rmsprop can be seen as a generalization of rprop and is capable to work with mini batches as well opposed to only full batches 40 adam edit adam 43 short for adaptive moment estimation is a 2014 update to the rmsprop optimizer combining it with the main feature of the momentum method 44 in this optimization algorithm running averages with exponential forgetting of both the gradients and the second moments of the gradients are used given parameters w t displaystyle w t and a loss function l t displaystyle l t where t displaystyle t indexes the current training iteration indexed at 1 displaystyle 1 adam s parameter update is given by m w t β 1 m w t 1 1 β 1 w l t 1 displaystyle m_ w t beta _ 1 m_ w t 1 left 1 beta _ 1 right nabla _ w l t 1 v w t β 2 v w t 1 1 β 2 w l t 1 2 displaystyle v_ w t beta _ 2 v_ w t 1 left 1 beta _ 2 right left nabla _ w l t 1 right 2 m w t m w t 1 β 1 t displaystyle hat m _ w t frac m_ w t 1 beta _ 1 t v w t v w t 1 β 2 t displaystyle hat v _ w t frac v_ w t 1 beta _ 2 t w t w t 1 η m w t v w t ε displaystyle w t w t 1 eta frac hat m _ w t sqrt hat v _ w t varepsilon where ε displaystyle varepsilon is a small scalar e g 10 8 displaystyle 10 8 used to prevent division by 0 and β 1 displaystyle beta _ 1 e g 0 9 and β 2 displaystyle beta _ 2 e g 0 999 are the forgetting factors for gradients and second moments of gradients respectively squaring and square rooting is done element wise as the exponential moving averages of the gradient m w t displaystyle m_ w t and the squared gradient v w t displaystyle v_ w t are initialized with a vector of 0 s there would be a bias towards zero in the first training iterations a factor 1 1 β 1 2 t displaystyle tfrac 1 1 beta _ 1 2 t is introduced to compensate this bias and get better estimates m w t displaystyle hat m _ w t and v w t displaystyle hat v _ w t the initial proof establishing the convergence of adam was incomplete and subsequent analysis has revealed that adam does not converge for all convex objectives 45 46 despite this adam continues to be used due to its strong performance in practice 47 variants edit the popularity of adam inspired many variants and enhancements some examples include nesterov enhanced gradients nadam 48 fasfa 49 varying interpretations of second order information powerpropagation 50 and adasqrt 51 using infinity norm adamax 43 amsgrad 52 which improves convergence over adam by using maximum of past squared gradients instead of the exponential average 53 adamx 54 further improves convergence over amsgrad adamw 55 which improves the weight decay sign based stochastic gradient descent edit even though sign based optimization goes back to the aforementioned rprop in 2018 researchers tried to simplify adam by removing the magnitude of the stochastic gradient from being taken into account and only considering its sign 56 57 this results in a significantly lower communication cost of transferring gradients from workers to the parameter server in this sense it serves to better compress the gradient information while having comparable convergence to standard sgd 57 this section needs expansion you can help by adding missing information june 2023 backtracking line search edit backtracking line search is another variant of gradient descent all of the below are sourced from the mentioned link it is based on a condition known as the armijo goldstein condition both methods allow learning rates to change at each iteration however the manner of the change is different backtracking line search uses function evaluations to check armijo s condition and in principle the loop in the algorithm for determining the learning rates can be long and unknown in advance adaptive sgd does not need a loop in determining learning rates on the other hand adaptive sgd does not guarantee the descent property which backtracking line search enjoys which is that f x n 1 f x n displaystyle f x_ n 1 leq f x_ n for all n if the gradient of the cost function is globally lipschitz continuous with lipschitz constant l and learning rate is chosen of the order 1 l then the standard version of sgd is a special case of backtracking line search second order methods edit a stochastic analogue of the standard deterministic newton raphson algorithm a second order method provides an asymptotically optimal or near optimal form of iterative optimization in the setting of stochastic approximation citation needed a method that uses direct measurements of the hessian matrices of the summands in the empirical risk function was developed by byrd hansen nocedal and singer 58 however directly determining the required hessian matrices for optimization may not be possible in practice practical and theoretically sound methods for second order versions of sgd that do not require direct hessian information are given by spall and others 59 60 61 a less efficient method based on finite differences instead of simultaneous perturbations is given by ruppert 62 another approach to the approximation hessian matrix is replacing it with the fisher information matrix which transforms usual gradient to natural 63 these methods not requiring direct hessian information are based on either values of the summands in the above empirical risk function or values of the gradients of the summands i e the sgd inputs in particular second order optimality is asymptotically achievable without direct calculation of the hessian matrices of the summands in the empirical risk function when the objective is a nonlinear least squares loss q w 1 n i 1 n q i w 1 n i 1 n m w x i y i 2 displaystyle q w frac 1 n sum _ i 1 n q_ i w frac 1 n sum _ i 1 n m w x_ i y_ i 2 where m w x i displaystyle m w x_ i is the predictive model e g a deep neural network the objective s structure can be exploited to estimate 2nd order information using gradients only the resulting methods are simple and often effective 64 approximations in continuous time edit for small learning rate η textstyle eta stochastic gradient descent w n n n 0 textstyle w_ n _ n in mathbb n _ 0 can be viewed as a discretization of the gradient flow ode d d t w t q w t displaystyle frac d dt w_ t nabla q w_ t subject to additional stochastic noise this approximation is only valid on a finite time horizon in the following sense assume that all the coefficients q i textstyle q_ i are sufficiently smooth let t 0 textstyle t 0 and g r d r textstyle g mathbb r d to mathbb r be a sufficiently smooth test function then there exists a constant c 0 textstyle c 0 such that for all η 0 textstyle eta 0 max k 0 t η e g w k g w k η c η displaystyle max _ k 0 dots lfloor t eta rfloor left mathbb e g w_ k g w_ k eta right leq c eta where e textstyle mathbb e denotes taking the expectation with respect to the random choice of indices in the stochastic gradient descent scheme since this approximation does not capture the random fluctuations around the mean behavior of stochastic gradient descent solutions to stochastic differential equations sdes have been proposed as limiting objects 65 more precisely the solution to the sde d w t q w t 1 4 η q w t 2 d t η σ w t 1 2 d b t displaystyle dw_ t nabla left q w_ t tfrac 1 4 eta nabla q w_ t 2 right dt sqrt eta sigma w_ t 1 2 db_ t for σ w 1 n 2 i 1 n q i w q w i 1 n q i w q w t displaystyle sigma w frac 1 n 2 left sum _ i 1 n q_ i w q w right left sum _ i 1 n q_ i w q w right t where d b t textstyle db_ t denotes the ito integral with respect to a brownian motion is a more precise approximation in the sense that there exists a constant c 0 textstyle c 0 such that max k 0 t η e g w k e g w k η c η 2 displaystyle max _ k 0 dots lfloor t eta rfloor left mathbb e g w_ k mathbb e g w_ k eta right leq c eta 2 however this sde only approximates the one point motion of stochastic gradient descent for an approximation of the stochastic flow one has to consider sdes with infinite dimensional noise 66 see also edit backtracking line search broken neural scaling law coordinate descent changes one coordinate at a time rather than one example differentially private stochastic gradient descent linear classifier online machine learning stochastic hill climbing stochastic variance reduction notes edit displaystyle odot denotes the element wise product references edit bottou léon bousquet olivier 2012 the tradeoffs of large scale learning in sra suvrit nowozin sebastian wright stephen j eds optimization for machine learning cambridge mit press pp 351 368 isbn 978 0 262 01646 9 1 2 bottou léon 1998 online algorithms and stochastic approximations online learning and neural networks cambridge university press isbn 978 0 521 65263 6 ferguson thomas s 1982 an inconsistent maximum likelihood estimate journal of the american statistical association 77 380 831 834 doi 10 1080 01621459 1982 10477894 jstor 2287314 bottou léon bousquet olivier 2008 the tradeoffs of large scale learning advances in neural information proces...