Meta tags:

Headings (most frequently used words):

least, squares, linear, relationship, to, regularization, method, contents, history, formulation, solution, example, uncertainty, quantification, statistical, testing, weighted, principal, components, measure, theory, see, also, references, further, reading, external, links, the, limitations, non, differences, between, and, nonlinear, tikhonov, lasso,

Text of the page (most frequently used words):
the (366), least (102), #squares (102), and (89), displaystyle (78), beta (75), model (59), regression (58), linear (57), for (39), that (39), analysis (36), data (35), boldsymbol (34), are (32), method (30), statistics (27), sum (26), edit (25), partial (23), errors (23), this (21), distribution (21), mathsf (21), with (20), parameters (20), statistical (19), function (18), error (18), gauss (17), isbn (17), can (17), independent (17), variance (16), mean (16), variables (16), non (15), from (15), estimator (15), variable (15), parameter (15), values (15), right (15), was (14), methods (14), estimation (14), normal (14), equations (14), 978 (14), correlation (13), residuals (13), delta (13), test (12), nonlinear (12), theorem (12), approximation (12), plot (12), fitting (12), lasso (12), best (12), zero (12), which (12), problem (12), solution (12), left (12), used (12), vector (11), generalized (11), bayesian (11), probability (11), where (11), one (11), not (11), when (11), may (10), location (10), residual (10), different (10), estimate (10), cs1 (9), observations (9), more (9), value (9), laplace (9), form (9), set (9), hat (9), about (8), using (8), maint (8), likelihood (8), general (8), series (8), unbiased (8), ridge (8), springer (8), doi (8), approach (8), estimates (8), gradient (8), regularization (8), all (8), main (8), combination (8), dependent (8), frac (8), there (8), wikipedia (7), first (7), models (7), exponential (7), multivariate (7), squared (7), fit (7), minimum (7), shape (7), theory (7), transformation (7), chart (7), numerical (7), weighted (7), also (7), some (7), its (7), differences (7), article (7), two (7), have (7), known (7), example (7), llsq (7), nllsq (7), toggle (6), link (6), algorithms (6), design (6), rank (6), time (6), density (6), principal (6), standard (6), ordinary (6), simple (6), effects (6), interval (6), van (6), prediction (6), random (6), absolute (6), arithmetic (6), markov (6), legendre (6), newton (6), history (6), formulation (6), given (6), such (6), points (6), under (5), use (5), commons (5), page (5), system (5), log (5), limit (5), var (5), statistic (5), maximum (5), selection (5), most (5), optimal (5), family (5), line (5), matrix (5), uncertainty (5), wiley (5), features (5), measurement (5), see (5), regularized (5), between (5), while (5), normally (5), distributed (5), objective (5), relationship (5), case (5), same (5), force (5), after (5), each (5), observation (5), had (5), appropriate (5), parabolic (5), work (5), table (4), contents (4), search (4), category (4), names (4), authors (4), list (4), multiple (4), missing (4), optimization (4), equation (4), retrieved (4), control (4), portal (4), applications (4), product (4), spectral (4), box (4), tests (4), components (4), poisson (4), binomial (4), logistic (4), heteroscedasticity (4), nonparametric (4), mixed (4), validation (4), alternative (4), sample (4), median (4), power (4), testing (4), estimators (4), distance (4), moments (4), point (4), loss (4), experiment (4), normalization (4), central (4), deviation (4), curve (4), predicted (4), total (4), links (4), 2008 (4), new (4), problems (4), 2013 (4), based (4), s2cid (4), learning (4), deviations (4), extension (4), penalty (4), will (4), any (4), corresponding (4), alpha (4), version (4), than (4), equivalent (4), minimization (4), solved (4), tikhonov (4), functions (4), only (4), similar (4), equal (4), found (4), spring (4), constant (4), since (4), these (4), but (4), has (4), find (4), derivatives (4), mathbf (4), iteration (4), account (4), here (4), would (4), ceres (4), his (4), hide (4), move (4), sidebar (4), subsection (4), view (3), terms (3), apply (3), wikimedia (3), categories (3), wikidata (3), index (3), mathematics (3), information (3), population (3), process (3), survival (3), conditional (3), cross (3), specific (3), adjustment (3), distributions (3), factor (3), degrees (3), freedom (3), robust (3), isotonic (3), semiparametric (3), adaptive (3), determination (3), pearson (3), moment (3), posterior (3), prior (3), inference (3), ordered (3), way (3), goodness (3), parametric (3), score (3), confidence (3), space (3), empirical (3), order (3), study (3), controlled (3), experiments (3), effect (3), scatter (3), response (3), square (3), covariance (3), polynomial (3), iteratively (3), introduction (3), john (3), sons (3), 471 (3), 1996 (3), reading (3), application (3), arxiv (3), journal (3), american (3), geer (3), sara (3), via (3), physics (3), press (3), quadratic (3), norm (3), contexts (3), solutions (3), thus (3), algorithm (3), term (3), measure (3), measured (3), create (3), out (3), other (3), variances (3), often (3), large (3), important (3), belong (3), assumption (3), uncorrelated (3), experimental (3), many (3), cases (3), usually (3), true (3), sigma (3), unknown (3), should (3), must (3), solving (3), iterative (3), convergence (3), jacobian (3), calculated (3), expressions (3), ldots (3), changes (3), closed (3), minimizes (3), get (3), phi (3), both (3), fluctuations (3), result (3), determine (3), been (3), what (3), conditions (3), probit (3), tools (3), languages (2), conduct (2), contact (2), privacy (2), policy (2), you (2), numeric (2), publisher (2), french (2), sources (2), short (2), description (2), single (2), econometrics (2), national (2), identification (2), engineering (2), studies (2), clinical (2), failure (2), proportional (2), frequency (2), domain (2), autoregressive (2), fuller (2), structural (2), smoothing (2), seasonal (2), stationarity (2), decomposition (2), cluster (2), manova (2), contingency (2), categorical (2), anova (2), homoscedasticity (2), simultaneous (2), coefficient (2), confounding (2), friedman (2), lehmann (2), bic (2), aic (2), chi (2), lagrange (2), bootstrap (2), rao (2), estimating (2), scale (2), specification (2), sampling (2), natural (2), observational (2), down (2), designs (2), trial (2), scientific (2), randomized (2), methodology (2), size (2), component (2), reduction (2), cleaning (2), unit (2), feature (2), scaling (2), transform (2), spearman (2), kendall (2), dependence (2), dispersion (2), range (2), geometric (2), outline (2), chebyshev (2), polynomials (2), orthogonal (2), studentized (2), background (2), quantile (2), local (2), segmented (2), predictor (2), reweighted (2), related (2), external (2), 2005 (2), berlin (2), 540 (2), 2021 (2), 2016 (2), beyond (2), further (2), s14 (2), international (2), bolasso (2), trevor (2), 103 (2), association (2), tibshirani (2), robert (2), 2009 (2), shrinkage (2), 1111 (2), 1987 (2), 2012 (2), 2023 (2), encyclopedia (2), oclc (2), cite (2), jeffrey (2), san (2), rafael (2), numbers (2), good (2), 1998 (2), 1805 (2), orbits (2), des (2), isaac (2), never (2), princeton (2), university (2), before (2), 1976 (2), references (2), depend (2), reason (2), reduced (2), still (2), discards (2), fully (2), coefficients (2), constraint (2), greater (2), unconstrained (2), context (2), placing (2), angle (2), constrained (2), dimension (2), minimize (2), although (2), treats (2), dimensions (2), special (2), called (2), occurs (2), diagonal (2), causes (2), fanning (2), seen (2), assuming (2), however (2), issue (2), typically (2), whether (2), make (2), assumptions (2), common (2), idea (2), made (2), assumed (2), calculation (2), minimized (2), approx (2), operatorname (2), quantification (2), derived (2), predict (2), hooke (2), law (2), our (2), kf_ (2), constitutes (2), measurements (2), specify (2), varepsilon (2), being (2), condition (2), criterion (2), satisfied (2), require (2), analytical (2), complicated (2), obtain (2), either (2), finite (2), initial (2), straight (2), become (2), next (2), then (2), number (2), aligned (2), setting (2), particular (2), negligible (2), into (2), fitted (2), rule (2), future (2), those (2), rather (2), limitations (2), were (2), plotted (2), against (2), plane (2), simplest (2), minimizing (2), consists (2), observed (2), able (2), came (2), discovered (2), astronomer (2), sun (2), motion (2), performed (2), published (2), went (2), mathematical (2), depending (2), define (2), carl (2), friedrich (2), technique (2), earth (2), taken (2), notably (2), pierre (2), simon (2), roger (2), studying (2), 1700 (2), logit (2), multinomial (2), appearance (2), upload (2), file (2), read (2), english (2), norsk (2), donate (2), menu (2), add, topic, mobile, cookie, statement, developers, code, legal, safety, contacts, disclaimers, text, available, additional, site, agree, registered, trademark, profit, organization, foundation, inc, creative, attribution, sharealike, license, rendered, parsoid, last, edited, september, 2026, utc, hidden, language, matches, articles, https, org, php, title, least_squares, oldid, 1374800160, czech, republic, japan, authority, databases, wikiproject, kriging, geostatistics, geographic, environmental, cartography, spatial, psychometrics, official, accounts, jurimetrics, demography, crime, census, actuarial, science, social, reliability, quality, probabilistic, chemometrics, medical, epidemiology, trials, bioinformatics, biostatistics, nelson, aalen, hazard, hitting, accelerated, aft, hazards, kaplan, meier, whittle, wavelet, fourier, autoregression, heteroskedasticity, arch, arima, jenkins, arma, xcf, pacf, autocorrelation, acf, breusch, godfrey, durbin, watson, ljung, johansen, dickey, granger, causality, break, cointegration, trend, elliptical, classification, discriminant, canonical, cochran, mantel, haenszel, mcnemar, graphical, cohen, kappa, ancova, partition, regressions, bernoulli, families, predictors, template, splines, mars, bayes, credible, der, waerden, jonckheere, terpstra, kruskal, wallis, mann, whitney, hodges, signed, wilcoxon, sign, normality, shapiro, wilk, jarque, bera, lilliefors, anderson, darling, kolmogorov, smirnov, student, wald, multiplier, ratio, comparisons, randomization, permutation, uniformly, powerful, tails, hypotheses, jackknife, resampling, tolerance, pivot, plug, scheffé, blackwellization, frequentist, sensitivity, robustness, asymptotics, divergence, efficiency, decision, functional, sufficiency, completeness, monotone, quasi, sectional, cohort, stochastic, assignment, interaction, factorial, blocking, questionnaire, opinion, poll, stratified, survey, replication, collection, detrending, differencing, preprocessing, dimensionality, truncation, winsorizing, outlier, min, max, standardization, fisher, anscombe, stabilizing, yeo, johnson, cox, transformations, processing, ecdf, heatmap, violin, stem, leaf, display, run, radar, pie, histogram, forest, fan, correlogram, biplot, bar, graphics, grouped, summary, tables, count, skewness, kurtosis, percentile, interquartile, variation, average, mode, lehmer, heinz, heronian, harmonic, cubic, contraharmonic, center, continuous, descriptive, topics, moving, differentiation, calibration, nodes, gaussian, quadrature, integration, surface, frisch, waugh, lovell, mallows, stepwise, exploration, aov, growth, structure, tau, rho, computational, media, wolberg, 25674, extracting, moortel, koen, april, multidirectional, strutz, 2nd, vieweg, 658, 11455, practical, 3rd, 74226, alternatives, toutenburg, 1997, 1969, york, 102, 18117, luenberger, kariya, kurata, 2004, hoboken, 470, 86697, björck, siam, 89871, 360, zare, habil, suppl, 23369194, pmid, 3549810, pmc, 1186, 1471, 2164, bmc, genomics, scoring, relevancy, combinatorial, lymphoma, diagnosis, bach, francis, 609778, 9781605582054, 1145, 1390156, 1390161, 2008arxiv0804, 1302b, bibcode, 0804, 1302, proceedings, 25th, conference, machine, icml, park, 482, 686, 11797924, 1198, 016214508000000337, 681, casella, george, bühlmann, peter, 2011, 9783642201929, high, dimensional, second, verlag, archived, original, 387, 84858, elements, jerome, hastie, 288, 2346178, jstor, 2517, 6161, tb02080, 267, royal, society, wieringen, wessel, lecture, notes, stat, 1509, 09169, june, 602, 123088844, 1214, aos, 1176350362, 587, annals, hallin, marc, october, 89997, 1002, 9780470057339, vnn102, environmetrics, gere, james, goodno, barry, 8th, stamford, conn, cengage, 741541348, 111, 57773, mechanics, materials, rencher, alvin, christensen, william, 155, 118, 39167, book, williams, huw, 1956, november, morgan, claypool, publishers, institute, great, britain, california, oak, drive, 94903, usa, 962422324, 68174, 433, quantifying, tyranny, please, 86187, dekking, frederik, michel, kraaikamp, cornelis, lopuhaä, hendrik, rik, meester, ludolf, texts, 9781852338961, publishing, modern, understanding, why, how, aldrich, doing, perspectives, yule, 121471194, 1751, 5823, tb00406, review, 2015, priceonomics, discovery, adrien, marie, paris, didot, 2027, nyp, 33433069112559, hdl, comets, nouvelles, méthodes, pour, détermination, orbites, comètes, belenkiy, ari, echague, eduardo, vila, groping, toward, hipparchus, equinox, hist, 0810, 4948, drum, kevin, 2024, mother, jones, groundbreaking, invention, heard, buchwald, jed, feingold, mordechai, oxford, 691, 15478, 101, origin, civilization, stigler, stephen, 1986, cambridge, belknap, harvard, 674, 40340, 1900, charnes, frome, equivalence, 353, 171, 1080, 01621459, 10481508, 169, root, proximal, projection, blup, blue, mmse, useful, due, tendency, prefer, gives, fewer, variants, fundamental, field, elastic, net, compressed, sensing, prime, increased, remaining, increasing, cause, driven, advantage, over, driving, deselects, automatically, selects, relevant, others, whereas, techniques, developed, including, bootstraps, samples, fealect, analyzes, operator, uses, show, like, above, multipliers, added, well, convex, programming, preferable, adds, leading, plus, tuning, lagrangian, ell, notable, statistician, prove, interpreted, integrable, vapnik, chervonenkis, represented, closely, approaches, closest, perpendicular, contrast, tries, direction, metric, preferentially, pca, equally, off, entries, null, along, unequal, simpler, depends, towards, larger, smaller, hand, suppose, nonetheless, implies, approximately, long, reasonably, property, specifically, follows, means, valid, expectation, necessary, nature, results, statistically, supports, asymptotic, similarly, conducted, derive, inferring, easy, follow, consequently, implying, limits, weights, denoted, estimated, replaced, denominator, generalizations, effective, assume, applying, expand, having, might, comprise, overdetermined, consider, drawn, obey, states, applied, forces, produce, contain, dots, considered, whenever, sought, yields, even, generally, biased, unique, minima, terminated, computed, direct, seidel, globally, concave, phenomenon, impossible, need, does, them, represent, parabola, appear, forth, otherwise, cdots, defining, written, notation, rearrangement, qquad, constants, involve, choosing, refined, obtained, successive, superscript, increments, shift, commonly, linearized, expansion, begin, 1ex, end, taylor, pseudoinverse, rightarrow, letting, putting, matrices, respectively, compute, following, note, comprises, requires, contains, leads, implicit, strictly, lead, take, presence, viewed, taking, pragmatic, balancing, formulating, intervals, hypothesis, provide, situation, subject, types, therefore, logically, consistent, considers, implications, sort, randomly, fluctuating, indicates, illustrating, indicating, consist, height, say, denoting, slope, worked, intercept, input, finds, indicate, adjusting, pairs, whose, adjustable, held, goal, fits, defined, difference, independently, formulated, 1808, centuries, workers, ways, implementing, adrain, 1810, proving, give, justification, 1822, state, sense, extended, early, demonstration, strength, newly, asteroid, january, 1801, italian, track, path, days, lost, glare, astronomers, desired, emerged, behind, without, planetary, predictions, successfully, allowed, hungarian, relocate, year, old, franz, xaver, von, zach, kepler, giuseppe, piazzi, 1809, calculating, celestial, bodies, claimed, possession, 1795, naturally, led, priority, dispute, credit, succeeded, connecting, principles, managed, complete, program, specifying, showed, indeed, changing, turned, around, asking, attempt, invented, clear, concise, exposition, described, algebraic, procedure, demonstrates, analyzing, within, ten, years, publication, adopted, tool, astronomy, extraordinarily, rapid, acceptance, prussia, italy, france, geodesy, development, evaluated, achieved, tried, purpose, symmetric, sided, now, call, felt, could, hoped, instead, 1757, 1789, 1799, joseph, boscovich, contrary, simply, trying, observe, record, accurately, averages, equinoxes, writing, moon, 1750, explaining, 1788, saturn, jupiter, librations, tobias, mayer, decrease, aggregation, increase, appeared, 1671, though, unpublished, again, perhaps, expressed, formally, 1722, cotes, culmination, several, advances, took, place, during, course, eighteenth, century, come, identity, sufficient, mild, standardized, identical, describes, fall, unknowns, approximated, core, refinement, negative, fixed, multilevel, binary, discrete, choice, part, conic, redirects, confused, free, item, projects, printable, download, pdf, print, export, switch, legacy, parser, shortened, url, permanent, actions, talk, tiếng, việt, oʻzbekcha, ўзбекча, اردو, українська, türkçe, ไทย, svenska, sunda, српски, srpski, slovenščina, русский, română, português, polski, bokmål, nynorsk, nederlands, македонски, latina, 한국어, қазақша, 日本語, italiano, հայերեն, magyar, हिन्दी, עברית, galego, français, suomi, فارسی, euskara, eesti, español, deutsch, dansk, čeština, català, български, azərbaycanca, العربية, afrikaans, top, personal, pages, recent, community, learn, help, contribute, current, events, navigation, jump, content,

Text of the page (random words):
re non negligible models of measurement error can be used such methods can lead to parameter estimates hypothesis testing and confidence intervals that take into account the presence of observation errors in the independent variables 10 an alternative approach is to fit a model by total least squares this can be viewed as taking a pragmatic approach to balancing the effects of the different sources of error in formulating an objective function for use in model fitting solution edit the minimum of the sum of squares is found by setting the gradient to zero since the model contains m parameters there are m gradient equations s β j 2 i r i r i β j 0 j 1 m displaystyle frac partial s partial beta _ j 2 sum _ i r_ i frac partial r_ i partial beta _ j 0 j 1 ldots m and since r i y i f x i β displaystyle r_ i y_ i f x_ i boldsymbol beta the gradient equations become 2 i r i f x i β β j 0 j 1 m displaystyle 2 sum _ i r_ i frac partial f x_ i boldsymbol beta partial beta _ j 0 j 1 ldots m the gradient equations apply to all least squares problems each particular problem requires particular expressions for the model and its partial derivatives 11 linear least squares edit main article linear least squares a regression model is a linear one when the model comprises a linear combination of the parameters i e f x β j 1 m β j ϕ j x displaystyle f x boldsymbol beta sum _ j 1 m beta _ j phi _ j x where the function ϕ j displaystyle phi _ j is a function of x displaystyle x 11 letting x i j ϕ j x i displaystyle x_ ij phi _ j x_ i and putting the independent and dependent variables in matrices x displaystyle x and y displaystyle y respectively we can compute the least squares in the following way note that d displaystyle d is the set of all data 11 12 l d β y x β 2 y x β t y x β displaystyle l d boldsymbol beta left y x boldsymbol beta right 2 y x boldsymbol beta mathsf t y x boldsymbol beta y t y 2 y t x β β t x t x β displaystyle y mathsf t y 2y mathsf t x boldsymbol beta boldsymbol beta mathsf t x mathsf t x boldsymbol beta the gradient of the loss is l d β β y t y 2 y t x β β t x t x β β 2 x t y 2 x t x β displaystyle frac partial l d boldsymbol beta partial boldsymbol beta frac partial left y mathsf t y 2y mathsf t x boldsymbol beta boldsymbol beta mathsf t x mathsf t x boldsymbol beta right partial boldsymbol beta 2x mathsf t y 2x mathsf t x boldsymbol beta setting the gradient of the loss to zero and solving for β displaystyle boldsymbol beta we get 12 11 2 x t y 2 x t x β 0 x t y x t x β displaystyle 2x mathsf t y 2x mathsf t x boldsymbol beta 0 rightarrow x mathsf t y x mathsf t x boldsymbol beta β x t x 1 x t y x y displaystyle boldsymbol hat beta left x mathsf t x right 1 x mathsf t y x y where x displaystyle x is the pseudoinverse of x displaystyle x non linear least squares edit main article non linear least squares there is in some cases a closed form solution to a non linear least squares problem but in general there is not in the case of no closed form solution numerical algorithms are used to find the value of the parameters β displaystyle beta that minimizes the objective most algorithms involve choosing initial values for the parameters then the parameters are refined iteratively that is the values are obtained by successive approximation β j k 1 β j k δ β j displaystyle beta _ j k 1 beta _ j k delta beta _ j where a superscript k is an iteration number and the vector of increments δ β j displaystyle delta beta _ j is called the shift vector in some commonly used algorithms at each iteration the model may be linearized by approximation to a first order taylor series expansion about β k displaystyle boldsymbol beta k f x i β f k x i β j f x i β β j β j β j k f k x i β j j i j δ β j displaystyle begin aligned f x_ i boldsymbol beta f k x_ i boldsymbol beta sum _ j frac partial f x_ i boldsymbol beta partial beta _ j left beta _ j beta _ j k right 1ex f k x_ i boldsymbol beta sum _ j j_ ij delta beta _ j end aligned the jacobian j is a function of constants the independent variable and the parameters so it changes from one iteration to the next the residuals are given by r i y i f k x i β k 1 m j i k δ β k δ y i j 1 m j i j δ β j displaystyle r_ i y_ i f k x_ i boldsymbol beta sum _ k 1 m j_ ik delta beta _ k delta y_ i sum _ j 1 m j_ ij delta beta _ j to minimize the sum of squares of r i displaystyle r_ i the gradient equation is set to zero and solved for δ β j displaystyle delta beta _ j 2 i 1 n j i j δ y i k 1 m j i k δ β k 0 displaystyle 2 sum _ i 1 n j_ ij left delta y_ i sum _ k 1 m j_ ik delta beta _ k right 0 which on rearrangement become m simultaneous linear equations the normal equations i 1 n k 1 m j i j j i k δ β k i 1 n j i j δ y i j 1 m displaystyle sum _ i 1 n sum _ k 1 m j_ ij j_ ik delta beta _ k sum _ i 1 n j_ ij delta y_ i qquad j 1 ldots m the normal equations are written in matrix notation as j t j δ β j t δ y displaystyle left mathbf j mathsf t mathbf j right delta boldsymbol beta mathbf j mathsf t delta mathbf y these are the defining equations of the gauss newton algorithm differences between linear and nonlinear least squares edit the model function f in llsq linear least squares is a linear combination of parameters of the form f x i 1 β 1 x i 2 β 2 displaystyle f x_ i1 beta _ 1 x_ i2 beta _ 2 cdots the model may represent a straight line a parabola or any other linear combination of functions in nllsq nonlinear least squares the parameters appear as functions such as β 2 e β x displaystyle beta 2 e beta x and so forth if the derivatives f β j displaystyle partial f partial beta _ j are either constant or depend only on the values of the independent variable the model is linear in the parameters otherwise the model is nonlinear need initial values for the parameters to find the solution to a nllsq problem llsq does not require them solution algorithms for nllsq often require that the jacobian can be calculated similar to llsq analytical expressions for the partial derivatives can be complicated if analytical expressions are impossible to obtain either the partial derivatives must be calculated by numerical approximation or an estimate must be made of the jacobian often via finite differences non convergence failure of the algorithm to find a minimum is a common phenomenon in nllsq llsq is globally concave so non convergence is not an issue solving nllsq is usually an iterative process which has to be terminated when a convergence criterion is satisfied llsq solutions can be computed using direct methods although problems with large numbers of parameters are typically solved with iterative methods such as the gauss seidel method in llsq the solution is unique but in nllsq there may be multiple minima in the sum of squares under the condition that the errors are uncorrelated with the predictor variables llsq yields unbiased estimates but even under that condition nllsq estimates are generally biased these differences must be considered whenever the solution to a nonlinear least squares problem is being sought 11 example edit consider a simple example drawn from physics a spring should obey hooke s law which states that the extension of a spring y is proportional to the force f applied to it y f f k k f displaystyle y f f k kf constitutes the model where f is the independent variable in order to estimate the force constant k we conduct a series of n measurements with different forces to produce a set of data f i y i i 1 n displaystyle f_ i y_ i i 1 dots n where y i is a measured spring extension 13 each experimental observation will contain some error ε displaystyle varepsilon and so we may specify an empirical model for our observations y i k f i ε i displaystyle y_ i kf_ i varepsilon _ i there are many methods we might use to estimate the unknown parameter k since the n equations in the m variables in our data comprise an overdetermined system with one unknown and n equations we estimate k using least squares the sum of squares to be minimized is 11 s i 1 n y i k f i 2 displaystyle s sum _ i 1 n left y_ i kf_ i right 2 the least squares estimate of the force constant k is given by k i f i y i i f i 2 displaystyle hat k frac sum _ i f_ i y_ i sum _ i f_ i 2 we assume that applying force causes the spring to expand after having derived the force constant by least squares fitting we predict the extension from hooke s law uncertainty quantification edit in a least squares calculation with unit weights or in linear regression the variance on the j th parameter denoted var β j displaystyle operatorname var hat beta _ j is usually estimated with var β j σ 2 x t x 1 j j σ 2 c j j displaystyle operatorname var hat beta _ j sigma 2 left left x mathsf t x right 1 right _ jj approx hat sigma 2 c_ jj σ 2 s n m displaystyle hat sigma 2 approx frac s n m c x t x 1 displaystyle c left x mathsf t x right 1 where the true error variance σ 2 is replaced by an estimate the reduced chi squared statistic based on the minimized value of the residual sum of squares objective function s the denominator n m is the statistical degrees of freedom see effective degrees of freedom for generalizations 11 c is the covariance matrix statistical testing edit if the probability distribution of the parameters is known or an asymptotic approximation is made confidence limits can be found similarly statistical tests on the residuals can be conducted if the probability distribution of the residuals is known or assumed we can derive the probability distribution of any linear combination of the dependent variables if the probability distribution of experimental errors is known or assumed inferring is easy when assuming that the errors follow a normal distribution consequently implying that the parameter estimates and residuals will also be normally distributed conditional on the values of the independent variables 11 it is necessary to make assumptions about the nature of the experimental errors to test the results statistically a common assumption is that the errors belong to a normal distribution the central limit theorem supports the idea that this is a good approximation in many cases the gauss markov theorem in a linear model in which the errors have expectation zero conditional on the independent variables are uncorrelated and have equal variances the best linear unbiased estimator of any linear combination of the observations is its least squares estimator best means that the least squares estimators of the parameters have minimum variance the assumption of equal variance is valid when the errors all belong to the same distribution 14 if the errors belong to a normal distribution the least squares estimators are also the maximum likelihood estimators in a linear model however suppose the errors are not normally distributed in that case a central limit theorem often nonetheless implies that the parameter estimates will be approximately normally distributed so long as the sample is reasonably large for this reason given the important property that the error mean is independent of the independent variables the distribution of the error term is not an important issue in regression analysis specifically it is not typically important whether the error term follows a normal distribution weighted least squares edit fanning out effect of heteroscedasticity main article weighted least squares a special case of generalized least squares called weighted least squares occurs when all the off diagonal entries of ω the correlation matrix of the residuals are null the variances of the observations along the covariance matrix diagonal may still be unequal heteroscedasticity in simpler terms heteroscedasticity is when the variance of y i displaystyle y_ i depends on the value of x i displaystyle x_ i which causes the residual plot to create a fanning out effect towards larger or smaller y i displaystyle y_ i values as seen in the residual plot to the right on the other hand homoscedasticity is assuming that the variance of y i displaystyle y_ i and variance of u i displaystyle u_ i are equal 9 relationship to principal components edit the first principal component about the mean of a set of points can be represented by that line which most closely approaches the data points as measured by squared distance of closest approach i e perpendicular to the line in contrast linear least squares tries to minimize the distance in the y displaystyle y direction only thus although the two use a similar error metric linear least squares is a method that treats one dimension of the data preferentially while pca treats all dimensions equally relationship to measure theory edit notable statistician sara van de geer used empirical process theory and the vapnik chervonenkis dimension to prove a least squares estimator can be interpreted as a measure on the space of square integrable functions 15 regularization edit main article regularized least squares tikhonov regularization edit main article tikhonov regularization in some contexts a regularized version of the least squares solution may be preferable tikhonov regularization or ridge regression adds a constraint that β 2 2 displaystyle left beta right _ 2 2 the squared ℓ 2 displaystyle ell _ 2 norm of the parameter vector is not greater than a given value to the least squares formulation leading to a constrained minimization problem this is equivalent to the unconstrained minimization problem where the objective function is the residual sum of squares plus a penalty term α β 2 2 displaystyle alpha left beta right _ 2 2 and α displaystyle alpha is a tuning parameter this is the lagrangian form of the constrained minimization problem 16 in a bayesian context this is equivalent to placing a zero mean normally distributed prior on the parameter vector lasso method edit an alternative regularized version of least squares is lasso least absolute shrinkage and selection operator which uses the constraint that β 1 displaystyle beta _ 1 the l 1 norm of the parameter vector is no greater than a given value 17 18 19 one can show like above using lagrange multipliers that this is equivalent to an unconstrained minimization of the least squares penalty with α β 1 displaystyle alpha beta _ 1 added in a bayesian context this is equivalent to placing a zero mean laplace prior distribution on the parameter vector 20 the optimization problem may be solved using quadratic programming or more general convex optimization methods as well as by specific algorithms such as the least angle regression algorithm one of the prime differences between lasso and ridge regression is that in ridge regression as the penalty is increased all parameters are reduced while still remaining non zero while in lasso increasing the penalty will cause more and more of the parameters to be driven to zero this is an advantage of lasso over ridge regression as driving parameters to zero deselects the features from the regression thus lasso automati...