Meta tags:
author=Gwern;
description=Cookbook of code for common selection & order statistics scenarios in economics, psychology, decision theory, drug development etc.;
keywords=cs/r, psychology/energy, statistics/order;

Headings (most frequently used words):

selection, max, simple, truncation, binary, common, scenarios, the, general, scenario, similar, links, multiple, places, tournament, variable, pipeline, index,

Text of the page (most frequently used words):
the (128), #selection (34), and (32), can (21), utility (19), max (19), that (18), for (17), threshold (14), may (14), are (14), expected (12), all (12), with (11), have (11), only (10), not (10), because (10), scenarios (10), one (9), variable (9), from (9), binary (9), simple (9), probability (9), than (8), what (8), place (8), gain (8), this (7), then (7), but (7), increases (7), here (6), how (6), normal (6), truncation (6), has (6), tournament (6), such (6), measurement (6), risk (6), small (6), similar (5), high (5), more (5), like (5), example (5), out (5), disease (5), mean (5), etc (5), value (5), our (5), given (5), they (5), large (5), typically (5), base (5), rate (5), matter (5), links (4), almost (4), enough (4), low (4), while (4), even (4), time (4), scenario (4), most (4), screening (4), random (4), datapoints (4), many (4), being (4), other (4), pipeline (4), model (4), where (4), new (4), embryo (4), does (4), simply (4), places (4), decision (4), day (3), javascript (3), useful (3), auc (3), look (3), you (3), dog (3), training (3), latent (3), pass (3), military (3), variables (3), bad (3), fail (3), also (3), probit (3), baserate (3), problem (3), distribution (3), top (3), stages (3), which (3), drug (3), apple (3), would (3), samples (3), when (3), instead (3), outcome (3), goes (3), number (3), selecting (3), will (3), percentage (3), statistics (3), common (3), annotation (2), site (2), hairsp (2), backlinks (2), disabled (2), index (2), any (2), roc (2), set (2), success (2), regardless (2), fall (2), cloning (2), successful (2), screen (2), thousands (2), dogs (2), over (2), plausible (2), get (2), analysis (2), selected (2), trait (2), passing (2), applications (2), into (2), score (2), either (2), etruncnorm (2), again (2), continuous (2), liability (2), changes (2), except (2), importance (2), 100 (2), truncated (2), regressed (2), usual (2), multiplied (2), sample (2), average (2), datapoint (2), estimate (2), easily (2), take (2), net (2), points (2), empty (2), handed (2), must (2), each (2), human (2), likely (2), far (2), starting (2), applying (2), candidates (2), applies (2), development (2), apply (2), sometimes (2), closer (2), families (2), extremely (2), parents (2), them (2), already (2), parameters (2), very (2), minimal (2), reduction (2), absolute (2), outcomes (2), others (2), case (2), there (2), better (2), maximized (2), some (2), its (2), maximum (2), drawing (2), expectedmax (2), want (2), minimize (2), way (2), multiple (2), fact (2), too (2), performance (2), regress (2), calculation (2), dependence (2), influenced (2), implicitly (2), defined (2), general (2), dismiss (2), claims (2), without (2), statistical (2), looks (2), variance (2), really (2), order (2), adblock, public, service, announcement, quote, send, anonymous, feedback, topic, bibliography, require, enabled, load, error, implication, medicine, sufficiently, germane, staring, charts, saying, doesn, big, possible, surprise, irrelevant, nearly, guaranteed, short, requires, tens, decades, select, donors, 4sd, potentially, heritabilities, automatically, clones, approaching, 3sd, who, hit, 1sd, luck, failing, succeeding, elite, medical, attempting, predict, those, certain, percentile, targeted, interventions, might, looking, rather, add, influence, resembles, course, calculating, size, extreme, corresponds, using, quantile, function, assumed, lower, bound, minimum, corresponding, could, monte, carlo, longer, depends, drops, concerned, demand, 001, few, thousand, come, actually, engaged, logic, flips, series, provides, predicting, final, patients, chances, total, smaller, intuitively, expect, rapid, collapse, accuracy, apparently, dazzling, futile, repeatedly, candidate, otherwise, dropped, favor, promising, pipelines, proceeds, software, vitro, tests, animal, models, clinical, trials, phases, breeding, create, seedlings, masse, fields, breeders, plant, bite, uniform, times, concentrates, inside, different, profiles, population, schizophrenia, little, healthy, reduce, happen, schizophrenics, halve, enormous, quite, after, perfection, save, good, none, worse, inverse, flipped, power, approaches, start, brings, wrinkles, own, now, influences, constant, offset, draw, push, thus, dis, minimizing, turned, maximizing, flipping, adding, multiply, through, transformation, gets, larger, desirable, although, never, above, assumes, normally, distributed, discrete, chance, developing, long, fixed, acts, same, implicit, rapidly, eventually, hitting, diminishing, returns, matters, reduces, just, placer, hadn, been, correlated, desired, critical, outweigh, decreases, unless, become, worth, noting, monotonically, metrics, regret, increase, asymptotically, approach, equivalent, guessing, becomes, scoring, true, falls, magnitude, always, style, gives, sds, per, multiples, formula, perfect, intrinsically, noisy, phenotypic, interested, underlying, genetics, imperfect, reliability, means, clear, thanks, joint, zeroed, opaque, hidden, away, specific, less, 1000, best, past, expressed, procedure, baseline, procedures, hunter, schmidt, reduced, dismissed, inadequate, additional, assumptions, about, applied, purpose, whom, valuing, ways, completely, useless, correlations, ultimate, criterion, truth, equally, both, invention, perpetual, motion, machine, difference, mouse, maze, running, blanket, dismissals, calculations, dogma, masquerading, neutral, numbers, malpractice, these, deeply, misleading, often, wrong, smuggle, their, answer, back, door, jumping, bare, correlation, theoretic, conclusion, unprofitable, maker, action, based, information, omitting, relevant, premises, single, digit, should, write, takes, digits, yet, convert, bigger, point, curves, addition, reporting, surely, wasn, appreciate, esthetics, wiggly, lines, 000, million, measuring, whether, adolescent, acne, alzheimer, further, priori, remarkable, saving, know, why, critics, tools, grounds, explained, curve, highly, certainty, notes, 2021, cookbook, code, economics, psychology, theory, mental, energy, substack, patreon, blog, support, key, link, popups, popovers, transclusions, collapsible, sections, tablesorting, image, zooming, enable, sidenotes, website, features, warning, skip, main, content, gwern,

Text of the page (random words):
common selection scenarios gwern net skip to main content warning javascript disabled for support of key website features link annotation popups popovers transclusions collapsible sections backlinks tablesorting image zooming sidenotes etc you must enable javascript site me new blog links patreon substack common selection scenarios cs r mental energy order statistics cookbook of code for common selection order statistics scenarios in economics psychology decision theory drug development etc 2021 06 11 notes certainty highly likely importance 5 similar the general selection scenario simple max multiple places tournament max binary variable simple max pipeline truncation selection truncation selection binary index selection many applications of statistics are implicitly selection scenarios such as disease screening critics sometimes dismiss tools on grounds such as the variance explained is only 5 or the auc curve is only 75 why is 5 not enough is it because it is a single digit number and looks small should i write it instead as 0 05 because that takes 3 digits or better yet convert it to r instead because a r 0 22 looks bigger than 0 05 at what percentage of variance would it be enough what is the point of drawing roc auc curves in addition to reporting a r 2 surely it wasn t to appreciate the esthetics of wiggly lines does it really not matter if i am applying this to 2 2 000 or 2 million datapoints and does it really not matter what i am measuring at all that no matter where or when or what the trait is whether it s adolescent acne or alzheimer s disease we know without any further analysis a priori that 5 is too small what a remarkable time saving number that 5 is these claims are all deeply misleading and often wrong as they smuggle in their answer by the back door jumping from a bare statistical fact the correlation is r to a decision theoretic conclusion it is unprofitable for the decision maker to take action a based on the information of r while omitting the decision relevant premises a 5 r 2 can only be dismissed as inadequate given additional assumptions about how it is applied to how many for what purpose and by whom valuing the outcomes in such ways for some scenarios 5 is extremely useful for other scenarios it is completely useless we can no more dismiss all correlations of r 0 05 for being 0 05 than we can apply p 0 05 as the ultimate criterion of truth to all claims equally both the invention of a perpetual motion machine and a difference in mouse maze running time blanket dismissals without calculations are dogma masquerading as neutral numbers and statistical malpractice the general selection scenario the utility of a selection procedure is defined by the gain over a baseline typically random selection selection procedures as defined by hunter schmidt can be reduced to 3 parameters the threshold the expected gain of selecting the max and the utility of a given gain sd scenarios such as selection of the best of n selection of all past a sd or percentage selection of datapoints which regress etc can be expressed this way the dependence on utility is clear regardless of how large the sd gain is thanks to the joint n r it can be zeroed out by the utility the dependence of the expected gain and threshold on n is more opaque and hidden away in our calculation of the sd gain because it is scenario specific it may be influenced by the n for example in simple selection of max the threshold implicitly increases with n 1 of 2 is far less a threshold than 1 of 1000 but the threshold may be set as a percentage or absolute score and not be influenced by n at all for the usual case where we do not have a perfect measurement of performance because performance is variable because the measurement is intrinsically noisy because the measurement is phenotypic while we are interested in the underlying genetics the imperfect reliability simply means that we regress each measurement to the mean and do our utility calculation on that simple max in tournament style selection our threshold is the expected value of the maximum of n regressed by r and that gives the expected gain in sds and if we have gain per sd then that multiples too so the formula is simply expectedmax n r utility it is worth noting that while the expected utility goes up monotonically in n and r other metrics like probability of selecting the max or regret will increase p max will in fact asymptotically approach 1 n equivalent to random guessing no matter r or how large the expected utility of selection becomes the probability of the top scoring datapoint being the true max falls as n increases even as its magnitude always goes up in selecting 1 out of n with a measurement correlated r with the desired latent variable the most critical variable is typically not r but n small changes in n will outweigh large increases decreases in r unless r is small or n has become large multiple places tournament if our tournament has places like 1 st 3 rd place it reduces to simple max but on n place 1 1 st place in a tournament of n is just the max of n 2 nd place is the max of n 1 because if the 1 st placer hadn t been in there 2 nd place would be 1 st place 3 rd place is max of n 2 etc as long as the number of places is fixed then it acts in the same way as simple max the implicit threshold increases with n and so the expected value of the place increases most rapidly with n eventually hitting diminishing returns where r matters more max binary variable the above assumes a normally distributed continuous variable being maximized many variables are discrete such as binary and we want to minimize them for example in embryo selection we want to minimize the chance of developing a given disease typically one either has it or not minimizing can be turned into maximizing by flipping it and adding a threshold for the liability threshold model the normal variable does not simply multiply by utility but goes through a transformation to a binary outcome as the normal gets larger the probability of the desirable outcome increases although never to 100 the utility here is simply the dis utility of the binary multiplied by the probability probit expectedmax n mean probit baserate r utility this brings in some new wrinkles of its own now the base rate influences the expected maximum as a constant offset instead of drawing datapoints from n 0 1 we draw from n base rate 1 the base rate may push the mean of all samples high or low and thus samples may already have a high or low probability of the bad outcome if for example samples have a mean probability of 99 then the utility from selection may be quite minimal after all even perfection can only save 1 bad outcomes almost all are good none worse than the others in the inverse case of 1 the problem is flipped in a given sample there may be no points which are better than the others the power of selection here is maximized when the base rate approaches 50 that is when selection is most useful and n r start to matter again the scenarios one applies this to may not be uniform in starting probability sometimes the base rate may be 1 other times it may be closer to 50 an example here would be embryo selection disease risk typically concentrates inside families and families can have extremely different risk profiles if 2 parents are selected at random they may have only the average 1 population risk of schizophrenia and so embryo selection does little for them they are already healthy embryo selection with plausible parameters can only reduce the risk from very small to very small and the utility is minimal but if the 2 parents happen to be schizophrenics then one can estimate the risk at closer to 50 and the reduction from selection can easily halve that risk to 25 and the utility of an absolute reduction of 25 is enormous simple max pipeline in the pipeline model of selection we are repeatedly applying r to k stages and a candidate has value only if they pass all k stages as the max otherwise they are dropped in favor of more promising candidates this model applies to drug development pipelines where a drug proceeds from software screening to in vitro tests to animal models to human clinical trials in 3 phases it can also apply to scenarios like apple breeding because one can create new apple seedlings en masse in large fields of candidates apple breeders typically pass fail a new plant with one bite in the pipeline model the logic of screening flips we may have a series of stages each of which provides a measurement predicting with r the final utility in human patients but r has k chances to fail so our total r k is likely far smaller than we intuitively expect given the rapid collapse of selection accuracy even apparently dazzling increases in a starting n are futile truncation selection in truncation selection we take the top x of datapoints such as 1 or 10 here the threshold no longer depends on n and n drops out except as we are concerned with how many we get out of selection on net if we demand the top 0 001 and only screen a few thousand points we may come up empty handed and if being empty handed is a problem then we must actually be engaged in one of the other scenarios the probability of a random sample passing is of course x the problem here is calculating what the expected size of the average datapoint as extreme as x we can estimate what an x corresponds to by using the quantile function on the assumed distribution but that is only a lower bound because that is the minimum for the normal distribution we can look up the expected value of the corresponding truncated normal distribution we could also monte carlo it easily the expected value of the truncated normal is then regressed as usual and multiplied by the utility etruncnorm x r utility the influence of the 3 variables resembles the simple max tournament except that x is similar to 1 n in importance 10 is 1 of 10 1 is 1 of 100 etc truncation selection binary again we might be looking at a binary variable rather than continuous like simple max we add on the liability threshold to look at changes in success probability probit etruncnorm x mean baserate r baserate utility for example in my dog cloning analysis we have a truncation selection elite military dogs 1 selected out of thousands on a binary trait passing training to be a successful military dog this is also the scenario that most medical screening applications fall into one has a score attempting to predict a disease you either have or have not and those in a certain percentile are targeted for interventions one possible surprise here is that r can be almost irrelevant if one can set the threshold high enough success can be nearly guaranteed regardless of how low r is while even r 1 may fall short if the threshold is low enough so in dog cloning successful training requires a latent variable 2 or so as only 10 may pass we can screen from tens of thousands of military dogs over the decades to select donors with expected normal latent variables of training as high as 4sd potentially with plausible heritabilities like 50 r 0 7 we then automatically get clones approaching 3sd who have to be hit by 1sd of bad luck to fail we go from failing almost all of the time to succeeding almost all of the time an implication for medicine is that any r can be useful if the threshold is sufficiently high this is more germane than staring at auc roc charts and saying that it doesn t look big to you index selection error javascript disabled backlinks similar links and the bibliography require js enabled to load similar links similar links by topic hairsp send anonymous feedback hairsp quote of the day site of the day annotation of the day adblock public service announcement