Meta tags:
author= Gwern;
description= Compilation of studies comparing observational results with randomized experimental results on the same intervention, compiled from medicine/economics/psychology, indicating that a large fraction of the time (although probably not a majority) correlation ≠ causality.;
keywords= economics, insight-porn, psychology/cognitive-bias, statistics/bayes/regression-to-mean, statistics/bias, statistics/causality;
Headings (most frequently used words):
how, often, does, correlation, causality, medical, economics, sociology, psychology, education, todo, backlinks, similar, links, bibliography,
Text of the page (most frequently used words):
the (1183), and (575), for (318), studies (247), that (211), randomized (202), from (150), estimates (147), were (142), with (134), results (115), non (108), was (105), experimental (102), are (90), between (86), effects (85), this (82), observational (80), not (78), trials (76), estimate (76), effect (71), nonexperimental (71), treatment (69), than (69), table (64), data (63), these (62), control (61), meta (60), study (60), outcomes (58), group (57), which (55), comparisons (54), impact (52), program (51), have (50), our (50), other (47), but (47), nonrandomized (47), comparison (46), bias (46), differences (46), two (46), difference (46), more (45), they (44), analysis (43), different (43), significant (43), all (42), risk (41), there (39), each (37), design (37), size (37), reviews (37), had (36), evidence (34), use (34), been (34), school (34), used (33), relative (33), average (33), one (33), controlled (33), across (32), assignment (31), would (31), compared (31), interventions (31), earnings (31), health (30), also (30), using (30), may (30), spending (30), medical (30), rcts (30), clinical (29), experiments (29), only (29), significantly (29), intervention (29), participants (29), research (28), same (28), groups (28), both (27), reported (27), trial (27), random (26), test (26), large (26), selection (26), review (25), class (25), some (25), analyses (25), did (25), confidence (24), assigned (24), included (24), mean (24), teacher (23), those (23), estimated (23), scores (23), found (23), statistically (23), case (23), training (22), programs (22), methods (22), can (22), findings (22), standard (22), experiment (22), based (22), example (22), within (22), therapy (21), outcome (21), characteristics (21), who (21), such (21), similar (20), student (20), out (20), among (20), over (20), first (20), population (20), plan (20), topic (19), sizes (19), systematic (19), cancer (19), students (19), about (19), how (19), could (19), into (19), evaluation (18), prior (18), regression (18), their (18), randomly (18), when (18), four (18), larger (18), published (18), measures (17), small (17), smaller (17), column (17), topics (17), showed (17), many (16), performance (16), classes (16), however (16), higher (16), points (16), score (16), level (16), cost (16), association (16), new (15), self (15), increase (15), high (15), quality (15), even (15), after (15), rct (15), absolute (15), percentage (15), adjustment (15), odds (15), adjusted (15), most (15), estimators (15), confounding (15), cohort (15), employment (14), disease (14), greater (14), very (14), where (14), close (14), low (14), total (14), well (14), lift (14), has (14), reduction (14), project (14), whether (14), children (14), trainees (14), related (13), work (13), point (13), variables (13), any (13), associated (13), consistent (13), find (13), estimator (13), true (13), method (13), figure (13), primary (13), cases (13), considered (13), ratio (13), unbiased (13), star (13), literature (13), schools (13), regular (13), interval (13), cited (13), mortality (13), harms (13), evaluating (12), effectiveness (12), designs (12), teachers (12), panel (12), likely (12), without (12), three (12), often (12), summary (12), conclusion (12), direction (12), second (12), pre (12), rcd (12), what (11), efficacy (11), welfare (11), intervals (11), important (11), sample (11), achievement (11), include (11), result (11), columns (11), compare (11), comparing (11), show (11), least (11), because (11), areas (11), made (11), statistical (11), significance (11), set (11), type (11), criterion (11), effective (11), assess (11), wellness (11), plans (11), highly (11), psychological (10), lipsey (10), wilson (10), causal (10), evaluations (10), 2002 (10), including (10), randomization (10), less (10), variable (10), education (10), while (10), heterogeneity (10), lower (10), during (10), deviation (10), row (10), surgery (10), selected (10), types (10), noted (10), per (10), might (10), area (10), overall (10), shows (10), empirical (10), patients (10), five (10), factors (10), employees (10), claims (10), six (10), nsw (10), breast (10), further (9), 24ya (9), cardiovascular (9), math (9), magnitude (9), suggest (9), ols (9), controls (9), researchers (9), should (9), matching (9), behaviors (9), consider (9), measure (9), variation (9), common (9), conclusions (9), available (9), risks (9), differed (9), treatments (9), eight (9), range (9), post (9), previous (9), concluded (9), led (9), differ (9), benefit (9), being (9), several (9), tested (9), workers (9), absenteeism (9), workplace (9), poverty (9), contradicted (9), load (8), 1998 (8), epidemiological (8), information (8), although (8), rule (8), following (8), 2013 (8), models (8), years (8), time (8), see (8), much (8), decision (8), 2018 (8), given (8), before (8), issue (8), identified (8), drug (8), procedure (8), gain (8), statistic (8), features (8), pairs (8), 100 (8), potential (8), year (8), participation (8), sites (8), job (8), energy (8), male (8), colleagues (8), model (7), 1983 (7), educational (7), 1987 (7), coaching (7), 1990 (7), 1999 (7), 27ya (7), social (7), 28ya (7), generally (7), conducted (7), calculated (7), through (7), versus (7), interest (7), still (7), people (7), demand (7), question (7), approaches (7), hypothesis (7), propensity (7), examine (7), contrast (7), searched (7), 1986 (7), particular (7), patient (7), subjects (7), positive (7), subsequent (7), number (7), criteria (7), provide (7), enough (7), benefits (7), 1988 (7), drawn (7), possible (7), shown (7), expenditures (7), cause (7), assessed (7), rates (7), against (7), care (7), approach (7), importance (7), baseline (7), jones (7), individual (7), divorce (7), residual (7), enrollee (7), oral (7), stronger (7), intake (7), links (6), error (6), shapiro (6), psychology (6), 1993 (6), family (6), therapies (6), 39ya (6), 2000 (6), 26ya (6), prevention (6), current (6), validity (6), indicating (6), errors (6), cross (6), though (6), comes (6), demographic (6), report (6), its (6), days (6), terms (6), run (6), since (6), key (6), just (6), received (6), better (6), tests (6), surgical (6), ratios (6), 40ya (6), prospective (6), chance (6), ioannidis (6), standardized (6), 000 (6), ability (6), yielded (6), present (6), authors (6), university (6), paper (6), does (6), papers (6), 38ya (6), pooled (6), quite (6), inferences (6), whereas (6), percentile (6), appendix (6), then (6), therefore (6), collected (6), age (6), status (6), addition (6), assessment (6), impacts (6), female (6), participate (6), women (6), rate (6), worksites (6), itt (6), eligible (6), econometric (6), degree (6), economics (6), females (6), members (6), specific (6), birth (6), multiple (6), death (6), initially (6), analysed (6), kunz (6), tools (6), exposure (6), smoking (6), suicide (6), innovation (6), backlinks (5), 43ya (5), journal (5), methodological (5), behavioral (5), 33ya (5), 36ya (5), maclehose (5), vitamin (5), role (5), contain (5), content (5), indicates (5), 2017 (5), precision (5), treated (5), few (5), separate (5), fixed (5), beyond (5), surveys (5), give (5), period (5), hours (5), sectional (5), factor (5), vary (5), individuals (5), long (5), relevant (5), advertising (5), due (5), million (5), produced (5), evaluated (5), checkout (5), leads (5), precise (5), way (5), equal (5), specification (5), obtained (5), adjustments (5), cochrane (5), separately (5), good (5), correlation (5), fold (5), performed (5), 2001 (5), zero (5), unpublished (5), ones (5), failure (5), single (5), wrong (5), his (5), krueger (5), background (5), them (5), measured (5), sets (5), negative (5), above (5), testing (5), major (5), procedures (5), food (5), implemented (5), 2016 (5), setting (5), correlational (5), nearly (5), existing (5), tables (5), restricted (5), annual (5), comprehensive (5), eligibility (5), plausible (5), fig (5), substantial (5), replicate (5), young (5), examined (5), crime (5), moving (5), families (5), housing (5), step (5), cps (5), ssa (5), econometricians (5), populations (5), afdc (5), supported (5), hypertension (5), syntheses (5), claim (5), bleed (5), coronary (5), acute (5), journals (5), lead (5), searches (5), britton (5), prognostic (5), calcium (5), hcts (5), causality (5), associations (5), contraceptives (5), day (4), bibliography (4), page (4), bulletin (4), comparative (4), implications (4), american (4), becker (4), scholastic (4), aptitude (4), synthesis (4), commonly (4), demonstration (4), match (4), policy (4), derived (4), potentially (4), despite (4), relatively (4), allows (4), others (4), fact (4), classroom (4), corresponding (4), predicted (4), behavior (4), questions (4), indicate (4), decisions (4), value (4), observations (4), suggests (4), generated (4), investigate (4), field (4), like (4), exact (4), reject (4), reports (4), answer (4), conversion (4), exposed (4), matched (4), produce (4), gave (4), digestive (4), discrepancies (4), databases (4), original (4), combined (4), 2004 (4), 22ya (4), alone (4), publication (4), natural (4), 25ya (4), yield (4), values (4), attrition (4), interaction (4), means (4), grouping (4), differential (4), take (4), main (4), variance (4), problem (4), secondary (4), kinds (4), almost (4), recent (4), compares (4), activity (4), either (4), provides (4), third (4), seven (4), thus (4), calculations (4), reduce (4), states (4), divided (4), kindergarten (4), determine (4), extent (4), basis (4), assessing (4), tennessee (4), problems (4), distribution (4), 1989 (4), 37ya (4), ways (4), lalonde (4), focused (4), carried (4), wilde (4), hollister (4), 2007 (4), nonparticipants (4), collection (4), rather (4), administrative (4), song (4), baicker (4), employee (4), frequently (4), visits (4), under (4), survey (4), screening (4), discrepancy (4), roi (4), 2010 (4), 16ya (4), change (4), patterns (4), appear (4), households (4), state (4), ates (4), double (4), readiness (4), mto (4), experimentally (4), ludwig (4), gender (4), proportion (4), developmental (4), variability (4), life (4), usually (4), whose (4), psid (4), enrollees (4), expected (4), 2014 (4), prevent (4), chronic (4), fractures (4), cereal (4), process (4), supplementation (4), myocardial (4), infarction (4), improved (4), adverse (4), increased (4), decrease (4), benson (4), concato (4), investigated (4), oxman (4), sacks (4), mix (4), situations (4), diet (4), recall (4), men (4), cigarettes (4), site (3), require (3), javascript (3), hedges (3), confirmation (3), hazelrigg (3), dehejia (3), wahba (3), friedlander (3), robins (3), 1995 (3), fraker (3), maynard (3), bloom (3), michalopoulos (3), hill (3), lei (3), york (3), manpower (3), corporation (3), glazerman (3), supplement (3), todo (3), relationship (3), happiness (3), degrees (3), around (3), generate (3), instrumental (3), calculate (3), instead (3), account (3), moved (3), 2008 (3), 18ya (3), independent (3), reading (3), policymakers (3), identifying (3), remain (3), capture (3), unique (3), development (3), controlling (3), techniques (3), unobservable (3), advertiser (3), overestimate (3), approximately (3), properly (3), listeners (3), sense (3), absence (3), said (3), ask (3), hence (3), metrics (3), truth (3), top (3), half (3), exposures (3), fail (3), fourth (3), seen (3), articles (3), equivalent (3), calculation (3), 2006 (3), 20ya (3), observed (3), tended (3), frequent (3), discrepant (3), varied (3), ideal (3), allowed (3), computation (3), pretest (3), conditions (3), specifically (3), cell (3), lowest (3), highest (3), sat (3), presurgical (3), put (3), mechanism (3), substantive (3), considerably (3), component (3), nonrandom (3), inclusion (3), aggregated (3), detail (3), statistics (3), extremely (3), past (3), choosing (3), suggested (3), smith (3), entire (3), reproduce (3), maybe (3), glass (3), 1980 (3), 46ya (3), colditz (3), miller (3), inconsistent (3), strong (3), considerable (3), theory (3), analysts (3), closely (3), except (3), classrooms (3), raw (3), answers (3), crucial (3), always (3), levels (3), providing (3), sign (3), ranging (3), usual (3), sometimes (3), entry (3), developed (3), increases (3), future (3), gains (3), uses (3), reducing (3), experience (3), slightly (3), somewhat (3), sufficient (3), indeed (3), misleading (3), invest (3), opposite (3), situation (3), cutoff (3), another (3), presents (3), sex (3), race (3), free (3), aide (3), principal (3), occurred (3), course (3), deviations (3), source (3), utilization (3), reliable (3), applied (3), provided (3), focus (3), sociology (3), full (3), probability (3), introduced (3), weight (3), markers (3), blood (3), pressure (3), domains (3), moreover (3), rules (3), lies (3), campus (3), gym (3), rich (3), particularly (3), biases (3), described (3), left (3), plots (3), expressed (3), tot (3), along (3), return (3), widely (3), savings (3), cutler (3), additional (3), 2011 (3), 15ya (3), productivity (3), event (3), simple (3), records (3), studied (3), yet (3), support (3), designed (3), 19ya (3), utilities (3), department (3), affects (3), affect (3), accounting (3), documented (3), occasionally (3), center (3), mothers (3), scale (3), 259 (3), crimes (3), violent (3), neighborhood (3), preprogram (3), offered (3), mobility (3), essentially (3), duncan (3), income (3), mental (3), will (3), unlike (3), tend (3), histories (3), unadjusted (3), ten (3), target (3), bar (3), corresponds (3), medicaid (3), reduced (3), similarly (3), 12ya (3), objective (3), routinely (3), making (3), 275 (3), sources (3), concordant (3), chemicals (3), bpa (3), search (3), 133 (3), items (3), scientific (3), epidemiologic (3), daily (3), contradictory (3), interpretation (3), symptomatic (3), intracranial (3), anticoagulant (3), blocker (3), defined (3), medline (3), striking (3), botulinum (3), toxin (3), dinoprostone (3), mechanical (3), ventilation (3), according (3), quasi (3), concurrent (3), heart (3), instances (3), hartz (3), allocated (3), methodology (3), historical (3), epidemiology (3), technique (3), allocation (3), blinding (3), lung (3), spurious (3), evident (3), saturated (3), fat (3), hiv (3), beta (3), carotene (3), bhat (3), mann (3), whitney (3), blind (3), public (2), service (2), annotation (2), hairsp (2), feedback (2), references (2), here (2), enabled (2), disabled (2), 388 (2), 395 (2), consulting (2), 1181 (2), 1209 (2), psychologist (2), cooper (2), borduin (2), 101 (2), 428 (2), 442 (2), integrative (2), 373 (2), 417 (2), appraisal (2), 31ya (2), adequacy (2), mandatory (2), levy (2), myers (2), washington (2), mathematica (2), replications (2), reeves (2), harvey (2), sheldon (2), white (2), 1997 (2), 29ya (2), causes (2), antioxidant (2), vitamins (2), imply (2), moderate (2), distinguish (2), shrunken (2), bacher (2), hicks (2), explanation (2), estimation (2), strongly (2), kane (2), 13ya (2), describing (2), order (2), skills (2), caution (2), appropriate (2), stakes (2), allow (2), month (2), closer (2), overestimates (2), width (2), improvement (2), suffer (2), caused (2), known (2), sensitivity (2), endogeneity (2), economic (2), engineering (2), resources (2), done (2), naturally (2), occurring (2), got (2), ads (2), measuring (2), know (2), 117 (2), 534 (2), fails (2), section (2), registration (2), view (2), little (2), summarizes (2), pixel (2), figures (2), users (2), gold (2), facebook (2), conditioning (2), linear (2), explain (2), 2019 (2), measurement (2), big (2), pubmed (2), april (2), binary (2), extracted (2), occur (2), 009 (2), 240 (2), 168 (2), projecting (2), passive (2), internal (2), note (2), largest (2), bit (2), simply (2), takes (2), 0001 (2), regressions (2), power (2), detect (2), finding (2), components (2), disparate (2), numbers (2), impossible (2), 119 (2), describe (2), adequately (2), haphazard (2), severe (2), unit (2), contained (2), juvenile (2), psychosocial (2), pupils (2), reflect (2), allowing (2), reason (2), baltimore (2), covariate (2), book (2), archived (2), medicine (2), combine (2), pattern (2), poor (2), remains (2), efficiency (2), summarized (2), 1976 (2), 50ya (2), met (2), quasiexperimental (2), 1979 (2), 47ya (2), improve (2), juveniles (2), accuracy (2), conduct (2), discussed (2), complete (2), fifth (2), again (2), looking (2), perform (2), rural (2), city (2), easy (2), entered (2), hand (2), conclude (2), aspects (2), rough (2), illustrate (2), carefully (2), taken (2), interesting (2), justify (2), costs (2), entries (2), look (2), enter (2), yes (2), gives (2), thought (2), 135 (2), seems (2), make (2), suppose (2), represents (2), closeness (2), lunch (2), broadly (2), throughout (2), cities (2), 1985 (2), schedule (2), etc (2), adjusting (2), scored (2), grade (2), below (2), various (2), motivated (2), involving (2), estimating (2), proper (2), substantially (2), attempted (2), called (2), neither (2), nor (2), etable (2), elected (2), obtain (2), endogenous (2), additionally (2), screenings (2), 4037 (2), 4106 (2), modules (2), completed (2), months (2), exercise (2), prespecified (2), sleep (2), choices (2), cholesterol (2), index (2), pharmaceutical (2), via (2), 140 (2), 160 (2), june (2), lasso (2), monthly (2), outside (2), 011 (2), presented (2), healthier (2), line (2), play (2), specified (2), participating (2), hra (2), variants (2), equation (2), includes (2), specifications (2), variety (2), explanations (2), typical (2), illustrates (2), treat (2), changes (2), repeat (2), 115 (2), investment (2), replicated (2), drugs (2), hospital (2), popular (2), community (2), running (2), null (2), meaningfully (2), invited (2), paid (2), amounts (2), remaining (2), online (2), limited (2), participated (2), direct (2), net (2), electricity (2), weather (2), heating (2), form (2), utility (2), household (2), opower (2), respectively (2), biased (2), consistently (2), must (2), arguments (2), directly (2), easily (2), magnuson (2), mcgroder (2), increasing (2), final (2), national (2), united (2), receipt (2), basic (2), generating (2), induced (2), schooling (2), arrests (2), opportunity (2), criminal (2), dropout (2), census (2), neighborhoods (2), adjust (2), move (2), unmeasured (2), moves (2), hirschfield (2), urban (2), voucher (2), sampling (2), observable (2), influence (2), series (2), extensive (2), cherlin (2), parental (2), contrary (2), product (2), parents (2), parent (2), earlier (2), unobservables (2), seriously (2), pretraining (2), administration (2), applicants (2), payments (2), 1975 (2), 51ya (2), federally (2), sponsored (2), identical (2), mdrc (2), depending (2), worked (2), together (2), wage (2), stay (2), occupations (2), participant (2), 137 (2), arises (2), 2022 (2), investigating (2), deaths (2), nine (2), favorable (2), length (2), head (2), circumference (2), gestational (2), kidney (2), hip (2), owing (2), older (2), bisphenol (2), diabetes (2), 2009 (2), 17ya (2), assayed (2), urine (2), 1000 (2), attack (2), endpoints (2), false (2), eating (2), biological (2), breakfast (2), standards (2), principles (2), serious (2), vaccines (2), uncontrolled (2), vascular (2), visceral (2), injury (2), laparoscopic (2), open (2), repair (2), inguinal (2), hernia (2), cmaj (2), addressing (2), feasible (2), harm (2), already (2), 4000 (2), profile (2), influential (2), general (2), 129 (2), stunted (2), stimulation (2), additive (2), achieved (2), follow (2), longer (2), acetylcysteine (2), glyceryl (2), trinitrate (2), fissure (2), curing (2), fewer (2), infants (2), intermittent (2), diazepam (2), administered (2), seizures (2), virus (2), infection (2), duration (2), cladribine (2), quintile (2), incidence (2), subsequently (2), 008 (2), surrogate (2), excluded (2), cognitive (2), computed (2), library (2), leading (2), bcg (2), stroke (2), horwitz (2), showing (2), black (2), publications (2), category (2), comparable (2), immunotherapy (2), anticoagulants (2), reviewed (2), quantitative (2), protect (2), healthcare (2), adjuvant (2), investigation (2), containing (2), feature (2), meeting (2), looks (2), investigations (2), resampling (2), sixty (2), covered (2), dose (2), response (2), relation (2), link (2), formaldehyde (2), anatomists (2), pathologists (2), intermediate (2), protects (2), obvious (2), dietary (2), slope (2), amount (2), frequency (2), questionnaires (2), hour (2), transmission (2), mechanisms (2), equally (2), cannot (2), developing (2), established (2), expanded (2), relate (2), attributed (2), externally (2), record (2), sequential (2), likelihood (2), adblock, announcement, quote, send, anonymous, superseded, services, patterson, kristal, neuhouser, potter, 786, 802, lonn, yusuf, update, 957, 965, cardiol, predictive, exception, unshrunken, roughly, 084, substantively, compliance, narrowing, blocks, shrinkage, overlapping, staiger, come, actual, closest, 995, cells, dependent, attitude, listed, block, rosters, growing, practitioners, skilled, improving, leveraging, weak, correlations, bring, revealing, proceed, professional, personnel, accountability, blazar, validating, attitudes, added, best, shot, learned, active, invariant, listener, omitted, correlate, listening, timescale, matters, platform, realized, get, happen, listen, valuable, difficult, convince, makers, consumes, implement, untouched, default, excluding, huang, consumer, audio, pandora, internet, radio, precisely, 535, 661, last, summarize, want, divide, followed, tells, tracked, conversions, equals, down, fare, equality, landing, comprising, 435, user, billion, impressions, unexposed, synthetic, markets, thousands, why, industry, accurately, gordon, embase, identify, fifty, 276, 180, 1966, 60ya, shikata, 001, occasional, timing, logarithm, events, arm, synthesized, extraction, diverse, simulate, predictor, equate, reasonable, projections, assume, tailored, projection, diminished, dramatically, smallest, confounds, involve, place, 029, 000075, borderline, interactions, conceptually, finally, twice, unusually, creating, variances, 733, describes, retrieving, proved, resulted, elimination, 103, dichotomous, eliminated, reasons, implausible, posttest, drew, tobler, devine, slavin, deliberately, manuscripts, regard, practical, contacts, willing, documents, 1992, 34ya, johns, hopkins, press, followup, doesn, anything, ordered, copy, mentions, notes, coded, date, archives, anywhere, colorado, 1977, 49ya, psychotherapy, mosteller, 343, 352, analyst, metric, aggregation, trivially, underestimate, 1193, believe, ambiguity, near, mostly, silent, consistency, empirically, certain, assumptions, correlated, executed, overridden, justifies, nonequivalent, units, cook, campbell, abuse, equated, reasonably, accurate, heinsman, shadish, 1996, 30ya, experimentation, approximate, district, indication, constructing, signs, differing, pair, subset, strict, argued, couple, presenting, sketch, elements, focuses, develops, early, elementary, discounting, appropriately, purposes, necessary, took, increasingly, critical, apply, create, contains, began, considering, conclusively, definitive, distance, observer, action, decided, stringent, indicated, weaken, stringency, vice, versa, perhaps, word, chosen, distributed, originally, nonmetropolitan, towns, manchester, maryville, inner, suburban, metropolitan, knoxville, nashville, memphis, chattanooga, beginning, curriculum, theoretically, reassigned, switched, grades, unaffected, rank, stanford, career, ladder, system, contemporaneously, exogenous, interested, construct, considerations, undertake, compelling, fashion, solely, drawing, kindergartens, derive, bootstrapping, attempting, guides, greatest, emphasis, maker, trying, wider, application, framework, complex, tried, domain, nonwhite, salaried, sales, having, incorrectly, underscoring, thereby, naive, ordinary, squares, 974, 272, engaging, actively, managing, body, mass, tenure, 937, programming, comprised, nutrition, physical, stress, registered, dietitians, clustered, january, gathered, continuously, biometrics, july, august, 2015, 11ya, indistinguishable, every, marathon, 10k, 024, inside, 757, 656, assumption, contemporaneous, spirit, demonstrate, suggesting, settings, visualization, times, generates, indicator, vector, instrument, completing, variant, strata, reductions, presence, advantageous, possibility, something, competing, performing, demonstrating, none, fall, intent, ease, bottom, establish, rely, pelletier, chapman, skew, abraham, end, publicly, independently, pal, affiliated, researcher, 2012, 14ya, quantiles, subcategory, office, facilities, ruled, demonstrates, employing, 459, 834, 300, off, successfully, earned, rewards, 350, communicated, start, permitted, combines, insurance, visit, namely, novel, cover, intended, exists, employer, financial
Text of the page (random words):
contrary most empirical studies of the effects of divorce on children have assumed that divorce is randomly assigned to children they do this by failing to control for the fact that divorce is the product of the parents temperaments resources and other stressors that face parents most of which will influence children s outcomes in their own right as a result studies comparing developmental outcomes of children with and without past parental divorces after controlling for a handful of family background characteristics are likely to confound the effects of divorce with the effects of unmeasured parent and child variables indeed studies that control for children s behavior problems prior to a possible divorce find much smaller apparent effects of the divorce itself cherlin et al 1998 28ya cherlin a chase lansdale l p mcrae c 1998 28ya effects of parental divorce on mental health throughout the life course american sociological review 63 239 249 these experiments can provide researchers with some sense for the bias that results from nonexperimental estimates as well as providing direct evidence for the causal effects of some developmental influence of interest for example wilde hollister 2007 19ya compare nonexperimental and experimental results for the widely cited tennessee student teacher achievement ratio star class size experiment the star experiment provides an unbiased estimate of the impact of class size on student achievement by comparing the average achievement levels of students assigned to small experimental and regular control classrooms however wilde and hollister also estimated a series of more conventional nonexperimental regressions that related naturally occurring class size variation within the set of regular classrooms to student achievement controlling for an extensive set of student demographic characteristics and socioeconomic status wilde hollister 2007 19ya how close is close enough testing nonexperimental estimates of impact against experimental estimates of impact with education test scores as outcomes table 1 compares the experimental and nonexperimental estimates of class size impacts by school the table shows substantial variability across schools in the effects of smaller classes on student standardized test scores in some cases eg schools b d and i the two sets of estimates are quite close but in some eg schools c e g and h they are quite different a comparison of the nonexperimental and experimental results as a whole reveals that the average bias ie the absolute difference between the experimental and nonexperimental impact estimates is on the order of 10 percentile points about the same as the average experimental estimate for the effects of smaller classes table 1 comparison of experimental and nonexperimental estimates for effects of class size on student test scores estimate statistically significant at the 5 cutoff school nonexperimental regression experimental estimate a 9 6 5 2 b 15 3 13 0 c 1 9 24 1 d 35 2 33 1 e 20 4 10 5 f 0 2 1 3 g 8 6 10 6 h 5 6 9 6 i 16 5 14 7 j 24 3 16 2 k 27 8 19 3 a second example of the bias that may result with nonexperimental estimates comes from the u s department of housing and urban development s moving to opportunity mto housing voucher experiment which randomly assigned housing project residents in high poverty neighborhoods of five of the nation s largest cities to either a group that was offered a housing voucher to relocate to a lower poverty area or to a control group that received no mobility assistance under the program ludwig duncan hirschfield 2001 25ya because of well implemented random assignment each of the groups on average should be equivalent subject to sampling variability with respect to all observable and unobservable preprogram characteristics ludwig j 1999 27ya experimental and non experimental estimates of neighborhood effects on low income families unpublished document georgetown university ludwig j duncan g hirschfield p 2001 25ya urban poverty and juvenile crime evidence from a randomized housing mobility experiment quarterly journal of economics 116 655 680 table 2 presents the results of using the randomized design of mto to generate unbiased estimates of the effects of moving from high to low poverty census tracts on teen crime the experimental estimates are the difference between average outcomes of all families offered vouchers and those assigned to the control group divided by the difference across the two groups in the proportion of families who moved to a low poverty area note the implication that these kinds of experimental data can be used to produce unbiased estimates of the effects of neighborhood characteristics on developmental outcomes even if the takeup rate is less than 100 in the treatment group and greater than 0 among the control group 4 the nonexperimental estimates simply compare families who moved to low poverty neighborhoods with those who did not ignoring information about each family s random assignment and relying on the set of prerandom assignment measures of mto family characteristics to adjust for differences between families who chose to move and those who do not 5 as seen in table 2 even after statistically adjusting for a rich set of background characteristics the nonexperimental measure the unmeasured approach leads to starkly different inferences about the effects of residential mobility compared with the unbiased experimental estimates for example the experimental estimates suggest that moving from a high to a low poverty census tract significantly reduces the number of violent crimes in contrast the nonexperimental estimates find that such moves have essentially no effect on violent arrests in the case of other crimes the nonexperimental estimates suggest that such moves reduce crime but the experimentally based estimates do not table 2 estimated impacts of moving from a high to a low poverty neighborhood on arrests per 100 juveniles from ludwig 1999 27ya based on data from the baltimore moving to opportunity experiment regression models also control for baseline measurement of gender age at random assignment and preprogram criminal involvement family s preprogram victimization mother s schooling welfare receipt and marital status estimated effect of dropout program on dropout rates statistically significant at the 5 cutoff level measure experimental se non experimental se sample size violent crime 47 4 24 3 4 9 12 5 259 property crime 29 7 28 9 10 8 14 1 259 other crimes 0 6 37 4 36 9 14 3 259 a final example comes from the national evaluation of welfare to work strategies randomized experiment designed to evaluate welfare to work programs in seven sites across the united states one of the treatment streams encouraged welfare recipient mothers to participate in education activities in addition to measuring outcomes such as clients welfare receipt employment and earnings the evaluation study also tested young children s school readiness using the bracken basic concepts scale school readiness subscale using a method for generating experimental estimates similar to that used in the mto analyses magnuson and mcgroder 2002 24ya examined the effects of the experimentally induced increases in maternal schooling on children s school readiness again the results suggest that nonexperimental estimates did not closely reproduce experimentally based estimates magnuson k a mcgroder s 2002 24ya the effect of increasing welfare mothers education on their young children s academic problems and school readiness joint center for poverty research working paper no 280 northwestern university a much larger literature within economics statistics and program evaluation has focused on the ability of nonexperimental regression adjustment methods to replicate experimental estimates for the effects of job training or welfare to work programs although the contexts represented by these programs may be less interesting to developmentalists the results of this literature nevertheless bear directly on the question considered in this article can regression methods with often quite detailed background covariates reproduce experimental impact estimates for such programs as one recent review concluded occasionally but not in a way that can be easily predicted glazerman et al 2002 pg46 see also bloom michalopoulos hill lei 2002 24ya glazerman s levy d myers d 2002 24ya nonexperimental replications of social experiments a systematic review washington dc mathematica policy research bloom h s michalopoulos c hill c lei y 2002 24ya can non experimental comparison group methods match the findings from a random assignment evaluation of mandatory welfare to work programs new york manpower demonstration research corporation allcott 2011 15ya social norms and energy conservation nearly all energy efficiency programs are still evaluated using non experimental estimators or engineering accounting approaches how important is the experimental control group to consistently estimated ates this issue is crucial for several of opower s initial programs that were implemented without a control group but must estimate impacts to report to state regulators while lalonde 1986 40ya documented that non experimental estimators performed poorly in evaluating job training programs and similar arguments have been made in many other domains weather adjusted non experimental estimators could in theory perform well in modeling energy demand the importance of randomized controlled trials has not yet been clearly documented to analysts and policymakers in this context without an experimental control group there are two econometric approaches that could be used the first is to use a difference estimator comparing electricity use in the treated population before and after treatment in implementing this i control for weather differences non parametrically using bins with width one average degree day this slightly outperforms the use of fourth degree polynomials in heating and cooling degree days this estimator is unbiased if and only if there are no other factors associated with energy demand that vary between the pre treatment and post treatment period a second non experimental approach is to use a difference in differences estimator with nearby households as a control group for each experiment i form a control group using the average monthly energy use of households in other utilities in the same state using data that regulated utilities report to the u s department of energy on form eia 826 the estimator includes utility by month fixed effects to capture different seasonal patterns for example there may be local variation in how many households use electric heat instead of natural gas or oil which then affects winter electricity demand this estimator is unbiased if and only if there are no unobserved factors that differentially affect average household energy demand in the opower partner utility versus the other utilities in the same state fig 6 presents the experimental ates for each experiment along with point estimates for the two types of non experimental estimators there is substantial variance in the non experimental estimators the average absolute errors for the difference and difference in differences estimators respectively are 2 1 and 3 0 across the 14 experiments the estimators are also biased on average in particular the mean of the ates from the difference in differences estimator is 3 75 which is nearly double the mean of the experimental ates what s particularly insidious about the non experimental estimates is that they would appear quite plausible if not compared to the experimental benchmark nearly all are within the confidence intervals of the small sample pilots by schultz et al 2007 19ya and nolan et al 2008 18ya that were discussed above evaluations of similar types of energy use information feedback programs have reported impacts of zero to 10 darby 2006 20ya what do workplace wellness programs do evidence from the illinois workplace wellness study jones et al 2018 workplace wellness programs cover over 50 million workers and are intended to reduce medical spending increase productivity and improve well being yet limited evidence exists to support these claims we designed and implemented a comprehensive workplace wellness program for a large employer with over 12 000 employees and randomly assigned program eligibility and financial incentives at the individual level over 56 of eligible treatment group employees participated in the program we find strong patterns of selection during the year prior to the intervention program participants had lower medical expenditures and healthier behaviors than non participants however we do not find significant causal effects of treatment on total medical expenditures health behaviors employee productivity or self reported health status in the first year our 95 confidence intervals rule out 83 of previous estimates on medical spending and absenteeism our selection results suggest these programs may act as a screening mechanism even in the absence of any direct savings differential recruitment or retention of lower cost participants could result in net savings for employers we invited 12 459 benefits eligible university employees to participate in our study 3 study participants n 4 834 assigned to the treatment group n 3 300 were invited to take paid time off to participate in our workplace wellness program those who successfully completed the entire program earned rewards ranging from 50 to 350 with the amounts randomly assigned and communicated at the start of the program the remaining subjects n 1 534 were assigned to a control group which was not permitted to participate our analysis combines individual level data from online surveys university employment records health insurance claims campus gym visit records and administrative records from a popular community running event we can therefore examine outcomes commonly studied by the prior literature namely medical spending and employee absenteeism as well as a large number of novel outcomes third we do not find significant effects of our intervention on 37 out of the 39 outcomes we examine in the first year following random assignment these 37 outcomes include all our measures of medical spending productivity health behaviors and self reported health we investigate the effect on medical expenditures in detail but fail to find significant effects on different quantiles of the spending distribution or on any major subcategory of medical expenditures pharmaceutical drugs office or hospital we also do not find any effect of our intervention on the number of visits to campus gym facilities or on the probability of participating in a popular annual community running event two health behaviors that are relatively simple for a motivated employee to change over the course of one year these null estimates are meaningfully precise particularly for two...
|
