Meta tags:
Headings (most frequently used words):
2014, blog, july, language, november, october, 27, 30, june, may, april, my, list, the, emnlp, reviews, is, not, point, models, past, tense, natural, processing, 15, 01, 10, 03, september, 31, 05, 02, 16, 26, 14, about, me, labels, archive, myth, of, strong, baseline, paper, with, mini, hyperparameter, search, bayesian, optimization, and, related, topics, machine, learning, new, algorithms, amr, semantics, but, close, maybe, reading, group, notes, counter, on, predict, hello, world, acl, picks, divergences, passed, through, bayes, rule, role, perplexity, versus, error, rate, for, modeling, an, easy, way, to, write, less, hurtful, don, say, you, waaaah, six, months, late,
Text of the page (most frequently used words):
the (550), and (279), that (240), this (174), for (136), you (128), but (90), not (72), like (71), with (70), have (67), can (63), are (61), they (60), from (57), was (54), paper (52), about (52), there (50), which (46), some (42), more (42), #learning (41), one (41), ago (40), what (40), think (40), 2014 (37), could (37), all (37), these (37), because (37), #language (36), how (35), model (34), get (34), really (34), just (33), than (32), something (32), then (32), problem (31), would (31), algorithms (30), years (30), good (29), words (29), word (29), any (28), has (28), when (27), time (27), very (27), way (25), models (25), also (25), here (25), into (25), better (25), don (24), know (24), things (24), data (23), even (23), out (23), machine (22), new (22), does (22), much (22), actually (22), now (21), see (21), will (20), first (20), other (20), point (19), baseline (19), comments (19), instance (19), should (19), say (19), basically (19), work (19), error (18), over (18), most (18), them (18), their (18), might (18), right (18), results (18), who (18), true (18), hal (17), thing (17), given (17), example (17), been (17), english (17), where (16), many (16), context (16), going (16), amr (15), strong (15), papers (15), let (15), had (15), though (15), people (15), sentences (15), predict (15), only (15), neural (15), may (14), reviews (14), blog (14), posted (14), your (14), answer (14), sentence (14), want (14), well (14), same (14), two (14), least (14), course (14), pretty (14), use (14), those (14), always (14), doesn (14), important (14), question (14), june (13), july (13), past (13), tense (13), world (13), search (13), using (13), okay (13), doing (13), perhaps (13), distribution (13), languages (13), part (13), conditional (13), cos (13), perplexity (12), rate (12), between (12), bit (12), probability (12), both (12), lots (12), best (12), count (12), april (11), september (11), reading (11), semantics (11), optimization (11), parsing (11), text (11), hard (11), our (11), why (11), mean (11), etc (11), try (11), similarity (11), maltparser (11), claim (11), hyperparameters (11), versus (10), theory (10), own (10), last (10), post (10), didn (10), possible (10), little (10), rather (10), probably (10), issue (10), lot (10), ways (10), idea (10), noun (10), semantic (10), learn (10), never (10), against (10), joint (10), dependency (10), structure (10), trying (10), december (9), october (9), november (9), emnlp (9), rule (9), algorithm (9), task (9), help (9), approach (9), since (9), particular (9), said (9), back (9), effect (9), were (9), based (9), learned (9), high (9), number (9), set (9), believe (9), nlp (9), find (9), case (9), interesting (9), talk (9), network (9), queen (9), arg0 (9), natural (9), february (8), modeling (8), evaluation (8), anything (8), make (8), often (8), reason (8), examples (8), cases (8), authors (8), full (8), order (8), liked (8), according (8), another (8), take (8), version (8), made (8), wrong (8), explicit (8), almost (8), look (8), far (8), different (8), regina (8), representation (8), poss (8), points (8), march (7), august (7), role (7), bayesian (7), information (7), grad (7), science (7), large (7), prediction (7), community (7), few (7), school (7), everyone (7), such (7), useful (7), nice (7), each (7), quite (7), compute (7), whether (7), reasonable (7), correct (7), errors (7), fact (7), argue (7), common (7), brain (7), win (7), yes (7), experiment (7), still (7), technique (7), cool (7), finding (7), yoav (7), embeddings (7), vectors (7), too (7), representations (7), verb (7), tree (7), great (7), features (7), big (7), bleu (7), january (6), less (6), hello (6), close (6), maybe (6), list (6), ideas (6), random (6), feel (6), predicting (6), via (6), long (6), called (6), times (6), thought (6), students (6), above (6), need (6), related (6), previous (6), edit (6), overall (6), under (6), next (6), bad (6), measure (6), appear (6), predicted (6), while (6), sense (6), used (6), note (6), setting (6), type (6), improvement (6), its (6), sort (6), gender (6), understand (6), linguistic (6), clearly (6), give (6), bar (6), stuff (6), someone (6), male (6), heard (6), his (6), gave (6), compare (6), variational (6), run (6), left (6), worse (6), read (6), event (6), space (6), comparison (6), looking (6), class (6), rest (6), possession (6), flow (6), accuracy (6), show (6), months (5), easy (5), write (5), divergences (5), through (5), bayes (5), acl (5), hyperparameter (5), analysis (5), structured (5), research (5), statistical (5), inference (5), days (5), available (5), speech (5), problems (5), coreference (5), please (5), comment (5), log (5), today (5), start (5), person (5), having (5), sure (5), nouns (5), sound (5), small (5), obvious (5), field (5), down (5), standard (5), history (5), after (5), real (5), aren (5), reasons (5), form (5), put (5), knows (5), precision (5), without (5), interpretation (5), did (5), present (5), refer (5), thanks (5), already (5), makes (5), divergence (5), foo (5), took (5), during (5), land (5), fairly (5), before (5), haven (5), enough (5), four (5), marginal (5), function (5), distance (5), following (5), recovered (5), projection (5), title (5), together (5), result (5), main (5), optimal (5), networks (5), usually (5), solve (5), parse (5), goldberg (5), love (5), arg (5), aka (5), woman (5), king (5), man (5), gets (5), local (5), hear (5), ring (5), imagine (5), imo (5), team (5), smita (5), especially (5), input (5), major (5), teach (5), concrete (5), everything (5), smac (5), assumption (5), system (5), group (4), notes (4), mathematics (4), math (4), future (4), come (4), deep (4), off (4), current (4), year (4), weeks (4), topic (4), translation (4), discourse (4), remember (4), anyone (4), human (4), concept (4), eat (4), perceptron (4), linear (4), review (4), place (4), seems (4), tell (4), else (4), written (4), solution (4), comes (4), usual (4), care (4), likes (4), her (4), produce (4), distributions (4), worked (4), tune (4), cannot (4), ran (4), train (4), means (4), pick (4), top (4), background (4), french (4), german (4), happened (4), strange (4), vary (4), add (4), code (4), hallucinate (4), workshop (4), isn (4), instructors (4), difference (4), fit (4), tech (4), looks (4), meaning (4), matter (4), beyond (4), prove (4), simple (4), generate (4), exactly (4), shows (4), bottom (4), row (4), perfect (4), ask (4), being (4), var (4), max_ (4), estimates (4), got (4), favorite (4), wasn (4), mapping (4), works (4), images (4), baroni (4), study (4), architecture (4), basic (4), boston (4), prefix (4), hallo (4), due (4), google (4), contexts (4), parameters (4), reasoning (4), prince (4), matrix (4), arg1 (4), petals (4), annotated (4), glass (4), globe (4), side (4), planet (4), alienable (4), mother (4), kind (4), amount (4), mod (4), kenji (4), student (4), she (4), abstract (4), whose (4), move (4), tuning (4), knowledge (4), wait (4), map (4), response (4), done (4), pos (4), speed (4), processing (4), rhetorical (4), gain (4), 2007 (3), picks (3), counter (3), mini (3), weblog (3), engineering (3), bob (3), author (3), award (3), parallel (3), making (3), scientific (3), michael (3), position (3), systems (3), computer (3), computation (3), test (3), recent (3), experience (3), training (3), scale (3), feedback (3), programming (3), proposed (3), hours (3), summarization (3), sentiment (3), questions (3), online (3), news (3), domain (3), clustering (3), labels (3), posts (3), home (3), anymore (3), free (3), happy (3), tacl (3), document (3), john (3), verbs (3), liu (3), helpful (3), decisions (3), mind (3), theirs (3), hate (3), coffee (3), process (3), probabilistic (3), clear (3), saying (3), around (3), classes (3), recall (3), seemed (3), easier (3), semi (3), crappy (3), guessing (3), tried (3), vocabulary (3), expected (3), guess (3), frequent (3), below (3), stop (3), average (3), issues (3), general (3), non (3), false (3), essentially (3), consistent (3), speakers (3), running (3), kevin (3), link (3), likely (3), across (3), morphologically (3), rich (3), linguistically (3), easily (3), phenomenon (3), discussion (3), awesome (3), helped (3), translating (3), japanese (3), places (3), identify (3), successful (3), him (3), focus (3), relation (3), believed (3), story (3), spent (3), hadn (3), myself (3), won (3), knew (3), support (3), binary (3), chose (3), along (3), estimate (3), noisy (3), max (3), direction (3), advantage (3), known (3), analogy (3), recovery (3), sum_b (3), quality (3), wondering (3), half (3), zhang (3), counts (3), visual (3), denotations (3), richard (3), christopher (3), manning (3), worth (3), barzilay (3), solving (3), marco (3), georgiana (3), dinu (3), kruszewski (3), found (3), systematic (3), counting (3), silly (3), blah (3), mundo (3), dunia (3), msr (3), semeval (3), tied (3), 3cosadd (3), cosine (3), distributional (3), conclusion (3), vector (3), faster (3), 100 (3), tasks (3), car (3), property (3), argument (3), correctly (3), annotation (3), compounds (3), simply (3), air (3), authority (3), alvin (3), inalienable (3), taken (3), takes (3), step (3), interlingua (3), house (3), julio (3), erin (3), wrote (3), excited (3), itself (3), surprised (3), whole (3), fast (3), akiko (3), flows (3), edge (3), greedy (3), understanding (3), replace (3), various (3), whatever (3), changed (3), topics (3), black (3), box (3), spearmint (3), seen (3), grid (3), akin (3), solved (3), kept (3), feature (3), spelling (3), seem (3), cut (3), optima (3), restarts (3), complex (3), fusion (3), scores (3), parser (3), style (3), logic (3), entailment (3), performance (3), predictive (3), resolution (3), weaker (3), comparing (3), baselines (3), bayesum (3), 2005 (2), 2006 (2), waaaah (2), six (2), late (2), hurtful (2), passed (2), myth (2), 2016 (2), retrieval (2), astrostat (2), slog (2), info (2), journal (2), articles (2), anthology (2), levels (2), quantum (2), corpora (2), wikipedia (2), thoughts (2), page (2), weighted (2), travel (2), biased (2), chatgpt (2), talking (2), earning (2), turns (2), book (2), month (2), scala (2), software (2), changes (2), respect (2), computational (2), statistics (2), linguistics (2), state (2), methods (2), classification (2), complete (2), reads (2), effort (2), short (2), huang (2), identifying (2), bilingual (2), contrast (2), positive (2), chen (2), yang (2), compression (2), mark (2), johnson (2), strings (2), grounded (2), unsupervised (2), modification (2), super (2), calls (2), honest (2), feelings (2), hurt (2), pernicious (2), manner (2), primary (2), accept (2), friendly (2), keep (2), later (2), friend (2), provide (2), recently (2), mention (2), involved (2), integrate (2), similar (2), loved (2), others (2), become (2), attack (2), contribution (2), method (2), experiments (2), specifically (2), opposed (2), sugar (2), socks (2), notion (2), low (2), combined (2), rules (2), realized (2), weights (2), plenty (2), predictions (2), alternatives (2), evaluate (2), life (2), penn (2), trained (2), kneser (2), ney (2), srilm (2), types (2), ten (2), identical (2), certainly (2), metrics (2), 1274 (2), absolutely (2), approaches (2), play (2), game (2), output (2), must (2), russian (2), feminine (2), happen (2), express (2), getting (2), newswire (2), convention (2), usage (2), native (2), falls (2), flight (2), knight (2), again (2), introduced (2), went (2), boss (2), focusing (2), energy (2), variability (2), speaker (2), explicitly (2), came (2), maja (2), tool (2), light (2), per (2), roughly (2), single (2), rst (2), shift (2), correlation (2), instructor (2), crazy (2), pose (2), several (2), rare (2), empathize (2), poul (2), erik (2), badminton (2), jiong (2), china (2), lgbt (2), gotten (2), interpret (2), token (2), label (2), match (2), self (2), choose (2), ever (2), lines (2), lazy (2), directly (2), reconstructed (2), three (2), absolute (2), chosen (2), slightly (2), definition (2), similarly (2), 000 (2), relevant (2), plots (2), representative (2), column (2), dimensions (2), until (2), switch (2), totally (2), bug (2), check (2), sum_a (2), sum_ (2), talked (2), conference (2), suspect (2), missed (2), sparsity (2), apply (2), clever (2), categorization (2), told (2), smoothing (2), solves (2), image (2), descriptions (2), socher (2), nips (2), fixed (2), trivial (2), uninteresting (2), algebra (2), twice (2), nickles (2), formulae (2), automatically (2), triumphalist (2), overtones (2), lack (2), wish (2), instead (2), reduce (2), segment (2), hit (2), detection (2), francesco (2), giorgio (2), dynamic (2), oracles (2), efficiently (2), computing (2), possibly (2), twitter (2), hola (2), helo (2), здраво (2), remarkably (2), handout (2), datasets (2), alternative (2), open (2), drat (2), eps (2), motivation (2), writing (2), varied (2), bag (2), appears (2), pmi (2), 200 (2), 500 (2), either (2), assume (2), dim (2), thrown (2), frequency (2), importance (2), dimensional (2), cluster (2), apples (2), strongly (2), brother (2), sister (2), grandson (2), summary (2), wins (2), worst (2), window (2), regularities (2), sparse (2), omer (2), levy (2), capture (2), cars (2), surprise (2), transform (2), clauses (2), subordinate (2), unfortunately (2), theme (2), missing (2), expect (2), disambiguated (2), string (2), consist (2), turned (2), genitive (2), fine (2), possessive (2), marker (2), prior (2), chinese (2), aunt (2), wals (2), mesa (2), grande (2), ətalʸ (2), famous (2), choosing (2), details (2), final (2), stole (2), triangle (2), 1sg (2), warning (2), amrs (2), closest (2), undergrad (2), undergraduate (2), degrees (2), statements (2), level (2), able (2), sources (2), fields (2), job (2), figure (2), further (2), inputs (2), uses (2), unimportant (2), art (2), trees (2), taking (2), create (2), dijkstra (2), shortest (2), path (2), historical (2), traffic (2), maps (2), costs (2), road (2), minutes (2), framework (2), derivative (2), strategy (2), active (2), bias (2), steps (2), preferably (2), equivalent (2), strategies (2), deal (2), folks (2), end (2), passes (2), settings (2), early (2), stopping (2), accurate (2), willing (2), presumably (2), pass (2), larochelle (2), default (2), broader (2), hps (2), once (2), svm (2), automatic (2), answering (2), beat (2), simultaneous (2), jordan (2), boyd (2), graber (2), daumé (2), iii (2), improve (2), spend (2), combination (2), labeling (2), oscar (2), correction (2), jointly (2), hurts (2), helps (2), notoriously (2), embedding (2), implementation (2), caching (2), neat (2), reminds (2), anyway (2), logical (2), cats (2), combine (2), felix (2), relatively (2), metaphor (2), meg (2), laboratory (2), connecting (2), hypothesis (2), control (2), varies (2), length (2), essays (2), respectively (2), hope (2), moses (2), goes (2), devlin (2), substantiate (2), reviewer (2), asked (2), source (2), skip (2), 2008, 2009, 2010, 2011, 2012, 2013, predi, 2015, 2017, 2018, archive, mainly, apperceptual, http, groundtruth, ganesh, swami, undirected, lowerbounds, upperbounds, forthcoming, mstatbiostat, inductio, machina, corrections, urls, deserve, logicomp, polynomial, simulation, chemical, dynamics, presburger, webdiarios, motocicleta, dual, submissions, busted, vision, statmt, academic, contacts, researchers, andy, key, shaping, publishing, nielsen, algorithmic, economics, postdoc, microsoft, nyc, oddhead, trends, iclr, tombone, metrical, star, tcs, theoretical, derivations, computations, slice, pizza, 124, stats, coin, designer, cloud, wrangling, conversation, scientist, hickok, brains, books, lingpipe, misinformation, geeking, greg, focs, awards, referee, reports, retracted, reformers, peek, behind, curtain, causal, social, parenting, recommendations, xor, hammer, agentic, coding, ascension, pro, gowers, unit, distances, geomblog, streamlined, optical, modern, architectures, direct, alignment, nuit, blanche, professor, emeritus, wadler, bayesmultimode, mode, amd64, microarchitecture, daniel, lemire, day, administration, federal, grants, complexity, teaching, survey, reviewing, poll, mcmc, loss, functions, journals, hiring, graphical, finite, adaptation, conferences, chunking, advising, acs, view, profile, subscribe, atom, older, newer, opinions, spring, sumit, basu, charles, jacobs, lucy, vanderwende, powergrading, amplify, grading, ioannis, konstas, mirella, lapata, inducing, plans, generation, jun, seok, kang, polina, kuznetsova, luca, yejin, choi, restaurant, inspections, heng, liang, haitao, violation, fixing, forced, decoding, scalable, karl, pichotta, denero, phrasal, ellen, riloff, ashequl, qadir, prafulla, surve, lalindra, silva, nathan, gilbert, ruihong, sarcasm, negative, situation, fei, fuliang, weng, guided, minh, thang, luong, frank, entire, discourses, capturing, continuity, jacob, eisenstein, normalization, file, sitting, directory, oct, delete, figured, belated, felt, wonder, txt, scathing, crying, skin, thickened, dismissing, defeats, purposes, providing, reject, reconcile, meet, mock, chance, started, addition, suggestion, exception, hedging, forceful, pop, psych, advice, ones, actions, spilling, cleaning, spilt, floor, occurrences, outlawed, rewrite, aware, removed, greatly, reduced, realize, signal, adopt, policy, longer, fewer, depart, constantly, battered, harsh, fair, evaluating, attacks, correlate, locks, mary, milk, cloze, extreme, strongest, justification, propose, assigns, measures, cross, entropy, empirical, divides, exponentiates, throwing, unseen, historically, chain, threw, combining, practice, technology, compete, recognizer, curious, building, project, stress, multiclass, soon, discovered, produced, rates, somewhere, 60s, 70s, fare, decided, wsj, portion, treebank, 48k, 1208, 5gram, smoothed, evaluated, latter, required, wants, scripts, highest, built, ppl1, 236, oov, ignored, calculation, quarter, includes, mandated, oovs, ahead, 43k, honestly, moderately, unigrams, frequencies, unrestricted, restricted, virtually, proposal, questionable, handcuffed, probabilities, 10194, 5357, 274, 251, 232, 230, 193, 14722, 1393, 1298, 512, 485, 439, 270, 163, 157, 108, roark, saraclar, collins, unfortunate, facto, assured, moon, neuter, stupid, artificial, encoding, memory, shaky, recollection, tend, idiosyncrasies, zealand, slippery, progressive, runs, yesterday, store, hamburger, head, leaves, tonight, definiteness, inspiration, teachers, definite, cnn, com, clicked, resigned, pressure, veterans, affairs, managers, media, article, shinseki, nonetheless, entities, precise, govern, pay, attention, named, phenomena, dramatically, conventions, socio, pin, parliament, mappings, hosed, overt, markings, plight, lune, name, consistently, copy, implicit, therefore, mandarin, wonderful, dagstuhl, yeah, invited, alex, philipp, helmut, hans, inviting, realization, share, retrospect, front, spect, bonnie, webber, marion, weller, martin, volk, marine, carpuat, jörg, tiedemann, popovic, deserves, credit, shed, commonly, morphology, determiners, unmarked, combines, tenses, appropriate, abstraction, widespread, among, initial, discussions, options, une, suggested, guide, picked, exercise, serious, yoga, noticed, initially, pushed, variable, female, immediately, fail, defeatist, attitude, unshift, overwhelming, majority, white, academia, opportunity, feeling, 1996, olympics, høyer, larsen, denmark, european, finals, gold, medal, dong, sport, dominated, indonesia, malaysia, growing, los, angeles, playing, kid, outlier, aspire, began, broadcasting, web, painting, laptop, emails, mostly, effectively, asking, range, unlike, race, outwardly, inferrable, noise, nuanced, mattered, aged, associate, prof, healthy, sorts, visibility, identifies, sufficed, queer, stem, interview, founder, lesbians, hrc, sympathize, precisely, instructorwhoismale, instructorilike, advisor, unable, attempt, failed, miserably, empiricist, theorist, designed, variables, cells, combinations, values, conditionals, computed, decide, measurement, sum, pedantic, define, versions, variants, uniform, sufficient, conditionalize, marginalize, finally, inspecting, onto, benign, approximations, plateau, differences, joints, larger, artifact, rise, weird, entirely, probable, matlab, bugs, myklrun, mykl, middle, max_a, reconstruct, perfectly, reconstruction, notice, dani, yogatama, noah, smith, skimmed, applied, hui, david, chiang, hallways, integral, incorporate, produces, fractional, peter, young, alice, lai, micah, hodosh, julia, hockenmaier, worlds, entailed, quoc, andrew, follow, gives, global, langvis, compositional, describing, nate, kushman, luke, zettlemoyer, artzi, dimes, unstructured, ccg, algebraic, germán, summarized, statement, conduct, annoyed, surrounding, despite, proper, secret, discover, hype, excessive, matthew, honnibal, allow, rewinds, disfluencies, denver, remove, old, arcs, detecting, disfluent, incremental, disfluency, sartorio, satta, jaokim, nivre, ryan, mcdonald, searnifying, approximate, oracle, achieve, incorrect, tabular, transition, caveats, acl14nlp, mentioned, servus, woid, món, welt, saluton, mondo, kaixo, mundua, hei, maailma, helló, világ, halo, こんにちは世界, sveika, pasaule, min, свету, verda, verden, olá, zdravo, svete, pozdravljen, svet, njatjeta, botë, свете, hej, världen, เฮลโลเวิลด์, merhaba, dünya, xin, chào, thế, giới, program, fun, collecting, difficult, overloading, 280, contribute, email, tweet, haldaume3, handouts, paste, respective, transcription, expanding, embeggings, closed, pairdirection, vocab, matters, tested, expansion, operation, 3cosmul, dot, germ, traditional, company, keeps, collect, separately, represented, scored, llr, rid, dimensionality, svd, nnmf, proportional, reduces, idf, collobert, weston, freely, extensively, literature, synonym, toefl, levied, imposed, requested, correlated, helicopters, motorcyles, dogs, elephants, shelf, selectional, preferences, pair, selects, gravity, associated, granddaughter, nearest, neighbor, pred, tie
Text of the page (random words):
conditional q a b from the true conditional p a b the middle row is the projection of this into two dimensions focusing on the divergence in the marginal and the bottom row is the projection onto the divergence in the conditional the title shows what the true distribution is in the form p a b p a b p a b p a b i chose this example because the joint has a correlation between a and b this example is fairly benign as the approximations become worse so do both of the recovered distributions in a fairly linear way until a plateau from the bottom row you can see that it s more important to get the conditional right than the marginal you can have a marginal that s quite far eg a kl of 1 5 and still get an almost perfect recovery of the conditional or joint but this is not true for large differences in the conditional b a one strange thing is that you often for different true joints see results that look like there s a very strange effect here in which a larger kl on b a can actually be better at the recovery of the conditional while worse at the recovery of the joint one can ask if this is an artifact of kl so let s switch to l1 and variational for the first set of plots and variational so in both l1 land and variational land you can do better on the conditional by being worse on the other conditional for the example that gave rise to the weird kl results we have the following for l1 which shows almost an identical effect for variational the effect is still the same okay so it s totally entirely possible perhaps probable that there s a bug in my code if you d like to look check out mykl m and myklrun m yes it s matlab let me know in the comments if there are bugs if you d like to look at more examples check out all ten examples posted by hal at 6 30 2014 09 30 00 am 1 comments 02 june 2014 role models during grad school my advisor suggested i identify a recent grad who has been to me successful i could then use him or her as a guide i picked someone he now knows who he is and the exercise was useful there are lots of ways to be successful in research land and this helped me focus rst relation topic shift i m fairly serious about yoga i ve had a lot of instructors over the years and noticed a high correlation between instructorilike and instructorwhoismale initially i believed this was because male instructors pushed more and that worked for me over time i realized that was not the full story i spent two weeks going to classes by instructors i hadn t had before to try to understand what variable s made the difference i ve believe now that a large part of the reason i like male instructors is precisely because they re male a female instructor would do some crazy pose and my brain would immediately say i could never do that a male instructor would do the same pose and my brain would say if he can do it so can i i d then try and fail several times but never with a defeatist attitude topic unshift i ve heard for a long time that having role models you can identify with is important as someone who has in almost all of my life fit into the overwhelming majority white male in tech academia it s been rare that i ve had the opportunity to really feel this effect for myself i try to believe things even if they haven t happened to me but it s always better when you can empathize rather than sympathize and it s easier to empathize when you ve actually been there the first time i remember feeling the effect of a role model who looks like me was at the 1996 olympics and poul erik høyer larsen denmark was the first european to ever win the badminton semi finals he then won the gold medal against dong jiong china this sport is dominated by indonesia china and malaysia growing up in a particular part of los angeles and playing badminton as a kid i was very much an outlier even though i d never heard for poul erik before everyone knew who jiong was his win gave me something i could aspire to a few years ago i began broadcasting my support of the lgbt community e g an hrc link on my web page and painting my laptop since then i ve gotten emails from several people mostly students effectively asking why there aren t more any lgbt role models in our community you can interpret community meaning anything in the nlp ml to cs to science tech range my answer i don t know it s hard to even know how large this community is because unlike things like race and binary gender it s not always outwardly inferrable with noise these issues effect tech is nuanced ways see for instance an interview with the founder of lesbians who tech or queer in stem for more this is all to say that having role models is important and yes it does matter who they are where they came from and what they look like it mattered to the high school aged version of me the grad school version of me and the associate prof version of me i m not saying anything new here but for our field to be healthy we need a large number of successful people who can be role models for all sorts of students and beyond token visibility is not enough because a single example of some particular label won t match with everyone who self identifies with that label the person i chose was yes a while male there were plenty to choose from but i chose him and others would not have sufficed posted by hal at 6 02 2014 11 52 00 am 4 comments 30 may 2014 past tense is not past tense i took part in a wonderful dagstuhl workshop this past february on translating morphologically rich languages yeah i also don t really know why i was invited p but many thanks to alex kevin philipp helmut and hans for inviting me i had a realization during this workshop that i thought i d share it s obvious in retrospect and perhaps in front spect for many of you much of this came up in the discussion with bonnie webber marion weller martin volk marine carpuat jörg tiedemann and maja popovic and maja deserves much credit for her awesome error analysis tool that helped shed some light on german one thing you commonly think of when translating into a morphologically rich language is that there s stuff you re going to have to hallucinate really this isn t an issue of morphology per se but just that this is one place where it s obvious for instance even going from english to french you ll have to hallucinate gender on your determiners un versus une and le versus la that s unmarked in english or when going from japanese which roughly combines present and future tenses into a single tense to english you ll have to hallucinate will at appropriate places an abstraction that i think was pretty widespread among the initial discussions in the workshop was that if you re going from language x to y there are basically two options phenomenon foo is explicit in y but implicit in x and therefore you ll have to hallucinate it i e tense is explicit in english but not in mandarin phenomenon bar is explicit in y and also explicit in x and so you can just copy it the problem is that 2 is just false even for things that you think it might not be false for that is to say just because two languages explicitly code for something that we give a consistent linguistic name to doesn t mean that they code for it consistently okay so you want examples an easy example is gender i ve been well assured that for instance french and russian both have explicit gender but just because some noun eg moon lune is feminine in french doesn t mean it s also feminine in russian in fact i think it s neuter you might argue gender is a stupid thing to pick because it s essentially an artificial encoding of who knows what how about tense that clearly has a semantic interpretation did something happen in the past the present or the future and so if languages x and y both express some particular tense they must be consistent in how they do it wrong now my memory is getting a bit shaky but my recollection is that for instance in newswire text it s very common in german to refer to things that have happened in the past in present tense to english speakers this is a strange convention we tend to refer to such things in past tense but it doesn t have to be so and of course english has it s own idiosyncrasies see the plight of the native german speaker who cannot understand english tense usage in new zealand news articles part of this is probably because tense even in english is a pretty slippery concept we native english speakers have no problem using present or progressive tense to refer to things that happened in the present john runs the past so yesterday i m running to the store and a hamburger falls on my head or the future my flight leaves at 8 00 tonight another easy example is definiteness thanks to kevin knight for this inspiration again our high school english teachers tell us that the definite has to refer to something that s already been introduced into context i just went to cnn com clicked on the very first link and the first sentence is the boss resigned under pressure and other veterans affairs managers are likely on the way out ok you could argue that the boss is already in the context of the us news media this is an article about shinseki but it s nonetheless very common to see english entities introduced using the and the precise rules that govern this may or may not be consistent across other languages the long and short of this is i like the fact that translation into morphologically rich languages makes us pay attention to linguistic divergence but that doesn t mean that divergences aren t there even when languages express the same set of linguistically named phenomena usage can vary dramatically be it for conventions socio linguistic reasons or other things that are hard to pin down it s just that by focusing all our energy on a very particular convention newswire parliament we can pretty easily learn these mappings because there s no variability add some variability and we re hosed even for languages with the same set of overt markings posted by hal at 5 30 2014 01 31 00 pm 11 comments 16 may 2014 perplexity versus error rate for language modeling it s fair to say that perplexity is the de facto standard for evaluating language models perplexity comes under the usual attacks what does it mean does it correlate with something we care about etc but here i want to attack it for a more pernicious reason it locks us in to probabilistic models background language modeling or more specifically history based language modeling as opposed to full sentence models is the task of predicting the next word in a text given the previous words for instance given the history mary likes her coffee with milk and a good language model might predict sugar and a bad language model might predict socks this is related to the notion of cloze probability it s quite clear that there is no right answer to any of these prediction problem as an extreme example given the history the there are any number of possible words that could go next there s just no way to know what the right answer is whether you re a machine or a person this is probably the strongest justification for a perplexity like measure since there s no right answer we ll let our learned model propose a probability distribution over all possible next words we say that this model is good if it assigns high probability to sugar and low probability to socks perplexity just measures the cross entropy between the empirical distribution the distribution of things that actually appear and the predicted distribution what your model likes and then divides by the number of words and exponentiates after throwing out unseen words the issue the issue here is that in order to compute perplexity your model must produce a probability distribution historically we ve liked probability distributions because they can be combined with other probability distributions according to the rules of probability eg bayes rule or chain rule of course we threw that out a long time ago when we realized that combining things for instance in log linear models worked a lot better in practice if you had a bit of data to tune the weights of the log linear models so the issue in my mind is that there s plenty of good technology out there for making predictions that does not produce probability distributions i think it s really unfortunate that non probabilistic approaches don t get to play the language modeling game because they produce the wrong sort of output according to the evaluation but not according to the real world i m not saying there aren t good reasons to like probabilistic models but just that alternatives are good and right now those alternatives cannot compete for instance roark saraclar and collins 2007 don t use perplexity at all and just go for word error rate of a speech recognizer around their perceptron based language model when i ran into this i was curious about building a language model using vw in the context of another project and also to stress test multiclass classification algorithms that scale well with respect to the number of classes as soon as i ran it i discovered the issue it produced results in the form of error rates as i recall it was a while ago the error rate was somewhere in the 60s or 70s i had absolutely no idea whether this was good or not it seemed reasonable to get a sense of how standard language models fare i decided to train a language model using srilm and evaluate it according to error rate to make my life easier i just ran it on the wsj portion of the penn treebank i used the first 48k sentences as train and the last 1208 sentences as test i trained a 5gram kneser ney smoothed language model and evaluated both perplexity and error rate the latter required a bit of effort if anyone wants the scripts let me know and i ll post them but basically i just take the lm s prediction to be the highest probability word given the context the language model i built had a perplexity ppl1 in srilm of 236 4 which seemed semi reasonable though of course pretty crappy there was an oov rate of 2 5 ignored in the perplexity calculation the overall error rate for this model was 75 2 this means that it was only guessing a quarter of words correct note that this includes the 2 5 errors mandated by oovs i also tried another version where all the model had to do was put the words in the right order in other words it knows ahead of time the set of words in the sentence and just has to pick between those 20 rather than between the full vocabulary 43k types this is maybe semi reasonable for mt the error rate under this setting was 66 8 honestly i expected it would be a lot better note that if you always guess the most frequent type in this data your error rate is 95 3 so why was it only moderately helpful 10 improvement to tell the language model what the set of possible words was basically because the model was always guessing really high probability unigrams below are the top ten predicted words when the model made an error with their frequenci...
|
