Meta tags:
description= Using large digital libraries to advance literary history;
Headings (most frequently used words):
the, of, ai, to, time, can, and, in, what, we, fiction, estimate, culture, posts, stone, shell, more, upside, make, us, intelligence, be, next, does, this, gpt, measure, why, how, well, do, mapping, is, cultural, technology, editing, marionette, theater, interesting, will, overconfident, should, artificial, person, shaped, language, models, predict, twist, story, have, any, relation, human, uncertainty, reason, not, trust, so, could, learn, from, predicting, three, pages, liberally, educated, students, need, than, consumers, save, matters, about, writing, at, price, using, passage, bag, words, methods, train, llms, no, model, latent, spaces, science, hasn, prepared, imagine, machine, learning, pagination, some, better, recent, categories, blogroll, archives, super, for, as, doesn, anyone, less, depressed, wait, good, idea, references,
Text of the page (most frequently used words):
the (769), and (390), that (326), this (166), but (160), for (147), not (132), models (125), are (120), can (113), about (112), with (107), will (97), have (90), #language (85), more (81), they (80), human (79), from (78), what (69), model (67), like (62), because (60), don (58), than (56), you (56), learning (54), has (49), one (49), our (48), new (48), #intelligence (47), could (46), how (45), think (45), when (43), was (43), other (42), may (42), there (42), people (41), time (41), need (39), also (39), their (39), all (39), post (39), just (38), some (36), would (35), good (34), culture (33), writing (32), know (32), them (32), social (31), even (31), fiction (30), make (30), only (30), who (30), been (30), https (29), machine (28), students (28), use (28), which (28), now (27), cultural (27), might (27), see (27), gpt (27), 2021 (26), things (26), space (26), find (26), way (26), where (25), these (25), different (25), reason (25), out (25), those (24), say (24), own (24), problem (24), why (24), should (23), want (23), readers (23), questions (23), still (23), using (23), artificial (22), between (22), data (22), meaning (22), story (21), its (21), here (21), org (21), passage (21), 2023 (20), work (20), next (20), instance (20), hard (20), had (20), question (19), into (19), measure (19), books (18), ways (18), let (18), words (18), something (18), much (18), many (18), get (18), technology (18), were (18), then (18), already (17), july (17), text (17), effects (17), plot (17), world (17), possible (17), same (17), author (17), point (16), research (16), understand (16), itself (16), does (16), researchers (16), each (16), parrots (16), often (16), minutes (16), blog (15), change (15), any (15), thinking (15), process (15), mean (15), years (15), actually (15), map (15), little (15), large (15), self (15), word (15), history (14), person (14), upside (14), take (14), range (14), science (14), yet (14), approach (14), neural (14), task (14), since (14), first (14), answer (14), june (13), 2017 (13), 2024 (13), project (13), after (13), right (13), agents (13), very (13), give (13), over (13), step (13), clip (13), end (13), stochastic (13), paper (13), feel (13), 2012 (12), scale (12), deep (12), predict (12), recent (12), better (12), beings (12), lot (12), whether (12), sense (12), look (12), making (12), really (12), date (12), across (12), specific (12), assignments (12), personality (12), comments (11), january (11), 2011 (11), analysis (11), topic (11), modeling (11), categories (11), literary (11), thing (11), clear (11), maybe (11), create (11), going (11), course (11), two (11), kind (11), less (11), doing (11), most (11), image (11), past (11), bad (11), least (11), enough (11), suspense (11), thought (11), challenge (11), his (11), above (11), apple (11), february (10), march (10), 2016 (10), 2018 (10), 2025 (10), interesting (10), expect (10), future (10), explore (10), latent (10), eventually (10), patterns (10), training (10), toward (10), keep (10), tedunderwood (10), force (10), positive (10), generative (10), used (10), debate (10), openai (10), several (10), prediction (10), help (10), style (10), code (10), references (10), both (10), back (10), working (10), ability (10), september (9), 2015 (9), historical (9), theater (9), reading (9), too (9), open (9), idea (9), through (9), perhaps (9), imagine (9), read (9), produce (9), general (9), run (9), goal (9), doi (9), knowledge (9), technologies (9), likely (9), single (9), while (9), effect (9), doesn (9), week (9), passages (9), started (9), learn (9), ask (9), chapter (9), report (8), com (8), october (8), 2013 (8), humanities (8), character (8), posts (8), tags (8), probably (8), well (8), instead (8), vast (8), stories (8), possibility (8), art (8), wrong (8), play (8), another (8), summary (8), collective (8), makes (8), century (8), pages (8), however (8), side (8), argument (8), believe (8), prompt (8), fully (8), underwood (8), critical (8), long (8), conversation (8), example (8), represent (8), recognize (8), short (8), part (8), describe (8), experiment (8), predictions (8), found (8), page (8), reference (8), real (8), balance (8), professors (8), happen (8), bluesky (8), consciousness (8), content (7), stone (7), december (7), august (7), 2014 (7), 2020 (7), search (7), methods (7), trend (7), transformer (7), shaped (7), marionette (7), seven (7), put (7), boring (7), few (7), number (7), ever (7), either (7), kinds (7), form (7), others (7), novel (7), system (7), being (7), important (7), behavior (7), yes (7), hand (7), sentence (7), humans (7), images (7), old (7), cost (7), your (7), scene (7), arxiv (7), aspects (7), picture (7), big (7), potential (7), agree (7), train (7), public (7), disciplines (7), fact (7), pretend (7), matters (7), asked (7), three (7), chatgpt (7), elapsed (7), narrative (7), press (7), construct (7), require (7), start (7), getting (7), sometimes (7), her (7), days (7), confidence (7), try (7), estimate (7), teach (7), save (7), trained (7), robin (7), education (7), vision (7), surprise (7), anthropic (7), depressed (7), write (6), shell (6), april (6), attention (6), problems (6), twist (6), quickly (6), always (6), worlds (6), version (6), ends (6), sort (6), library (6), door (6), whole (6), seem (6), early (6), suspect (6), add (6), dall (6), writers (6), seems (6), place (6), field (6), especially (6), great (6), similar (6), nothing (6), means (6), prefer (6), made (6), hasn (6), did (6), conference (6), value (6), ted (6), abs (6), rather (6), book (6), anyone (6), down (6), usually (6), decide (6), tools (6), mapping (6), produced (6), surprising (6), role (6), around (6), understanding (6), mostly (6), llms (6), simple (6), api (6), method (6), provide (6), before (6), assistant (6), reasons (6), easier (6), experience (6), described (6), every (6), themselves (6), github (6), best (6), voice (6), term (6), university (6), uncertainty (6), bum (6), someone (6), novels (6), breaks (6), moral (6), super (6), media (6), aesthetic (6), persona (6), required (5), reader (5), account (5), victorian (5), article (5), digital (5), 19c (5), teaching (5), analytics (5), saying (5), become (5), useful (5), reality (5), storytelling (5), matter (5), surprised (5), pattern (5), never (5), game (5), exploring (5), support (5), characters (5), styles (5), users (5), starting (5), trying (5), without (5), individual (5), created (5), larger (5), mental (5), resemble (5), title (5), gets (5), claim (5), interested (5), doubt (5), test (5), isn (5), bert (5), certain (5), job (5), haven (5), carbon (5), feature (5), dreams (5), artists (5), white (5), material (5), risks (5), dangers (5), dangerous (5), consequences (5), biases (5), pictures (5), position (5), behind (5), explicitly (5), historians (5), conflict (5), offer (5), texts (5), response (5), given (5), forms (5), far (5), earlier (5), advance (5), second (5), stable (5), jane (5), statistical (5), currently (5), remember (5), measuring (5), called (5), compare (5), core (5), challenges (5), day (5), final (5), address (5), edit (5), reply (5), characterize (5), set (5), average (5), axis (5), until (5), student (5), principles (5), home (5), valid (5), sound (5), suggest (5), distance (5), continue (5), chang (5), 900 (5), necessarily (5), installments (5), hound (5), difficult (5), askell (5), personhood (5), claude (5), power (5), invented (5), void (5), bots (5), stateful (5), name (4), log (4), november (4), creative (4), era (4), theory (4), genre (4), strategies (4), phenomenon (4), humanists (4), libraries (4), decade (4), direction (4), cyberpunk (4), hidden (4), wider (4), moment (4), analogous (4), news (4), helpful (4), allow (4), everything (4), works (4), familiar (4), interactive (4), fictional (4), tend (4), thousands (4), friends (4), understood (4), having (4), plans (4), sure (4), thoughts (4), reach (4), assume (4), presumably (4), living (4), impressive (4), order (4), below (4), reasoning (4), evidence (4), limited (4), access (4), trust (4), knows (4), four (4), turn (4), bit (4), guess (4), hope (4), prepared (4), low (4), against (4), mark (4), journal (4), perspective (4), handle (4), notes (4), trans (4), bender (4), proceedings (4), spaces (4), imagined (4), intent (4), merely (4), suggested (4), easy (4), essay (4), closely (4), air (4), failed (4), respond (4), internet (4), looks (4), unreal (4), engine (4), consider (4), didn (4), discourse (4), critics (4), scholars (4), universities (4), tech (4), seemed (4), move (4), nsf (4), begin (4), perspectives (4), cover (4), develop (4), multiple (4), fine (4), structure (4), corpora (4), reflect (4), web (4), wide (4), exactly (4), genres (4), wouldn (4), published (4), mind (4), pose (4), tends (4), center (4), tasks (4), yauney (4), david (4), running (4), spite (4), agreement (4), she (4), memory (4), hours (4), hour (4), gave (4), results (4), slightly (4), autonomy (4), tune (4), estimates (4), bag (4), experiments (4), valuable (4), focused (4), cases (4), compared (4), provided (4), tell (4), assumptions (4), honest (4), instruction (4), year (4), thirty (4), explain (4), asks (4), won (4), anything (4), sciences (4), mla (4), cccc (4), thesis (4), gentle (4), effectively (4), break (4), assistance (4), essays (4), asking (4), ourselves (4), insight (4), feelings (4), initial (4), advice (4), guide (4), comes (4), ago (4), expected (4), jobs (4), automated (4), implies (4), exciting (4), talk (4), review (4), encouraging (4), serial (4), accuracy (4), roughly (4), baskervilles (4), holmes (4), avoid (4), names (4), scaling (4), birds (4), sky (4), theatrical (4), mask (4), signals (4), last (4), altman (4), persons (4), ease (4), retrieved (4), golden (4), gate (4), sentimental (4), marionettes (4), agent (4), email (3), view (3), site (3), sign (3), wordpress (3), 2026 (3), 2019 (3), planet (3), dan (3), correlation (3), mining (3), diction (3), interpretive (3), criticism (3), discovery (3), building (3), liberally (3), educated (3), consumers (3), overconfident (3), distant (3), emergence (3), babel (3), places (3), invent (3), almost (3), fear (3), worry (3), doors (3), piece (3), thread (3), corporations (3), autonomous (3), loosely (3), robots (3), alien (3), taught (3), within (3), examples (3), genuinely (3), original (3), authors (3), universe (3), bigger (3), call (3), borrowed (3), maps (3), interactively (3), build (3), worse (3), reproduce (3), solving (3), primates (3), life (3), predicting (3), nineteenth (3), twitter (3), boxes (3), mistake (3), leave (3), off (3), smart (3), chimpanzee (3), individuals (3), quite (3), complex (3), association (3), certainly (3), taking (3), supposed (3), fill (3), released (3), clearly (3), verbal (3), obvious (3), energy (3), importantly (3), weigh (3), footprint (3), argue (3), successful (3), spatial (3), charlie (3), edu (3), generation (3), house (3), national (3), black (3), net (3), katherine (3), foundation (3), transparency (3), systems (3), enjoy (3), shared (3), concerns (3), note (3), gives (3), allowed (3), analogy (3), political (3), everyone (3), expertise (3), negative (3), absolutely (3), existing (3), impossible (3), attempt (3), practice (3), together (3), features (3), particular (3), visual (3), although (3), graduate (3), tuning (3), purpose (3), companies (3), differences (3), publication (3), service (3), somehow (3), free (3), bias (3), typically (3), immediate (3), templates (3), creatively (3), necessary (3), study (3), consistent (3), definition (3), captioned (3), phrase (3), traces (3), communication (3), hear (3), state (3), simply (3), computer (3), discuss (3), event (3), roundtable (3), consensus (3), stanford (3), implications (3), simon (3), willison (3), greg (3), computational (3), measured (3), 1980 (3), accurate (3), interpretable (3), defined (3), relatively (3), concepts (3), thousand (3), seen (3), show (3), needs (3), assistants (3), bamman (3), explains (3), nature (3), man (3), night (3), again (3), close (3), whatever (3), narrator (3), childhood (3), replies (3), amount (3), turbo (3), plus (3), ambiguity (3), providing (3), high (3), learned (3), front (3), count (3), yesterday (3), ideal (3), notice (3), cautious (3), needed (3), explicit (3), events (3), spoken (3), passed (3), weeks (3), user (3), interaction (3), durations (3), counting (3), precisely (3), feels (3), length (3), represents (3), entity (3), specify (3), documents (3), based (3), corey (3), changed (3), fall (3), willing (3), once (3), got (3), reflection (3), preserve (3), solve (3), ideas (3), further (3), wheel (3), along (3), fake (3), spin (3), cheat (3), metacognition (3), suggesting (3), midjourney (3), harder (3), angle (3), suggestions (3), beginning (3), wanted (3), putting (3), technical (3), lives (3), html (3), concept (3), allows (3), framed (3), available (3), talking (3), case (3), poses (3), plato (3), novelty (3), family (3), serialization (3), ryan (3), anders (3), group (3), arguing (3), narratives (3), turns (3), expectation (3), mystery (3), installment (3), encourage (3), divergence (3), summaries (3), tempting (3), written (3), interfaces (3), laugh (3), cringe (3), telling (3), products (3), provides (3), caddies (3), interacting (3), argues (3), abstract (3), demo (3), pygmalion (3), chatbot (3), evans (3), obsessed (3), flight (3), spectrum (3), alignment (3), passive (3), accessible (3), passively (3), textbook (3), points (3), wait (3), kapoor (3), farrell (3), gopnik (3), monosemanticity (3), singularity (3), editing (3), manipulate (3), printing (3), modes (3), durable (3), protean (3), legal (3), freedom (3), journey (3), kleist (3), constructed (3), distasteful (3), advantage (3), protagonist (3), inner (3), penny (3), network (3), morality (3), moltbook (3), animals (3), familiars (3), subscribed (2), subscribe (2), 2010 (2), google (2), cohen (2), american (2), under (2), politics (2), uncategorized (2), statistics (2), literature (2), reproducibility (2), replication (2), poetic (2), pace (2), disciplinary (2), collection (2), 20c (2), gender (2), broad (2), older (2), extrapolating (2), deepfakes (2), fears (2), borges (2), optimistically (2), money (2), portal (2), fantasy (2), 2020s (2), subgenre (2), sinister (2), etc (2), led (2), considering (2), decades (2), inscrutable (2), transforming (2), containing (2), robot (2), collaborative (2), script (2), protein (2), folding (2), intelligent (2), music (2), activity (2), expectations (2), increasing (2), fan (2), developed (2), variety (2), arthur (2), devoted (2), safe (2), children (2), symbolic (2), coordinates (2), desires (2), structures (2), bananas (2), advances (2), shrug (2), none (2), envision (2), includes (2), jerome (2), lived (2), true (2), generates (2), expressing (2), intention (2), inference (2), implied (2), contest (2), pursuing (2), unfamiliar (2), manner (2), broadly (2), learns (2), photo (2), television (2), detail (2), plausible (2), recently (2), unlikely (2), threat (2), ready (2), struggling (2), lately (2), said (2), contemporary (2), efforts (2), cause (2), reduction (2), actively (2), agenda (2), renewable (2), sham (2), deceptive (2), campaign (2), mashable (2), skeptical (2), lies (2), institute (2), skeptically (2), limits (2), interest (2), berkeley (2), gray (2), chen (2), ilya (2), sutskever (2), zero (2), resource (2), gov (2), wang (2), autoregressive (2), martin (2), daniel (2), jacob (2), eisenstein (2), william (2), corr (2), size (2), improve (2), discussed (2), bommasani (2), 100 (2), soni (2), lauren (2), abolitionist (2), networks (2), activist (2), newspapers (2), culturalanalytics (2), list (2), production (2), bode (2), non (2), oct (2), recap (2), gebru (2), mcmillan (2), major (2), 610 (2), 623 (2), 1145 (2), 3442188 (2), 3445922 (2), acm (2), fairness (2), accountability (2), information (2), reframe (2), imply (2), illuminate (2), fascinating (2), transform (2), deeply (2), contrast (2), connects (2), plan (2), intelligences (2), tropes (2), conscious (2), various (2), care (2), ethical (2), dilemma (2), controlled (2), movable (2), type (2), war (2), meant (2), shape (2), common (2), observation (2), machines (2), angel (2), diffusion (2), region (2), fun (2), themes (2), differently (2), embeddings (2), dizzying (2), devoid (2), rhetorical (2), broader (2), outside (2), alternatives (2), forward (2), danger (2), computing (2), resources (2), huggingface (2), helping (2), languages (2), distinguish (2), flexibility (2), associate (2), statement (2), challenging (2), requires (2), correctly (2), contrary (2), acceptable (2), audiences (2), delicate (2), share (2), token (2), practices (2), disappointing (2), starts (2), instructions (2), select (2), dictionary (2), named (2), subjects (2), tendency (2), grounded (2), communicative (2), 616 (2), intentional (2), sequences (2), defines (2), trajectory (2), parts (2), anyway (2), natural (2), changing (2), academics (2), credit (2), discussion (2), applications (2), princeton (2), prepare (2), follows (2), mimno (2), away (2), loud (2), automatically (2), document (2), fast (2), modest (2), measurement (2), error (2), confident (2), separate (2), systematically (2), coming (2), showing (2), addressed (2), signs (2), six (2), heard (2), dreaming (2), child (2), incident (2), couch (2), hands (2), water (2), remembers (2), 10080 (2), moderate (2), passing (2), interpretability (2), myself (2), prompts (2), cheaper (2), correlated (2), 121 (2), inter (2), closer (2), took (2), months (2), llm (2), main (2), pointed (2), complete (2), added (2), allowing (2), extremely (2), answers (2), capacity (2), assess (2), quinn (2), dombrowski (2), query (2), morning (2), window (2), coffee (2), dressed (2), seeing (2), car (2), ego (2), responding (2), slowly (2), twice (2), pretty (2), encouraged (2), responses (2), five (2), summarize (2), include (2), taken (2), description (2), dialogue (2), units (2), times (2), understands (2), higher (2), transformed (2), lower (2), definitely (2), estimating (2), labeled (2), predicted (2), hundred (2), showed (2), manually (2), literally (2), says (2), conclude (2), 250 (2), calculated (2), passes (2), minute (2), automate (2), addresses (2), hundreds (2), watkins (2), marc (2), 1984 (2), rudy (2), scarborough (2), joint (2), overview (2), annoying (2), patching (2), pedagogy (2), sufficient (2), capable (2), framing (2), came (2), fresh (2), puts (2), emphasis (2), turning (2), critique (2), normative (2), venture (2), meta (2), adapting (2), intellectual (2), advanced (2), gather (2), interpret (2), rehearse (2), processes (2), originally (2), simulate (2), assignment (2), simulating (2), replicate (2), attitude (2), wisdom (2), optimistic (2), elegiac (2), opportunity (2), spins (2), weird (2), cynical (2), foreground (2), externalize (2), focus (2), imitate (2), trace (2), adjusting (2), practical (2), college (2), avoids (2), class (2), exams (2), solution (2), daughter (2), team (2), price (2), imperfect (2), department (2), critically (2), admittedly (2), illustration (2), variance (2), basic (2), level (2), 2030s (2), courses (2), recommend (2), consumer (2), looking (2), collar (2), choose (2), emerge (2), solved (2), discover (2), today (2), coherent (2), warhol (2), tobin (2), harvard (2), smuts (2), 2009 (2), paradox (2), sinykin (2), sap (2), choi (2), acl (2), meeting (2), piper (2), revelation (2), mcgrath (2), modernist (2), haugtvedt (2), magazines (2), periodicals (2), garrett (2), dames (2), twenty (2), speak (2), beekman (2), woman (2), previously (2), presented (2), multi (2), chunks (2), appropriate (2), related (2), crude (2), trusting (2), cliffhanger (2), standard (2), ensure (2), epistemic (2), rhythms (2), doctor (2), footprints (2), answered (2), predictable (2), steel (2), inferences (2), draw (2), diss (2), cosine (2), check (2), supply (2), masked (2), moreover (2), flying (2), blind (2), shouldn (2), studying (2), reveals (2), tended (2), actual (2), drawn (2), johnson (2), claiming (2), correlate (2), sees (2), ernie (2), happens (2), sesame (2), street (2), beyond (2), impulse (2), warner (2), presents (2), spike (2), jonze (2), mallarmé (2), stéphane (2), neuromancer (2), amanda (2), richie (2), stuart (2), creeped (2), reminds (2), full (2), truth (2), anthropomorphize (2), pure (2), sounding (2), interface (2), impersonal (2), lie (2), reminders (2), owners (2), sounded (2), scarlett (2), johansson (2), eyes (2), opinions (2), determine (2), react (2), grumpy (2), anthropocentric (2), gendered (2), laughter (2), creature (2), emotion (2), caddy (2), quietly (2), extension (2), personal (2), siri (2), coalesce (2), woo (2), sam (2), insist (2), fair (2), risk (2), ambitious (2), downsides (2), lean (2), league (2), boots (2), frame (2), monolith (2), stack (2), overflow (2), threads (2), repos (2), entry (2), daunting (2), prove (2), 1990s (2), program (2), disagree (2), overconfidence (2), bugs (2), evolve (2), strong (2), adding (2), couldn (2), wilder (2), bryanwilder (2), ebooks (2), 2022 (2), jowett (2), phaedrus (2), narayanan (2), normal (2), mollick (2), manning (2), shalizi (2), circuits (2), index (2), extracting (2), sonnet (2), samaltman (2), dark (2), depressing (2), sounds (2), costs (2), benefits (2), purely (2), sharpen (2), swing (2), forth (2), grok (2), heat (2), eliminate (2), load (2), bearing (2), ideological (2), weapons (2), chance (2), habits (2), produces (2), outweigh (2), sustained (2), similarly (2), adjectives (2), directions (2), neurons (2), vertigo (2), bridge (2), able (2), among (2), sref (2), stylistic (2), dimensional (2), vector (2), james (2), fail (2), survey (2), skeptics (2), potentially (2), mature (2), generally (2), stage (2), orders (2), magnitude (2), greater (2), ill (2), substitute (2), false (2), agi (2), generalizable (2), entities (2), path (2), capability (2), independent (2), command (2), external (2), else (2), decisions (2), agency (2), nerds (2), recognized (2), naive (2), tree (2), grace (2), personas (2), shtick (2), struggle (2), sentimentality (2), wince (2), voyage (2), puppet (2), sitting (2), lap (2), pinocchio (2), schiller (2), staged (2), humanity (2), flaw (2), actors (2), alas (2), weakness (2), inhabit (2), accounts (2), riff (2), dancer (2), gesture (2), truly (2), aesthetically (2), discomfort (2), cookie (2), puppets (2), cheesy (2), morally (2), maltese (2), audience (2), patron (2), memento (2), menu (2), website, comment, loading, collapse, bar, manage, subscriptions, privacy, join, 596, subscribers, top, archives, scottbot, irregular, seasr, presentations, sapping, parezco, digo, monk, meandre, installation, ngram, viewer, geoffrey, nunberg, digitally, assisted, collocates, english, 1810, artfl, 18thconnect, blogroll, licensed, commons, attribution, sharealike, unported, license, visualization, unsupervised, undigitized, sociology, romantic, representativeness, reception, poetry, ocr, correction, nonconsumptive, ngrams, methodology, metadata, math, lsa, linguistics, impressionistic, historicism, comparison, bayesian, 18c, rss, dataset, 1700, 1922, structuralist, theories, computers, instability, standards, outlines, jdh, newer, pagination, movements, qanon, eager, parallel, armored, endlessly, impenetrable, servant, refuses, pod, bay, heroines, chalk, chosen, leading, bookshelf, personally, mashup, wardrobes, encylopedias, tlön, futuristic, eric, desmazieres, emotions, servile, excite, terrify, labyrinth, infinite, eye, alternate, shelves, god, extrapolation, weirder, journalism, invariably, invite, servants, antagonists, terminator, hal, paternal, condescension, dread, reactions, domains, creators, collaboratively, emerges, exploration, solitary, undertaken, geniuses, scratch, origin, folk, tales, rearranging, moves, riffing, established, involved, creating, replace, interconnected, proposed, grown, ups, settings, illustrations, rackham, graphic, welcome, collaborators, happening, sites, tens, competitors, enter, startup, motto, subreddit, dungeon, endeavors, including, needn, attempts, imagining, collaborator, amino, acids, infancy, improved, foolish, ignore, clues, bizarrely, parochial, writer, stacking, supposedly, overrated, buzz, relax, realist, goals, achieve, paragraph, wander, understandably, ultimately, wolfgang, köhler, 1925, mentality, apes, driven, mario, klingemann, byline, extrapolate, expressed, symbols, opinion, probability, sentences, shivery, crossed, significant, gary, marcus, ernest, davies, correlations, fusing, abstractions, lemur, hold, umbrella, surprisingly, 1910s, output, baby, daikon, radish, tutu, walking, dog, involving, underway, developments, imaginary, translate, descriptions, opera, zombie, apocalypse, receding, fight, rescue, replicants
Text of the page (random words):
d presumably needs to make two cups of coffee then shaves and showers and gets dressed before stepping out his front door and seeing a car 2 making coffee showering and getting dressed take at least an hour there s some ambiguity about whether to count the implicit reference to yesterday since this is the next morning as time elapsed in the passage but let s say no since yesterday is not actually described so an hour to 90 minutes 3 1 25 hours have elapsed multiplying by 60 minutes an hour that s 75 minutes 4 75 minutes 5 low confidence because of ambiguity about a reference to the previous day using some code generously shared by quinn dombrowski i gave the model four query reply sequences like this then asked it to characterize a new passage with the same instructions to assess performance on this simple task i just extracted the minutes reported in step 4 and compared them to our human estimates from 2017 other forms of content analysis might require the model to edit or mark up the text provided in the prompt that s doable but i thought i would start with something simple i added step 5 allowing the model to describe its own confidence because in early experiments i found the model s tendency to editorialize extremely valuable the answers to step 5 were also fun to read as replies scrolled up the page because my new sorcerer s assistant complained volubly about the ambiguity of its task 30 minutes low confidence as the passage is more focused on the poetic and symbolic aspects of the scene rather than providing a clear sense of time in some cases it refused the task altogether and threw the question about confidence back in my face n a high confidence that no specific amount of time is described this capacity for backtalk is a feature not a bug i learned a lot from it but after i adjusted my prompt to address the ambiguities the assistant correctly pointed out complete refusal to answer the question was rare how well does gpt 4 estimate time i had the turbo model code 483 passages its predictions correlated with human estimates at r 59 i only asked gpt 4 to code 121 passages because it s more expensive to run than turbo and it achieved r 68 this is not as good as inter human agreement 74 but it s closer to human readers than to bag of words models 35 49 and of course gpt 4 does the work more quickly than human readers it took the three of us several months to generate this data but my llm experiment was run in a couple of days plus given an api large language models are easier to use than other forms of machine learning the main challenge is to describe your question systematically this is a method that could realistically be used by researchers with relatively little programming experience the total cost to my openai account was 22 of that amount about 8 was testing the prompts and running on the cheaper turbo api the final run on gpt 4 cost 14 if i had completely replicated our work from 2017 on gpt 4 it would have cost 140 so a project like this will not bankrupt researchers but reliance on openai is still a bad idea in the long run universities should develop their own models and apis both for reasons of institutional autonomy survival and so that researchers can inspect and tune the models however one could exaggerate the problems of interpretability created by a closed model watching the model think out loud about each passage actually gave me a better sense of how results were produced than i could extract from my own records of human research in 2017 see below for an example from gpt 4 human readers don t usually document their thought process this verbosely also because human beings can t erase their own memories it s not possible for us to repeat the same question with slightly different language if i try that on myself i m going to remember what i thought before and probably think it again but a model can wake up each moment with eternal sunshine of the spotless mind abridged passage from jane eyre and signs for aught we know may be but the sympathies of nature with man when i was a little girl only six years old i one night heard bessie leaven say to martha abbot that she had been dreaming about a little child of late i had often recalled this saying and this incident for during the past week scarcely a night had gone over my couch that had not brought with it a dream of an infant which i sometimes hushed in my arms sometimes dandled on my knee sometimes watched playing with daisies on a lawn or again dabbling its hands in running water it was a wailing child this night and a laughing one the next now it nestled close to me and now it ran from me but whatever mood the apparition evinced whatever aspect it wore it failed not for seven successive nights to meet me the moment i entered the land of slumber assistant replies 1 the narrator recalls a saying about dreaming of children being a sign of trouble and remembers an incident from her childhood she then describes her recent dreams of infants over the past week 2 the passage mentions the narrator s childhood memory which happened when she was six years old and the past week of dreams the time elapsed in the passage is the week of dreams 3 1 week has elapsed at seven days a week that s 7 days 24 hours in a day so 168 hours 60 minutes in an hour so 10080 minutes 4 10080 minutes 5 moderate confidence as the childhood memory is not counted as time passing in the scene what no model can do while i think large language models will be tremendously useful assistants in content analysis i don t think we can dispense with multiple human readers as david bamman explains in a good twitter thread the core challenges of this work are a coming up with a new construct to measure b demonstrating that it can be measured and c showing that there s value in doing so intersubjective agreement between human beings is still the only way to know that we have addressed those questions that s why i wouldn t feel comfortable just making up my own definition of say suspense teaching a model to measure it and then running the model across a thousand books the problem with that approach is not that llms have measurement error as we ve seen in greg yauney s paper measurement methods with much more error can be productive the problem is that it isn t clear what a single researcher s construct means in the first place to be confident that we re measuring something called suspense we need to show that multiple people recognize it as suspense and in spite of all the rhetorical personification in this blog post which i trust you have understood rhetorically a model is not a separate person so a project of this kind still needs to start by getting several human beings to read systematically and compare notes before the human codebook is translated into a prompt on the other hand i do think language models will allow us to pose questions that are currently hard to pose at scale questions about plot and character for instance require reading a whole book while applying delicate decision criteria that may be hard to remember and keep stable across many books language models will help here not just because they re fast but because they can provide a relatively stable yardstick by which to measure slippery concepts even at a modest scale of analysis this is a preliminary report on a very strange world for me the most surprising take away from this experiment was not that deep learning is more accurate than statistical nlp but that it may also be in some ways more interpretable because a language model has to think out loud it tends to automatically document its own reasoning this is useful even when or especially when the model doesn t think you ve defined the construct it s supposed to measure clearly enough references 1 gérard genette n arrative discourse an essay in method trans jane e lewin ithaca cornell univ press 1980 97 2 ted underwood why literary time is measured in minutes new literary history 85 2 2018 351 65 3 greg yauney ted underwood and david mimno computational prediction of elapsed narrative time 2019 4 simon willison a simple python wrapper for the chatgpt api 2023 tags fiction gpt 4 llms time categories interpretive theory machine learning social effects of machine learning transformer models mapping the latent spaces of culture post author by tedunderwood post date october 21 2021 on tues oct 26 the center for digital humanities at princeton will sponsor a roundtable on the implications of stochastic parrots for the humanities to prepare for that roundtable they asked three humanists to write position papers on the topic mine follows i ll give a 5 min 500 word précis at the event itself this is the 2000 word version with pictures it also has a doi if you want a stable version to cite the technology at the center of this roundtable doesn t yet have a consensus name some observers point to an architecture the transformer 1 on the dangers of stochastic parrots focuses on size and discusses large language models 2 a paper from stanford emphasizes applications foundation models are those that can adapt to a wide range of downstream tasks 3 each definition identifies a different feature of recent research as the one that matters to keep that question open i ll refer here to deep neural models of language a looser category however we define them neural models of language are already changing the way we search the web write code and even play games academics outside computer science urgently need to discuss their role on the dangers of stochastic parrots deserves credit for starting the discussion especially since publication required tenacity and courage i am honored to be part of an event exploring its significance for the humanities the argument that bender et al advance has two parts first that large language models pose social risks and second that they will turn out to be misdirected research effort anyway since they pretend to perform natural language understanding but do not have access to meaning 615 i agree that the trajectory of recent research is dangerous but to understand the risks language models pose i think we will need to understand how they produce meaning the premise that they simply do not have access to meaning tends to prevent us from grasping the models social role i hope humanists can help here by offering a wider range of ways to think about the work language does it is true that language models don t yet represent their own purposes or an interlocutor s state of mind these are important aspects of language and for stochastic parrots they are the whole story the article defines meaning as meaning conveyed between individuals and grounded in communicative intent 616 but in historical disciplines it is far from obvious that all meaning boils down to intentional communication between individuals historians often use meaning to describe something more collective because the meaning of a literary work for example is not circumscribed by intent it is common for debates about the meaning of a text to depend more on connections to books published a century earlier or later than on reconstructing the author s conscious plan 4 i understand why researchers in a field named artificial intelligence would associate meaning with mental activity and see writing as a dubious proxy for it but historical disciplines rarely have access to minds or even living subjects we work mostly with texts and other traces for this reason i m not troubled by the part of stochastic parrots that warns about the human tendency to attribute meaning to text even when the text is not grounded in communicative intent 618 616 historians are already in the habit of finding meaning in genres nursery rhymes folktale motifs ruins political trends and other patterns that never had a single author with a clear purpose 5 if we could only find meaning in intentional communication we wouldn t find much meaning in the past at all so not all historical researchers will be scandalized when we hear that a model is merely stitching together sequences of linguistic forms it has observed in its vast training data 617 that s often what we do too and we could use help a willingness to find meaning in collective patterns may be especially necessary for disciplines that study the past but this flexibility is not limited to scholars the writers and artists who borrow language models for creative work likewise appreciate that their instructions to the model acquire meaning from a training corpus the phrase unreal engine for instance encourages clip to select pictures with a consistent cartoonified style but this has nothing to do with the dictionary definition of unreal it s just a helpful side effect of the fact that many pictures are captioned with the name of the game engine that produced them in short i think people who use neural models of language typically use them for a different purpose than stochastic parrots assumes the immediate value of these models is often not to mimic individual language understanding but to represent specific cultural practices like styles or expository templates so they can be studied and creatively remixed this may be disappointing for disciplines that aspire to model general intelligence but for historians and artists cultural specificity is not disappointing intelligence only starts to interest us after it mixes with time to become a biased limited pattern of collective life models of culture are exactly what we need while i m skeptical that language models are devoid of meaning i do share other concerns in stochastic parrots for instance i agree that researchers will need a way to understand the subset of texts that shape a model s response to a given prompt culture is historically specific so models will never be free of omission and bias but by the same token we need to know which practices they represent if companies want to offer language models as a service to the public say in web search they will need to do even more than know what the models represent somehow a single model will need to produce a picture of the world that is acceptable to a wide range of audiences without amplifying harmful biases or filtering out minority discourses bender et al 614 that s a delicate balancing act historians don t have to compress their material as severely since history is notoriously a story of conflict and our sources were interested participants few people expect historians to represent all aspects of the past with one correctly balanced model on the contrary historical inquiry is usually about comparing perspectives machine learning is not the only way to do this but it can help for instance researchers can measure differences of perspective by training multiple models on different publication venues or slices of the timeline 6 when research is organized by this sort of comparative purpose the biases in data are not usually a reason to refrain from modeling but a reason to c...
|
